A transpection is a dialog for learning. James Bach describes it here. Transpection is a technique we use a lot to refine ideas for presentations, for articles, for our course, or for our own understanding. Sometimes it’s all of them put together. Transpective sessions with James have led me sharpen ideas and to do work of which I’m very proud—on test coverage, for example (articles here, here, and here). Sometimes the conversation happens in speech (as in this dialog on what scripts tell us, or this one between James and Mike Kelly titled “Is There A Problem Here?” Sometimes the conversation happens in an instant message system like Skype. The former is more dynamic; the latter leaves a written transcript, which can be handy.
A few months back, James Bach and I took some time for a transpection session. Initially, the goal was to provide a demonstration of transpection for a particular student of James’, but then we realized that not only the session but its content might be of more general interest. James started the conversation.
James: Consider the definition: “A test consists of at least an input and an expectation.. Is that true? What does it mean?
Michael: Let’s apply your three-word heuristic for critical thinking: “Huh? Really? So?”
James: Can we go deeper with it?
Michael: Sure. Would sitting and observing a system that is already doing something (or not) be considered an “input”?
James: What do you think?
Michael: My first approach is to look at statements like the one you provide above and ask how broadly we could interpret them. I also try to falsify them. So… One could falsify it by asking, “Can I think of a case in which I don’t provide input, and it’s still a test. Where I don’t have an expectation, and it’s still a test?”
Michael: One approach to answering to that question would be to ask “Well, what’s an input? What’s an expectation?”
James: So, let’s imagine an acrylic box. Inside the box runs a clock. You can see the clock face. You can’t interact with the clock (except to look at it in normal light). Can the clock be tested?
Michael: I can certainly “ask questions in order to evaluate it” (that’s the essence of your definition of testing), or “gather information with the intention of informing a decision” (which is Jerry Weinberg’s). I can conjecture. I can observe.
James: But can you provide an input? And can you test it?
Michael: In one sense, yes. I provide input by observing it. I don’t know if other people would be willing to take such a broad interpretation of input, but I could do that for the purpose of the exercise.
James: That seems like an odd definition of input. I don’t get it. It doesn’t process you looking at it. It doesn’t react to you looking at it.
Michael: It does seem like an odd defintion. And yet…I put myself into the system of the clock. So what do YOU mean by “input”?
James: I will be happy to tell you. But first can you define what input means to you? What would be a reasonable definition?
Michael: In computer and testing talk, it usually means to provide data to a function for processing.
James: I agree. What does “provide data” mean?
Michael: It could mean to enter a sequence of keystrokes at the keyboard; to move a mouse and click on an element of the screen; to connect the computer to some stream of network traffic. To feed the machine with something to process. It more generally means, I think, to alter the current functioning of the system in some way.
James: To exert control over the system?
Michael: Very broadly, yes. Again, I’m not sure that some would accept such a general defintion, but I would.
James: I think we can divide input into symbolic and non-symbolic input and explicit and implicit input. Symbolic input is data processed by the computer; data meaning “bits”, and “processed” meaning processed using the microprocessor. Non-symbolic input would be anything else, such as heat or shock. Then there’s explicit input—the input you knowingly provide—and implicit input: the input that influences the system without your knowledge or intent. Once you set an option, that option becomes implicit input for the function that refers to it. But in the case of the clock, there’s no explicit input. The implicit input is the previous state of the clock. Non-symbolic input includes the temperature of the room, voltage level, air pressure perhaps. But the tester does not control those in my example. So, can you test it?
Michael: Implicit input would also include the programming or design of the clock, yes? The engineering of the clock; its manufacture, its intended purpose?
James: Well, the programming is the structure of the product. It’s “input” for the microprocessor, but we aren’t testing that, are we? I don’t think those are inputs. The concept of input separates it from function and structure.
Michael: How do we know that we’re NOT testing it?
James: What do you mean?
Michael: Think of the Notepad exercise that we do in our course. Most people observe a bug in Notepad. They don’t realize that they’re testing Windows, which is where the bug actually is; Notepad calls that Windows function, and apparently few other products do. The people think they’re testing Notepad, and they are, of course. But they’re testing not only Notepad, but also the system in which Notepad lives.
James: Relate that to the clock, please. The clock is sealed. Can it be tested? You can see it run. Can you test it?
Michael: Well, you mentioned implicit input as the input that influences the system without your knowledge or intent. You can make inferences about it. You can observe it. You can evaluate the behaviour of the clock with respect to its implicit inputs. That’s testing, in my view.
James: You aren’t providing any input, and yet you are testing.
Michael: Input is there.
James: Can we imagine a system without input?
Michael: Yes; we’d call it a closed system.
James: Yes, I tried to construct one, with the clock.
Michael: And in real life, there aren’t any.
James: Not so. The clock is a closed system.
Michael: As soon as you put me in there, it’s not closed any more.
James: We’re not testing you. We’re testing the clock.
Michael: I’m involved.
James: No, you’re not.
Michael: Oh. I thought I was testing it.
James: You’re the tester, being asked to test the clock. You aren’t the product. The product is the clock. Can we test the clock? Furthermore, the clock has no implicit input.
Michael: Nope. But we’re not really testing only the product. We’re testing how the product relates to something (or more typically someone, or their values).
James: Of course. But don’t get that confused with input.
Michael: The clock absolutely has implicit input, unless we infer that God popped it into existence. In which case, the implicit input is God’s will.
James: You are not providing input to the clock.
Michael: I am not providing explicit input.
James: There is no implicit input either.
Michael: How did it get there, then?
James: “Getting there” is not input. That’s construction.
James: You can’t just make things up, my friend. It helps to have definite ideas about what words mean.
Michael: Somebody made up the clock. Someone designed it. Someone wound it.
James: Yes, I’m talking about the word “input”.
Michael: Was winding the clock not input?
James: It’s an electronic clock with no internal data or state.
Michael: If it has no internal data or state, it’s gonna have a tough job keeping time.
James: It’s a set of instructions that run. It places data in memory but does not ever accept data or look at data. It’s a very long set of instructions, not decisions; a straight line program that places one value after another in memory. Pre-programmed.
Michael: There’s no loop?
James: No loop. It’s a fixed set of instructions that run in a straight line for, say, one year before the instructions run out.
Michael: It seems to me that one test would be to watch it for a year. Plus, I don’t know in advance what non-symbolic and implicit input it might take.
James: It takes non-symbolic input, in the form of temperature, etc. Let’s say those are fixed and out of your control and it takes no symbolic input at all, explicit or implicit.
Of course the program is input to the microprocessor, but we aren’t testing the microprocessor, as such. My point is: even if it is true that no symbolic input (the classic sort of computer input) is taken, and even if no explicit or implicit input happens (and certainly no explicit input) we can still test it. We agree that we can test it, even if you admit all my strange conditions. We can test it by watching it and evaluating it against our expectations, or against another clock. Now: do we have to plan the test in advance for it to be a test? Or can we concoct it as we go?
Michael: I’d say that at the moment we have a question, or an observation for which we can back-generate a question, we have a test.
James: What about expectation? Do we need that?
Michael: “I wonder if it will do that?” and “I wonder why it did that?” are testing questions, in my view.
James: Can either of those questions result in a bug found? I’m doubtful on that point.
Michael: Yes, absolutely. “Hey… the hands fell off the clock. I wonder why it did that?” I often find that I generate an expectation after the event.
James: So you do have an expectation. Or is it that you have an “expectation generator?”
Michael: I’m not sure if people would call “an expectation of which I became conscious later” an “expectation”.
James: By definition, it is an expecation. You say that it is. Why wouldn’t someone call that an expectation?
Michael: I observe that people use “expectation” in a couple of different ways, at least. One is in advance of an observation. The other is after the fact, often expressed in terms of a surprise. “I didn’t expect that!” “What DID you expect?” “Uh… I don’t know, but I didn’t expect that! I expected no penguin to suddenly appear on top of the television set.”
The problem that I see with that relates to the business of a test having to have an expected, predicted result. If we lock on to that, as long as the calculator says something like “4″ after “2 + 2 =”, we can justify (or rationalize) missing too many bugs, in my view. Performance problems. Excessive precision, or imprecision. Usability problems. Reliability problems—what answer would the calculator give if we tried that test again? Might it give the answer “4″ no matter what input we provide?
James: So “expected” can be conflated with “predicted” and that’s bad. That’s limiting?
Michael: I think so.
James: Me too. But if we have a real-time expectation generator, perhaps we could call that…what’s the word?… oracle. At least, “oracle” covers it.
Michael: I’ll suggest that the oracle can be developed and applied retrospectively. “Oh, that IS a problem. That WAS a problem.”
James: That’s different. Let me suggest a hierarchy. 1) A prediction. 2) An evaluation on the fly. 3) A retrospective evaluation at any later point in time. The first one is the classic “expectation”. The first two are the classic oracle that defines a test. The third can turn anything that wasn’t a test into a test retrospectively. A week later, a tour that I made can become a test that I ran, if I become aware of an expectation that applies to that memory of what happened.
Michael:Yes. When I use a new product, I’m testing it. I start off optimistic, and after three weeks of frustration, I can say that the test failed, even though I didn’t set out to test. I might have to rely on memory, or go dumpster-diving for data, or try to recreate the test that I now realize was a test. But it was a test.
James: Okay. So, applying this to the original question. “A test consists of at least an input and an expectation.” Want to try to rephrase that?
Michael: Okay, let me try. “A test consists of at least an input (which may be explicit or implicit, symbolic or non-symbolic) and an evaluation linked to an observation (where the evaluation may have been predicted, generated at the same time as the observation, or applied retrospectively) by an oracle.”
James: That doesn’t fit the clock example, which involves no input provided by the tester.
Michael: Okay, so..”which may be explicit or implicit, symbolic or non-symbolic, and which may or may not be provided by the tester…”
James: Do you really think that a test consists of an input?
James: We agree that some sort of input, in some sort of extremely broad sense is there, but just because it’s there, does that mean the test consists of it?
Michael: Yeah, you’re right. “Consists” seems like the wrong word in that light. Maybe a test is an observation—or a set of observations—over time, where the input is optional.
James: When we took away the explicit and implicit symbolic input, it still seemed obvious that we could test. Could we observe a static system and be testing it, or does the system have to be operating?
Michael: Yes, I agree we could observe a static system and be testing it.
James: So, you could stare at a CD and be testing the video game that’s stored there. I think that’s more commonly called inspection or review.
Michael: Yes. And as you and I once argued, it doesn’t matter if we call it inspection or review, it’s still testing. I lost that argument. You won it when you made that point: review and inspection are testing too. Questioning something in order to evaluate it.
James: Testing as commonly understood, I think, involves putting something through its paces. Exercising it.
Michael: Yes. I’m more careful these days to call that test execution.
James: But inspection might be part of the testing process. To avoid confusion, I no longer say “static testing” and “dynamic testing”. I say “inspection” and “testing”. To me, test execution means performing a test. Although some of my “testing” involves inspection, if my intent is to evaluate without having the product perform its function, I call that inspection outright.
Michael: We can test an idea too. When I’m reviewing or inspecting or performing a test, what I’m putting through the paces is my model of the object under test. That is, sometimes we’re not testing the object in and of itself. We’re testing the relationship between the object and our model of it. (By object, I mean the target of our investigation, which may be tangible or executable or neither.)
James: But we test an idea by putting it through its paces.
Michael: We might call that “transpection”, rather than inspection or testing!
James: I don’t want to get hung up on this too much. I’m just saying that my preference is to apply the word “testing” mainly to what I once would have called “dynamic testing”, even though the thought process applies to static things as well.
Michael: Yes. For the same reason, I don’t want to get hung up on the words that people use for “testing” and “checking”, as long as they’re conscious of the distinction. I’d like people to be alert to the possibilities available in ideas that are lumped into single words—and vice versa. Words and ideas are many-to-many, many-to-one, and one-to-many, but one-to-one correspondences are pretty rare. As soon as we come up with one, someone creative will come along and use it as a metaphor!
James: Given that, I want to say what my take on a “test” is, now. A test consists of two things: coverage and oracle; not “input and expectation”, but coverage and oracle. Coverage means an observation of some aspect of the product in action. Oracle means a principle or mechanism by which we recognize a problem. Applying to the clock example: Yes, we can obviously test the clock. We can observe it working, and we can have an oracle, such as a second clock.
Michael:I’d also like to point out the importance of emotional oracles, like surprise or confusion or frustration or amusement. Those are the kinds of oracles that don’t suggest a right answer on their own, but which definitely point the to possibility of a problem for some person. Those emotional reactions don’t tell us what the problem is, necesssarily, but they are signals that tell us to look into something more deeply. If an oracle is a heuristic principle or mechanism by which we recognize a problem. then emotional reactions definitely qualify as oracles.
James: Yes. I think the IEEE concept of “input and expectation” is a special case created by a not-very-imaginative test manager. “Input” leads to coverage; “expectation” is one expression of an oracle. “Input” implies control by the tester, but control is not strictly necessary to test.
Michael:Yes—to test: questioning a product in order to evaluate it, or gathering information to inform a decision.
James: Obviously, we don’t have control over most of what a product does, and yet we test it. Much of what the product does is invisible to us, and yet we test it.
Michael:That’s really important, isn’t it? When you think about the complexity of even a simply program and the system that it interacts with, there’s really very little that’s within our control, or within our capacity to observe it.
James: True, but we can get leverage on that. There’s also a third thing that we talk about in our class. procedures. You need to know how to do the test; you need to observe something about the product as it runs (perhaps controlling it as it runs, but perhaps not); and you need to be able to recognize a problem if it occurs. Arguably the latter two things imply the first.
Michael: So, with the clock experiment or with any other test, we’re following what we whimsically call the Universal Test Procedure 2.0.
1. Model the test space.
2. Determine oracles.
3. Determine coverage.
4. Identify Test Procedures.
Those things are test design, “knowing how to do the test”, as you say. Then we
5. Configure the product.
6. Operate the product.
That’s your “perhaps controlling it” part. In the case of the clock, configuration was already done, and operation is ongoing.
7. Observe the product
8. Evaluate the produc.
When we link the observation in (7) with the oracles in (2), that’s your “recognizing a problem if it occurs”, and that the basis for evaluation. And in a real testing situation, we’d also
9. Apply a stopping heuristic
10. Report on what we’ve found.
James: Yeah, that’s an expansion of the ideas.
Michael: That’s a lot richer than “input and expected result”, isn’t it? Plus we’ve now got some ideas on implicit and explicit inputs, and symbolic and non-symbolic inputs. A very productive transpection session!