Blog: A Transpection Session: Inputs and Expected Results

A transpection is a dialog for learning. James Bach describes it here. Transpection is a technique we use a lot to refine ideas for presentations, for articles, for our course, or for our own understanding. Sometimes it’s all of them put together. Transpective sessions with James have led me sharpen ideas and to do work of which I’m very proud—on test coverage, for example (articles here, here, and here). Sometimes the conversation happens in speech (as in this dialog on what scripts tell us, or this one between James and Mike Kelly titled “Is There A Problem Here?” Sometimes the conversation happens in an instant message system like Skype. The former is more dynamic; the latter leaves a written transcript, which can be handy.

A few months back, James Bach and I took some time for a transpection session. Initially, the goal was to provide a demonstration of transpection for a particular student of James’, but then we realized that not only the session but its content might be of more general interest. James started the conversation.

James: Consider the definition: “A test consists of at least an input and an expectation.. Is that true? What does it mean?

Michael: Let’s apply your three-word heuristic for critical thinking: “Huh? Really? So?” :)

James: Can we go deeper with it?

Michael: Sure. Would sitting and observing a system that is already doing something (or not) be considered an “input”?

James: What do you think?

Michael: My first approach is to look at statements like the one you provide above and ask how broadly we could interpret them. I also try to falsify them. So… One could falsify it by asking, “Can I think of a case in which I don’t provide input, and it’s still a test. Where I don’t have an expectation, and it’s still a test?”

James: Okay.

Michael: One approach to answering to that question would be to ask “Well, what’s an input? What’s an expectation?”

James: So, let’s imagine an acrylic box. Inside the box runs a clock. You can see the clock face. You can’t interact with the clock (except to look at it in normal light). Can the clock be tested?

Michael: I can certainly “ask questions in order to evaluate it” (that’s the essence of your definition of testing), or “gather information with the intention of informing a decision” (which is Jerry Weinberg’s). I can conjecture. I can observe.

James: But can you provide an input? And can you test it?

Michael: In one sense, yes. I provide input by observing it. I don’t know if other people would be willing to take such a broad interpretation of input, but I could do that for the purpose of the exercise.

James: That seems like an odd definition of input. I don’t get it. It doesn’t process you looking at it. It doesn’t react to you looking at it.

Michael: It does seem like an odd defintion. And yet…I put myself into the system of the clock. So what do YOU mean by “input”?

James: I will be happy to tell you. But first can you define what input means to you? What would be a reasonable definition?

Michael: In computer and testing talk, it usually means to provide data to a function for processing.

James: I agree. What does “provide data” mean?

Michael: It could mean to enter a sequence of keystrokes at the keyboard; to move a mouse and click on an element of the screen; to connect the computer to some stream of network traffic. To feed the machine with something to process. It more generally means, I think, to alter the current functioning of the system in some way.

James: To exert control over the system?

Michael: Very broadly, yes. Again, I’m not sure that some would accept such a general defintion, but I would.

James: I think we can divide input into symbolic and non-symbolic input and explicit and implicit input. Symbolic input is data processed by the computer; data meaning “bits”, and “processed” meaning processed using the microprocessor. Non-symbolic input would be anything else, such as heat or shock. Then there’s explicit input—the input you knowingly provide—and implicit input: the input that influences the system without your knowledge or intent. Once you set an option, that option becomes implicit input for the function that refers to it. But in the case of the clock, there’s no explicit input. The implicit input is the previous state of the clock. Non-symbolic input includes the temperature of the room, voltage level, air pressure perhaps. But the tester does not control those in my example. So, can you test it?

Michael: Implicit input would also include the programming or design of the clock, yes? The engineering of the clock; its manufacture, its intended purpose?

James: Well, the programming is the structure of the product. It’s “input” for the microprocessor, but we aren’t testing that, are we? I don’t think those are inputs. The concept of input separates it from function and structure.

Michael: How do we know that we’re NOT testing it?

James: What do you mean?

Michael: Think of the Notepad exercise that we do in our course. Most people observe a bug in Notepad. They don’t realize that they’re testing Windows, which is where the bug actually is; Notepad calls that Windows function, and apparently few other products do. The people think they’re testing Notepad, and they are, of course. But they’re testing not only Notepad, but also the system in which Notepad lives.

James: Relate that to the clock, please. The clock is sealed. Can it be tested? You can see it run. Can you test it?

Michael: Well, you mentioned implicit input as the input that influences the system without your knowledge or intent. You can make inferences about it. You can observe it. You can evaluate the behaviour of the clock with respect to its implicit inputs. That’s testing, in my view.

James: You aren’t providing any input, and yet you are testing.

Michael: Input is there.

James: Can we imagine a system without input?

Michael: Yes; we’d call it a closed system.

James: Yes, I tried to construct one, with the clock.

Michael: And in real life, there aren’t any.

James: Not so. The clock is a closed system.

Michael: As soon as you put me in there, it’s not closed any more.

James: We’re not testing you. We’re testing the clock.

Michael: I’m involved.

James: No, you’re not.

Michael: Oh. I thought I was testing it.

James: You’re the tester, being asked to test the clock. You aren’t the product. The product is the clock. Can we test the clock? Furthermore, the clock has no implicit input.

Michael: Nope. But we’re not really testing only the product. We’re testing how the product relates to something (or more typically someone, or their values).

James: Of course. But don’t get that confused with input.

Michael: The clock absolutely has implicit input, unless we infer that God popped it into existence. In which case, the implicit input is God’s will.

James: You are not providing input to the clock.

Michael: I am not providing explicit input.

James: There is no implicit input either.

Michael: How did it get there, then?

James: “Getting there” is not input. That’s construction.

Michael: Bah. :)

James: You can’t just make things up, my friend. It helps to have definite ideas about what words mean.

Michael: Somebody made up the clock. Someone designed it. Someone wound it.

James: Yes, I’m talking about the word “input”.

Michael: Was winding the clock not input?

James: It’s an electronic clock with no internal data or state.

Michael: If it has no internal data or state, it’s gonna have a tough job keeping time.

James: It’s a set of instructions that run. It places data in memory but does not ever accept data or look at data. It’s a very long set of instructions, not decisions; a straight line program that places one value after another in memory. Pre-programmed.

Michael: There’s no loop?

James: No loop. It’s a fixed set of instructions that run in a straight line for, say, one year before the instructions run out.

Michael: It seems to me that one test would be to watch it for a year. Plus, I don’t know in advance what non-symbolic and implicit input it might take.

James: It takes non-symbolic input, in the form of temperature, etc. Let’s say those are fixed and out of your control and it takes no symbolic input at all, explicit or implicit.

Of course the program is input to the microprocessor, but we aren’t testing the microprocessor, as such. My point is: even if it is true that no symbolic input (the classic sort of computer input) is taken, and even if no explicit or implicit input happens (and certainly no explicit input) we can still test it. We agree that we can test it, even if you admit all my strange conditions. We can test it by watching it and evaluating it against our expectations, or against another clock. Now: do we have to plan the test in advance for it to be a test? Or can we concoct it as we go?

Michael: I’d say that at the moment we have a question, or an observation for which we can back-generate a question, we have a test.

James: What about expectation? Do we need that?

Michael: “I wonder if it will do that?” and “I wonder why it did that?” are testing questions, in my view.

James: Can either of those questions result in a bug found? I’m doubtful on that point.

Michael: Yes, absolutely. “Hey… the hands fell off the clock. I wonder why it did that?” I often find that I generate an expectation after the event.

James: So you do have an expectation. Or is it that you have an “expectation generator?”

Michael: I’m not sure if people would call “an expectation of which I became conscious later” an “expectation”.

James: By definition, it is an expecation. You say that it is. Why wouldn’t someone call that an expectation?

Michael: I observe that people use “expectation” in a couple of different ways, at least. One is in advance of an observation. The other is after the fact, often expressed in terms of a surprise. “I didn’t expect that!” “What DID you expect?” “Uh… I don’t know, but I didn’t expect that! I expected no penguin to suddenly appear on top of the television set.”

The problem that I see with that relates to the business of a test having to have an expected, predicted result. If we lock on to that, as long as the calculator says something like “4″ after “2 + 2 =”, we can justify (or rationalize) missing too many bugs, in my view. Performance problems. Excessive precision, or imprecision. Usability problems. Reliability problems—what answer would the calculator give if we tried that test again? Might it give the answer “4″ no matter what input we provide?

James: So “expected” can be conflated with “predicted” and that’s bad. That’s limiting?

Michael: I think so.

James: Me too. But if we have a real-time expectation generator, perhaps we could call that…what’s the word?… oracle. At least, “oracle” covers it.

Michael: I’ll suggest that the oracle can be developed and applied retrospectively. “Oh, that IS a problem. That WAS a problem.”

James: That’s different. Let me suggest a hierarchy. 1) A prediction. 2) An evaluation on the fly. 3) A retrospective evaluation at any later point in time. The first one is the classic “expectation”. The first two are the classic oracle that defines a test. The third can turn anything that wasn’t a test into a test retrospectively. A week later, a tour that I made can become a test that I ran, if I become aware of an expectation that applies to that memory of what happened.

Michael:Yes. When I use a new product, I’m testing it. I start off optimistic, and after three weeks of frustration, I can say that the test failed, even though I didn’t set out to test. I might have to rely on memory, or go dumpster-diving for data, or try to recreate the test that I now realize was a test. But it was a test.

James: Okay. So, applying this to the original question. “A test consists of at least an input and an expectation.” Want to try to rephrase that?

Michael: Okay, let me try. “A test consists of at least an input (which may be explicit or implicit, symbolic or non-symbolic) and an evaluation linked to an observation (where the evaluation may have been predicted, generated at the same time as the observation, or applied retrospectively) by an oracle.”

James: That doesn’t fit the clock example, which involves no input provided by the tester.

Michael: Okay, so..”which may be explicit or implicit, symbolic or non-symbolic, and which may or may not be provided by the tester…”

James: Do you really think that a test consists of an input?

Michael: Hmm…

James: We agree that some sort of input, in some sort of extremely broad sense is there, but just because it’s there, does that mean the test consists of it?

Michael: Yeah, you’re right. “Consists” seems like the wrong word in that light. Maybe a test is an observation—or a set of observations—over time, where the input is optional.

James: When we took away the explicit and implicit symbolic input, it still seemed obvious that we could test. Could we observe a static system and be testing it, or does the system have to be operating?

Michael: Yes, I agree we could observe a static system and be testing it.

James: So, you could stare at a CD and be testing the video game that’s stored there. I think that’s more commonly called inspection or review.

Michael: Yes. And as you and I once argued, it doesn’t matter if we call it inspection or review, it’s still testing. I lost that argument. You won it when you made that point: review and inspection are testing too. Questioning something in order to evaluate it.

James: Testing as commonly understood, I think, involves putting something through its paces. Exercising it.

Michael: Yes. I’m more careful these days to call that test execution.

James: But inspection might be part of the testing process. To avoid confusion, I no longer say “static testing” and “dynamic testing”. I say “inspection” and “testing”. To me, test execution means performing a test. Although some of my “testing” involves inspection, if my intent is to evaluate without having the product perform its function, I call that inspection outright.

Michael: We can test an idea too. When I’m reviewing or inspecting or performing a test, what I’m putting through the paces is my model of the object under test. That is, sometimes we’re not testing the object in and of itself. We’re testing the relationship between the object and our model of it. (By object, I mean the target of our investigation, which may be tangible or executable or neither.)

James: But we test an idea by putting it through its paces.

Michael: We might call that “transpection”, rather than inspection or testing!

James: I don’t want to get hung up on this too much. I’m just saying that my preference is to apply the word “testing” mainly to what I once would have called “dynamic testing”, even though the thought process applies to static things as well.

Michael: Yes. For the same reason, I don’t want to get hung up on the words that people use for “testing” and “checking”, as long as they’re conscious of the distinction. I’d like people to be alert to the possibilities available in ideas that are lumped into single words—and vice versa. Words and ideas are many-to-many, many-to-one, and one-to-many, but one-to-one correspondences are pretty rare. As soon as we come up with one, someone creative will come along and use it as a metaphor!

James: Given that, I want to say what my take on a “test” is, now. A test consists of two things: coverage and oracle; not “input and expectation”, but coverage and oracle. Coverage means an observation of some aspect of the product in action. Oracle means a principle or mechanism by which we recognize a problem. Applying to the clock example:  Yes, we can obviously test the clock. We can observe it working, and we can have an oracle, such as a second clock.

Michael:I’d also like to point out the importance of emotional oracles, like surprise or confusion or frustration or amusement. Those are the kinds of oracles that don’t suggest a right answer on their own, but which definitely point the to possibility of a problem for some person. Those emotional reactions don’t tell us what the problem is, necesssarily, but they are signals that tell us to look into something more deeply. If an oracle is a heuristic principle or mechanism by which we recognize a problem. then emotional reactions definitely qualify as oracles.

James: Yes. I think the IEEE concept of “input and expectation” is a special case created by a not-very-imaginative test manager. “Input” leads to coverage; “expectation” is one expression of an oracle. “Input” implies control by the tester, but control is not strictly necessary to test.

Michael:Yes—to test:  questioning a product in order to evaluate it, or gathering information to inform a decision.

James: Obviously, we don’t have control over most of what a product does, and yet we test it. Much of what the product does is invisible to us, and yet we test it.

Michael:That’s really important, isn’t it? When you think about the complexity of even a simply program and the system that it interacts with, there’s really very little that’s within our control, or within our capacity to observe it.

James: True, but we can get leverage on that. There’s also a third thing that we talk about in our class. procedures. You need to know how to do the test; you need to observe something about the product as it runs (perhaps controlling it as it runs, but perhaps not); and you need to be able to recognize a problem if it occurs. Arguably the latter two things imply the first.

Michael: So, with the clock experiment or with any other test, we’re following what we whimsically call the Universal Test Procedure 2.0.

1. Model the test space.
2. Determine oracles.
3. Determine coverage.
4. Identify Test Procedures.

Those things are test design, “knowing how to do the test”, as you say.  Then we

5. Configure the product.
6. Operate the product.

That’s your “perhaps controlling it” part. In the case of the clock, configuration was already done, and operation is ongoing.

7. Observe the product
8. Evaluate the produc.

When we link the observation in (7) with the oracles in (2), that’s your “recognizing a problem if it occurs”, and that the basis for evaluation. And in a real testing situation, we’d also

9. Apply a stopping heuristic
10. Report on what we’ve found.

James: Yeah, that’s an expansion of the ideas.

Michael: That’s a lot richer than “input and expected result”, isn’t it? Plus we’ve now got some ideas on implicit and explicit inputs, and symbolic and non-symbolic inputs. A very productive transpection session!

14 Responses to “A Transpection Session: Inputs and Expected Results”

  1. An excellent transpection session. Thanks for sharing this with us!

  2. Simon Morley says:

    I enjoyed that!
    It was giving me “last days of Socrates” flashbacks – which is not a bad thing btw!

    I think ideas around observation get underrated and how crucial it is to avoid testing with your “eyes wide shut” (my whimsical view!)

    More transcripts of transpections please!

  3. Even my collegue and me got into a very small (verbal)transpection session this morning. The outcome:

    Isn’t the above also known as the definition of an “observation” (instead of a definition of “testing”)?

    1. Asking a question about a natural phenomenon
    2. Making observations of the phenomenon
    3. Hypothesizing an explanation for the phenomenon
    4. Predicting a logical consequence of the hypothesis
    5. Testing the hypothesis by an experiment, an observational study, or a field study
    6. Creating a conclusion with data gathered in the experiment

    I’m not sure that this is a typical notion of “observation”. I’d call it the experimental or scientific method. But that may be a language issue, rather than a meaning issue.

  4. Erik Breuker says:

    How do you know it is a clock?
    If you start assuming it is a clock, you might ignore some observations.

    Well, we’re told from the outset that it’s a clock. It’s reasonable to start with some assumptions. Testing tests not only the product, but the assumptions that we have about it. But you’re right to point out the risk in making assumptions. Thanks for that.

  5. Thanks Michael for sharing this with us.
    I read this article now for the 3rd time today. When also reading between the lines it is an impressive piece.

    I see here several lessons for me not in a particular order.
    1. a basic teaching about prediction and testing
    2. a lesson about a process how James and you started with a general statement and James triggered you to think and learn in such a manner you are able to provide direction to the discussion
    3. a lesson about a process where James shows respect to your way of thinking and provide guidance to form you mind
    4. a lesson how I would respond on the same questions and if I can align with the answers provided
    5. a lesson that I have to learn and use this way of thinking, not copying the questions and answers, instead getting the
    6. the power of transcripts. I already learnt something for myself when reading back transcripts of weekendtesting sessions I attended and especially those I didn’t attend.
    7. A lesson where “you” in this transpection could be “me” and that I have to learn to make the change happen
    8. a lesson how to formulate questions with your own values in mind open for discussion
    9. a lesson that there is room for multiple discussions and understanding. As I see it:
    - you started with showing an example using the meaning of input
    - you guided us in the example what testing is
    - finished the example and continued hardly noticed with discussing what you have done and came to a new word: transpection, while doing this adding other values to it like reviews and inspection is also testing.
    - come to another mutual understanding also about the meaning of input
    - provided a usable solution were the transcript provides a very good picture of the background of the creation of it
    10. There are more lessons to learn when reading it, and even more when practicing it.

    Perhaps I did more then your expectations were with this piece and do I see more lessons then the intentions were. For me this piece is valuable.

    Thanks again!
    Jeroen

  6. Markus says:

    Michael,
    I liked the analysis from the starting point to the conclusion, but could you provide a definition for “Transpection”?

    Regards
    Markus

    It’s in the top line of the post, but to save you scrolling up, here it is again. Cheers!

  7. [...] A Transpection Session: Inputs and Expected Results [...]

  8. I would have to say it is not only possible to test the clock-in-the-box but actually necessary.

    I see it as an exercise when you have to test part of a system which you have no control over.

    For example I’ve had problems with integration to the third party systems that gave absolute nonsense errors about things nobody could think of at that time and it messed up the correct behavior of the primary system pretty badly. We could do nothing but to observe what happened. Almost no possible way to change input data by end user. It either happened or not. But it ended up as very useful experience about testing.

    I discussed your exercise with my colleague Rasmus and we found at least few ways to test it without giving it direct input itself
    1) Expectations – f.e: What format does it show time? Is it understandable?
    2) End-values – turnover of seconds/minutes/hours where f.e 59 -> 00
    3) Load testing – how much does it starts to lie in 10 seconds, 1 minute, 1 hour, 1 day, 1 month, 1 year etc compared to let’s say quantum clock or NIST-F1.
    4) What time zone time is it showing? Can be tricky because look at India’s time zone for example.
    5) How long does the battery last before it shuts down? or before it starts to “lie”? How rapidly does it start to lie when batteries are running lower?
    6) How are the digits shown? are they visible via any other angle? Are they too small or too big?

    And few ways to have direct input without moving or touching the box itself
    1) Put powerful-enough magnet next to the box to see what happens.
    2) Set EMP-bomb off near the box to see what happens.

    With best regards
    Oliver V.

  9. Veretax says:

    A great little discussion, however i disagree with James’s Assertion that the clock is a closed system.

    It’s a general systems thinking skill to consider what might be inside or outside the system in question, and what the boundaries might be. We want to simplify things without oversimplifying them, and we want to make them sufficiently complex so that we are likely to broaden our understanding. So you’re right to question that.

    But don’t worry; both James and I are aware that there ain’t no such thing as a closed system in any ultimate sense. A “system” isn’t some absolute thing but (once again) a relationship between some person(s) and something being studied. A system is, to us, “a set of things in meaningful relationship.” When we talk about closed systems, they’re only closed in the sense that we’ve agreed not to think of things that are outside of them with respect to the meanings and relationships that we’re questioning. We “close” a system to the degree that we want to simplify it, to the degree we’re actively ignoring things that we don’t deem important or relevant. Poking at those premises, as you’ve done here, can be really valuable.

    Whether it is battery operated, windup, or electric, if let alone long enough it will stop running absent user intervention. That intervention could involve setting the hours, minutes, PM/AM, Alarm, etc. It could be installing or replacing a battery, plugging into a working socket, or winding it. In my experience with Clocks if you pull it out of the box and just set it on the table they typically just sit there. (which is a bit different from most watches which come with a battery in them typically today.)

    Taking an old-fashioned wind-up watch out of the box sometimes induces motion into the flywheel, just enough to get things going again for a little while. Uri Geller took advantage of this, urging people to go to their jewellery boxes to observe that he had started the stopped clocks for them with his mind.

    But still a pretty solid discussion, thanks for sharing Michael!

    And thank you for your comment.

  10. Dwain says:

    Thank you Michael and James for continually pushing the testing community to realize our potential. I struggle everytime I have to create test cases with inputs, conditions, pre-conditions, and expected results. These always seem to be constructed from requirements that will never be 100% accurate due to a variety of dynamic elements and are only a ‘point-in-time’ capture of what we think the requirement(s)should be. We don’t know what we don’t know.

    Testing, to me, is the observation of the product under test and providing my comments of how it interacts/communicates with me and whether it conforms to how I expected it to act/react. During this observation, I may provide input or not, depeding on what I observe. By observing, I am interacting with the product and I am processing what it means to me. I then inform the stakeholders of my experiences and observations and they ultimately decide if that fits their mental and business models.

    I prefer to call it the “EX” factors of testing; a blend of Exploratory and User Experience methods.

    Thanks, Dwain, for your comment. Sounds like the kind of testing that I can believe in.

    In addition to that, you might like to investigate some of Cem Kaner’s work on scenario testing, to increase your capacity to notice problems that matter to your clients (and their clients). The goal here is to develop tests that can help you and them understand what the product means to them, as well as to you.

    All the best!

  11. Petteri Lyytinen says:

    Reading through that session was really thought-inspiring, so thank you for putting it up in your blog. Here’s some of my thoughts on the post:

    About input:

    This may be a crazy idea, but reading your dialog about closed systems and observations immediately reminded me of the classic Schrödinger’s cat paradox. I can easily come up with several questions related to the system and its setup and there’s only two (realistically) possible outcomes but there’s no way of telling which one to expect – except afterwards. Neither of the outcomes would be surprising, considering the scenario. So, I have a bunch of relevant questions and a limited set of expectations. Does that constitute a test?

    About expectations and predictions:

    I completely agree with you saying that “predicted” in relation to expectations can be a bad thing, especially for a tester, as I see it could narrow down your field of vision and a tester needs to be open-minded and imaginative in order to really be effective. This reminds me of something I read in the book “Sophie’s World” (by Jostein Gaarder), related to David Hume’s theory of causation:

    The idea is that you strike a black billiard ball so that it hits a white ball and then you observe what happens. Most everyone would expect that when the black ball hits the white one, it starts moving. However, Hume claimed that it was only the force of habit that made us see a causal link behind this. “We cannot perceive the black billiard ball as being the cause of the white ball’s movement. Therefore, we cannot prove that the black billiard ball will always set the white one in motion” ie. the expectation that one event follows another is not in the objects themselves, but in our mind.

    Another related thought that just occurred to me is a quote from the science-fiction animation “Ghost in the Shell” (they’re talking about martial arts but I think the same principle can easily be applied here): “Over-specialize and you breed in weakness”. In the movie this refers to becoming predictable, which could be used against you in a combat situation. In the context of testing, I see the risk as becoming too narrow-minded – even if it means you’re really good in some very specific area – which could prove harmful to your effectiveness as a tester, overall.

  12. While attending to your class, and talking about this test, i was pondering about the following:

    James said the following:
    You can’t interact with the clock (except to look at it in normal light). Can the clock be tested?

    I was wondering, would a test require input to the object one is observing, or could the observer also recieve input, as part of the test? Is the ‘input’ (as mentioned in the definition “A test consists of at least an input and an expectation” relevant to the object we are observing or could it also be the observer itself?

    To me, the part between the curly brackets seems to be very essential. Lets say the input could also be input for the observer. James mentioned “looking“. In that case maybe we do have input: the lightwaves. They are reflecting from the object into one of our senses: our eyes.

  13. Michael Kuth says:

    As someone who has recently transitioned to a software testing role and is still developing my underlying testing thought process I loved reading this discussion of what constitutes a test.

    I do however disagree with one of James’ statements that limited the scenario.

    James: It takes non-symbolic input, in the form of
    temperature, etc. Let’s say those are fixed and out of
    your control and it takes no symbolic input at all,
    explicit or implicit.

    I would say that you have missed one implicit non-symbolic input that is an input to every test so it is rarely thought of as such: time. The clock must take time as an input otherwise it is, by definition, not a clock. It does not matter whether it achieves (or attempts to achieve) the function of a clock with a loop or a series of sequential instructions. If after each second, minute, hour, week, month or year it does not inform the user that the unit of time has passed it is not a clock.

    Michael replies: Hmmm. You could think of time as a non-symbolic input. On the other hand, is it input if you can’t vary or control it in any way? Maybe. My answer at this moment is that I might consider time as a factor in the test, rather than an input to it.

    As for thinking of something as a clock (or not), there are some interesting questions of ontology here. You say, “If after each second, minute, hour, week, month or year it does not inform the user that the unit of time has passed it is not a clock.” Is that true? There are plenty of digital clocks that don’t have second hands; thus they don’t inform the user that a second has passed (except for the one at the top of the minute). Such clocks typically don’t display the day, week, month, or year, either. Do these qualify as clocks? Does a bucket that drips out water at a regular rate constitute a clock? An egg timer, that drips out sand? Is Stonehenge a clock, even though it doesn’t display digits or hands? Meanwhile, can you think of anything that doesn’t take time as an input?

    I would argue that clock-ness depends on your relationship to the object. Recently I was listening to a lecture on the philosophy of science, in which the lecturer introduced the idea of variation in the forms of thermometers. One big issue in philosophy of science is whether our definitions will be too restrictive or too permissive. Some thermometers that display the temperature on a graduated scale contain mercury in a sealed glass container; others contain coloured water. Electronic thermometers contain no liquid and display the measurement digitally. Are they thermometers? Most people would probably agree.

    Is a birdbath a thermometer? Most people would probably disagree, at first. Several years ago, I visited Albuquerque in April. When I woke up one morning, I was impressed at how cold it the night before. How cold? There was ice in the birdbath, so it had evidently been cold enough to freeze the water. I used the birdbath as a way of measuring the temperature. So is a birdbath a thermometer, or not a thermometer? After hearning that story, maybe a few more people would agree that a birdbath might be a thermometer.

    Even when you consider testing a static system, such as staring at a CD to test the video game that’s stored there, using time as an input one could reasonably expect that for a certain period of time the CD does not change in any way (or conversely, that it should self destruct after 30 days to reduce rental return costs). Although the tester is not exerting control over the CD, time is and therefore the test has an input.

    James: We agree that some sort of input, in some sort
    of extremely broad sense is there, but just because
    it’s there, does that mean the test consists of it?

    Michael: Yeah, you’re right. “Consists” seems like the
    wrong word in that light. Maybe a test is an
    observation—or a set of observations—over time, where
    the input is optional.

    I would argue that at least one input is not optional. I would define a test as “an observation or set of observations over time, where other inputs are optional.” The beginning of the discussion against me would likely be, “Can you test a static system at a certain instance of time where the instance of time is irrelevant to the test?” How would a test be defined then?

    I’d be careful about relevance, since relevance is not an attribute of something (and certainly not of a test). Relevance is a relationship between some person and the object of what they’re considering; what’s relevant to you may not be relevant to me, and vice versa. You can demonstrate this, too: notice the reaction you get when you say to a programmer, “Well, it worked this time.”

    I like the way you’re anticipating and wresting with counter-arguments to your position. I’d call that good introspective work. Thanks for adding to the discussion.

  14. Griffin Jones says:

    Thank you both for sharing this. Listening to flow of the conversation is helpful to me.

    Two scenarios popped into my mind:

    a. Let’s imagine an acrylic box (very large). Inside the box runs a pair of neuron stars. You can see the stars. You can’t interact with the stars. Can the stars be tested. – I would answer “yes”, http://blogs.discovermagazine.com/80beats/2008/07/07/neutron-stars-prove-einstein-right-again/

    b. Let’s imagine an acrylic box. Inside the box is an audio/visual display of Mike playing his new testing game with James. You can see/hear everything. You can’t interact with Mike and James. Can you test Mike’s new game? I would answer “yes”.

Leave a Reply