People sometimes seem to struggle with a concept that’s central to testing, the concept of “oracle”. In the three-day Rapid Software Testing class, we define an oracle as
a principle or mechanism by which we recognize a problem.
Sometimes I like to emphasize that oracles are fallible and context-dependent. When that’s so, I say that an oracle is
a heuristic principle or mechanism by which we recognize a problem.
That means that an oracle can work but might fail in helping us to recognize a problem. A heuristic is something that helps us learn; it’s a fallible method for solving a problem or making a decision. So an oracle, as a special kind of heuristic, helps us to make a particular decision in answer to the question, “Is there a problem here?” If the answer is Yes, it’s because an oracle is enabling us to recognize a problem.
For some time, I’ve been surprised at how tricky a concept this seems to be for some people. Possibly, I thought, it was because people simply weren’t used to thinking about oracles; for many it’s a new word and a new concept. Maybe the problem was with “heuristic”, or “principle”, or “mechanism”. Recently, though, I began to wonder: perhaps the difficulty comes from the fact that people aren’t used to thinking deeply about problems. I tested that idea by asking some people, and I found that they struggled in describing how they thought of problems. So what is a problem, anyway?
Here are two ways of thinking about problems that I’ve found useful.
1) A problem is “a difference between what is perceived and what is desired.” (Dewey, J. (1933), How We Think: A Restatement of the Relation of Reflective Thinking to the Educative Process) I first learned this definition of “problem” in a session led by Don Gray at the Amplifying Your Effecctiveness Conference a few years ago, and I believe it shows up in several places in Jerry Weinberg‘s work.
2) A problem is “an undesirable situation that is significant to and maybe solvable by some agent, though probably with some difficulty.” (G.F Smith, “Towards a Heuristic Theory of Problem Structuring”, quoted in Weick, Karl E. Sensemaking in Organizations. Sage Publications, Inc, 1995.)
Both definitions, I think, point to the same thing: problems are based on desires, perceptions, and situations. But both definitions have a minor problem for me, which is that they do not make explicit something I think to be very important: desires, perceptions, and situations are centred around people and context. Problems are subject to the Relative Rule:
For any abstract X, X is X to some person, at some time.
People often talk of problems (bugs, defects, issues, and so forth) as though they were attributes of a product or of a situation. But problems aren’t attributes as such. They’re relationships between some person(s), the product, and the system that includes them, and the situation that encompasses it all. A problem is a problem for some person, at some time. Differences, perceptions, and desires are like that too; they’re relative. If I were to expand out Dewey’s notion of “problem”, it would look like this:
A problem is a difference (according to some person) between what is perceived (by some person) and what is desired (by some person) (all at some point in time).
There are several implications here.
- The problem might be in the difference, or in the perception, or in the desire.
- Different people will have different notions of differences, different perceptions, different desires.
- A problem may also be influenced by timing; something that’s a problem today might not be a problem tomorrow, and something that isn’t a problem today might be a catastrophe tomorrow.
- Since, as testers, we’re in the business of finding potential problems, we must be alert to the enormous variety of people who may be affected by the product and the project.
- Our mission as testers typically includes the task of identifying possible problems. As such, we should resist the temptation to dismiss something as “not a problem”, because…
- Something that we might dismiss as a trivial problem for some people might be a serious problem for other people and…
- Something that we might see as a serious problem in our perception might appear as a trivial problem for other people so…
- We’re obliged as testers to identify a possible problem in terms of its most serious consequences for people who might matter to the product owner. However…
- We’re not in the business of deciding whether something is a problem or not, so for a given problem, our testing clients or the project owners decide on its ultimate significance, and on their response to it.
So the bottom line is that problems are slippery. A problem is not a thing in a product or in the world, but a relationship between some situation and some person.
Smith’s emphasis on an ability to solve a problem may matter too. People often accept things that they perceive as beyond anyone’s ability to solve. For example, I can’t do anything about a meteor hitting my computer in the next few minutes, so I’m not going to treat the possibility that it might happen as a problem. Yet again, I should be careful to suppress any of my own prejudices that nothing can be done with respect to a given problem. That’s a decision for those who build and those who own the product, since they may have information and more resources of which I am unaware.
One of the fundamental questions of testing is “Is there a problem here?” Oracles are media, in the McLuhan sense. Media are tools, extensions of ourselves that enhance, enable, accelerate, or intensify our capacity to do things. McLuhan pointed out, though, that media are agnostic about what they extend. If our concept of “problem” is limited, our oracles will extend and accelerate our ability to recognize a limited set of problems. In other words, with too narrow an idea of what a problem might be, oracles may only help us to recognize too narrow a set of possible problems.
So: one key to understanding and applying oracles skilfully is to recognize the richness and diversity of what we might mean by problem. What we’re testing is not simply source code or a collection of compiled object code files. We’re testing products or services that are systems, sets of things in meaningful relationship to each other. Those systems are related to other systems. Products don’t stand on their own. Every product is part of a system that includes people who build it, people who support it, people who maintain it, people who buy it, people who use it directly, and people who are affected by it. That’s an incomplete list of people who might experience problems related to the product or system. Try asking these questions:
- What are the elements of system that I’m testing?
- Who might have a relationship to that system, or to its elements?
- Who else might have a relationship to it?
- What desires might they have that are related to the system?
- What aspects of the system might provide value to those people by fulfilling their desires? What might threaten that value by dashing their desires?
- How might people perceive the system? What might they see, hear, smell, touch, taste, or feel as they use it or interact with it?
- What might influence, magnify, sharpen, or distort their perceptions?
- What differences might they experience between their perceptions and their desires?
- Could someone, or something—some change in the situation—help them to solve or otherwise deal with such differences?
- What might change in the system over time? How might people’s perceptions, desires, or notions of differences change over time?
- What ideas, tools, or conversations might help me to recognize perceptions, desires, and differences?
Reflecting on these questions periodically, even briefly, may help you to expand your notion of what a problem is. That in turn may extend your ability to use oracles to recognize potentional problems in the system you’re testing.