Blog Posts from April, 2012

All Oracles Are Heuristic

Wednesday, April 25th, 2012

In which the conversation about heuristics and oracles continues…

“So what’s the difference,” I asked my tester friend Tony, “between an oracle and a heuristic?”

“Hmm. Well, I’ve read the Rapid Testing stuff, and you and James keep saying an oracle is a principle or mechanism by which we recognize a problem.

“Yes,” I said. “That’s what we call an oracle. What’s the difference between that and a heuristic?”

“An oracle helps us recognize a problem, but it’s not a method for solving a problem, or for making a decision.” He suddenly paused.

“Wait,” he said. “There’s that question you say testers should always be asking—Is there a problem here? An oracle does help us make a decision: it helps us to decide whether there’s a problem in the product we’re testing. And oracles can fail, too. So an oracle’s not different from a heuristic; an oracle is a heuristic. They’re the same.”

“Okay,” I said. “But that’s like saying ‘an iPhone isn’t different from a smartphone; an iPhone is a smartphone. They’re the same.'”

“But? But what? What’s the problem with that? Aren’t all iPhones smartphones?”

“Well, I’d say so,” I replied. “But let me ask you: are all smartphones iPhones?”

He paused for a second. “Oooh. Oracles are heuristic, but not all heuristics are oracles. An oracle is a heuristic, but it’s a specific kind of heuristic. Okay, let me see if I’ve got this: tossing a coin is a heuristic for making a decision. A heuristic approach for making a decision, I mean. You’d use the Coin Toss heuristic in some contexts—random decisions, or unimportant decisions, or… or intractable decisions, or decisions that you want to be fair. The approach can fail. It might not be a fair coin. Or it might be a high-stakes decision that shouldn’t be left to chance. So the Coin Toss heuristic might work, it can fail.”

“Right,” I said. “Tossing a coin is a heuristic approach for making a decision.”

“But it’s not an oracle,” Tony said, “because tossing a coin doesn’t help us to recognize a problem. So tossing a coin is a heuristic, but it’s not an oracle.”

“All right. What does an oracle do for us?”

Tony started confidently. “An oracle is something that gives us the right answer, so that we can compare it to the result the product gives us. If there’s a difference between the oracle’s answer and the product’s result, there’s a problem. If the product’s answer is the same as the oracle’s answer, then there’s no problem.”

“Are you sure about that?” I asked. “Is a specification an oracle?”

“Yes. The specification tells us how the product is supposed to behave.”

“And how reliable are the specifications where you work?”

Tony paused, and then he grinned. “Okay. They suck, to be honest with you,” he said. “They’re ambiguous. They’re unclear. They’re incomplete; they usually miss a bunch of requirements. They contradict each other, sometimes on the same page. So we have to talk about them a lot to clear them up—and then when we sort things out, the job of updating the written spec usually gets left for last, if it ever happens at all.”

“Still,” I said, “if you see an inconsistency between the spec and the product, you at least suspect a problem, don’t you?”

“Well, yeah. When the spec and the product disagree, there’s usually a problem somewhere—either with the product, or with the spec. Or both. When we’re not sure, the program manager is usually the one who clears things up. Sometimes the programmers fix the product. Sometimes the the product turns out to be right, and it’s the spec that’s wrong—but then we know at least the BA’s ought to fix the spec, even if they don’t get around to it right away.”

“So if you use a specification as an oracle, it’s somewhat reliable, but it’s not guaranteed to be right. What does that sound like?”

He paused again. “It’s a heuristic. An oracle is a special kind of heuristic. An oracle is a heuristic principle or mechanism by which we recognize a problem.

“That’s the way I like to say it these days, yes,” I replied. “For one thing, having the word ‘heuristic’ in the definition of ‘oracle’ seems to help people recognize that there’s some kind of distinction to be made between heuristics and oracles. But for another, I think it’s important to emphasize that oracles help us to learn things. And that, since they’re heuristics, oracles are fallible and context-dependent. No oracle comes with a guarantee that it’s giving you the right answer. An oracle can only point you to a possible problem.

Tony’s brow furrowed again.

To be continued…

Heuristics for Understanding Heuristics

Friday, April 20th, 2012

This conversation is fictitious, but it’s also representative of several chats that I’ve had with testers over the last few weeks.

Tony, a tester friend, approached me recently, and told me that he was having trouble understanding heuristics and oracles. I have a heuristic approach for solving the problem of people not understanding a word:

Give ’em a definition.

So, I told him:

A heuristic is a fallible method for solving a problem or making a decision.

After I tried the “Give ’em a definition” heuristic, I tested to see if Tony seemed to understand. His eyes were a little glazed over. I applied a heuristic for making the decision, did he get it?

When someone’s eyes glaze over, they don’t get it.

Heuristics aren’t guaranteed to work. For example, sometimes the general “Give ’em a definition” heuristic solves the problem of people not understanding something, and sometimes it doesn’t. In the latter case, I apply another heuristic:

Give ’em an explanation.

So I told him:

“When you know how to solve a problem, you might follow a rule. When you’re not so sure about how to solve the problem, following a rule won’t help you. Not knowing how to solve a problem means not knowing which rule to apply, or whether there’s a rule at all. When you’re in uncertain conditions, or dealing with imperfect or incomplete information, you apply heuristics—methods that might work, or that might fail.

“As an adjective, ‘heuristic’ means ‘serving to discover’ or ‘helping to learn’. When Archimedes realized that things that sink displace their volume of water, and things that float displace their mass, he ran naked through the streets of Athens yelling, ‘Eureka!’ or ‘I’ve discovered it!’ ‘Eureka’ and ‘heuristic’ come from the same root word in Greek.

Tony was listening thoughtfully, but his brow was still furrowed. So I applied another teaching heuristic:

Give ’em something to compare.

I said, “Here’s one way of understanding heuristics: compare ‘heuristic’ with ‘algorithm’. An algorithm is a method for solving a problem that’s guaranteed to have a right answer. So an algorithm is like a rule that you follow; a heuristic is like a rule of thumb that you apply. Rules of thumb usually work, but not always.”

Sometimes providing a comparable idea solves the problem of understanding something, and sometimes it doesn’t. Tony nodded, but still looked a little puzzled. I wasn’t sure I had solved the problem, so I applied a new heuristic:

Point ’em to a book.

I suggested that he read George Polya’s book How to Solve It. “In that book, Polya presents a set of ideas and questions you can ask yourself that can help you to solve math problems.”

“Wait… I thought you always solved math problems with algorithms,” Tony said.

“That’s when you know how to solve the problem. When you don’t, Polya’s suggestions—heuristics—can get you started. They don’t always work, but they tend to be pretty powerful, and when one doesn’t work, you try another one. You never know which questions or ideas will help you solve the problem most quickly. So you practice this cycle: apply a heuristic, and if you’re still stuck, try another one. After a while, you develop judgement and skill, which is what you need to apply heuristics well. Polya talks about that a lot. He also emphasizes just how much heuristics are fallible and context-dependent.”

Mind you, neither Tony nor I had a copy of Polya’s book right handy, and Tony wanted to understand “heuristics” better now. The “point ’em to a book” heuristic had failed this time, even though it might have worked in a different context. So I tried yet another heuristic to solve the problem:

Point ’em to another book.

I suggested that he read Gut Feelings by Gerd Gigerenzer. “In that book, Gigerenzer emphasizes that heuristics tend to be fast and frugal (that is, quick and inexpensive). That’s important, he says: humans need heuristics because they’re typically dealing with bounded rationality.”

Uh-oh. Tony’s eyes had glazed over again at the mention of “bounded rationality”. So I applied a heuristic:

Even when it’s a deep concept, a fast and frugal explanation might do.

After all, Polya says that a heuristic isn’t intended to be perfect. Instead, heuristics are provisional and context-dependent. So in order to provide a quick understanding of “bounded rationality”, I said, “In a nutshell, bounded rationality is a situation when you have incomplete knowledge, imperfect understanding, and limited time.”

He grinned, and said, “What, like when you’re testing? Like most of the time in life?”

“Yes. Billy Vaughan Koen, in another book, Discussion of the Method, says that the engineering method is ‘to cause the best change in a poorly understood situation within the available resources.'”

“So he’s saying that engineers apply heuristics?” Tony asked. “I guess that makes sense, since engineers solve problems in ways that usually work, but sometimes there are failures.”

He seemed to be getting it. But I wanted to test that, so I applied a heuristic for making the decision, “Does he get it?

Ask the student to provide an example.

So I said, “I think you might have it. But can you provide me with an example of a heuristic?”

He said, “Okay. I think so.” He paused. “Here’s a heuristic for solving the problem of opening a door: ‘Pull on the handle; push on the plate.’ That’s what you do when you get to a door, right? It’s a heuristic that usually works. Well… it might fail. It could be one of those annoying doors that have handles on both sides, where you have to push the handle or pull the handle to open the door. It might be one of those doors that opens both ways, like the doors for restaurant kitchens, so there’s no handle. The door might not even have a handle or a plate; it might have a knob. In that case, you apply another heuristic: ‘Turn the knob’. That’s a solution for the problem of opening a door that doesn’t have a handle or a plate. But that heuristic might fail too. The door might be locked, even though the knob turns. It might be one of those fancy doors that have dead-bolt locks and knobs that don’t turn. It might not have a knob at all; it might have one of those old-fashioned latches. So none of those heuristics guarantees a solution, but each one might help to solve the problem of getting through the door.”

“Great! I think you’ve got it.”

“To be precise about it,” he said, “you can’t be sure, so you’re applying heuristics that help you to make the decision that I get it.”

I laughed. “Right. So what’s the difference,” I asked, “between an oracle and a heuristic?”

He paused.

(to be continued…)

Problems with Problems

Wednesday, April 4th, 2012

People sometimes seem to struggle with a concept that’s central to testing, the concept of “oracle”. In the three-day Rapid Software Testing class, we define an oracle as

a principle or mechanism means by which we recognize a problem.

Sometimes I like to emphasize that oracles are fallible and context-dependent. When that’s so, I say that an oracle is

a heuristic principle or mechanism means by which we recognize a problem.

(Updated in 2019 to reflect a change that we made some time ago. –MAB)

That means that an oracle can work but might fail in helping us to recognize a problem. A heuristic is something that helps us learn; it’s a fallible method for solving a problem or making a decision. So an oracle, as a special kind of heuristic, helps us to make a particular decision in answer to the question, “Is there a problem here?” If the answer is Yes, it’s because an oracle is enabling us to recognize a problem.

For some time, I’ve been surprised at how tricky a concept this seems to be for some people. Possibly, I thought, it was because people simply weren’t used to thinking about oracles; for many it’s a new word and a new concept. Maybe the problem was with “heuristic”, or “principle”, or “mechanism”. Recently, though, I began to wonder: perhaps the difficulty comes from the fact that people aren’t used to thinking deeply about problems. I tested that idea by asking some people, and I found that they struggled in describing how they thought of problems. So what is a problem, anyway?

Here are two ways of thinking about problems that I’ve found useful.

1) A problem is “a difference between what is perceived and what is desired.” (Dewey, J. (1933), How We Think: A Restatement of the Relation of Reflective Thinking to the Educative Process) I first learned this definition of “problem” in a session led by Don Gray at the Amplifying Your Effecctiveness Conference a few years ago, and I believe it shows up in several places in Jerry Weinberg‘s work.

2) A problem is “an undesirable situation that is significant to and maybe solvable by some agent, though probably with some difficulty.” (G.F Smith, “Towards a Heuristic Theory of Problem Structuring”, quoted in Weick, Karl E. Sensemaking in Organizations. Sage Publications, Inc, 1995.)

Both definitions, I think, point to the same thing: problems are based on desires, perceptions, and situations. But both definitions have a minor problem for me, which is that they do not make explicit something I think to be very important: desires, perceptions, and situations are centred around people and context. Problems are subject to the Relative Rule:

For any abstract X, X is X to some person, at some time.

People often talk of problems (bugs, defects, issues, and so forth) as though they were attributes of a product or of a situation. But problems aren’t attributes as such. They’re relationships between some person(s), the product, and the system that includes them, and the situation that encompasses it all. A problem is a problem for some person, at some time. Differences, perceptions, and desires are like that too; they’re relative. If I were to expand out Dewey’s notion of “problem”, it would look like this:

A problem is a difference (according to some person) between what is perceived (by some person) and what is desired (by some person) (all at some point in time).

There are several implications here.

  • The problem might be in the difference, or in the perception, or in the desire.
  • Different people will have different notions of differences, different perceptions, different desires.
  • A problem may also be influenced by timing; something that’s a problem today might not be a problem tomorrow, and something that isn’t a problem today might be a catastrophe tomorrow.
  • Since, as testers, we’re in the business of finding potential problems, we must be alert to the enormous variety of people who may be affected by the product and the project.
  • Our mission as testers typically includes the task of identifying possible problems. As such, we should resist the temptation to dismiss something as “not a problem”, because…
  • Something that we might dismiss as a trivial problem for some people might be a serious problem for other people and…
  • Something that we might see as a serious problem in our perception might appear as a trivial problem for other people so…
  • We’re obliged as testers to identify a possible problem in terms of its most serious consequences for people who might matter to the product owner. However…
  • We’re not in the business of deciding whether something is a problem or not, so for a given problem, our testing clients or the project owners decide on its ultimate significance, and on their response to it.

So the bottom line is that problems are slippery. A problem is not a thing in a product or in the world, but a relationship between some situation and some person.

Smith’s emphasis on an ability to solve a problem may matter too. People often accept things that they perceive as beyond anyone’s ability to solve. For example, I can’t do anything about a meteor hitting my computer in the next few minutes, so I’m not going to treat the possibility that it might happen as a problem. Yet again, I should be careful to suppress any of my own prejudices that nothing can be done with respect to a given problem. That’s a decision for those who build and those who own the product, since they may have information and more resources of which I am unaware.

One of the fundamental questions of testing is “Is there a problem here?” Oracles are media, in the McLuhan sense. Media are tools, extensions of ourselves that enhance, enable, accelerate, or intensify our capacity to do things. McLuhan pointed out, though, that media are agnostic about what they extend. If our concept of “problem” is limited, our oracles will extend and accelerate our ability to recognize a limited set of problems. In other words, with too narrow an idea of what a problem might be, oracles may only help us to recognize too narrow a set of possible problems.

So: one key to understanding and applying oracles skilfully is to recognize the richness and diversity of what we might mean by problem. What we’re testing is not simply source code or a collection of compiled object code files. We’re testing products or services that are systems, sets of things in meaningful relationship to each other. Those systems are related to other systems. Products don’t stand on their own. Every product is part of a system that includes people who build it, people who support it, people who maintain it, people who buy it, people who use it directly, and people who are affected by it. That’s an incomplete list of people who might experience problems related to the product or system. Try asking these questions:

  • What are the elements of system that I’m testing?
  • Who might have a relationship to that system, or to its elements?
  • Who else might have a relationship to it?
  • What desires might they have that are related to the system?
  • What aspects of the system might provide value to those people by fulfilling their desires? What might threaten that value by dashing their desires?
  • How might people perceive the system? What might they see, hear, smell, touch, taste, or feel as they use it or interact with it?
  • What might influence, magnify, sharpen, or distort their perceptions?
  • What differences might they experience between their perceptions and their desires?
  • Could someone, or something—some change in the situation—help them to solve or otherwise deal with such differences?
  • What might change in the system over time? How might people’s perceptions, desires, or notions of differences change over time?
  • What ideas, tools, or conversations might help me to recognize perceptions, desires, and differences?

Reflecting on these questions periodically, even briefly, may help you to expand your notion of what a problem is. That in turn may extend your ability to use oracles to recognize potential problems in the system you’re testing.

(What’s an oracle? An oracle is a heuristic for addressing the question “Is there a problem here?” But wait… that raises the question, “What’s a heuristic?” Find out here.)

This blog post was inspired and sharpened by conversations with Anne-Marie Charrett, Peter Walen, and James Bach. Thanks to them.