In the past, James Bach and I have defined “oracle” as “a principle or mechanism by which we recognize a problem,” and we’ve focused on principles rooted in ideas about consistency and inconsistency. At its foundation, applying an oracle and recognizing a problem always involves some principle that links an observation of a product to someone’s desires for it. That is: we observe a problem when we see some kind of undesirable relationship between related things: inconsistency with things someone wants, consistency with things someone doesn’t want.
Most traditional ideas about oracles focus on mechanisms—typically in the form of a reference document, or a tool that provides “the correct answer”. (There are enormous problems with the notion of an oracle as the source of the correct answer, but I’ll save that for a later post.) Documents and tools are media—things that lie between ourselves and some person who matters. Media, as McLuhan points out, extend, accelerate, enhance, enable, or intensify some human capability. Oracles are media that extend, accelerate, enhance, enable, or intensify our ability to recognize a problem—presumably a problem for some person who matters.
In a conversation in 2009, Cem Kaner remarked to me that a tester is himself or herself a test instrument. As I’ve reflected on that, I’ve come to realize that the role of the tester is that of a medium—one that extends, accelerates, enhances, enables, or intensifies our clients’ abilities to understand their products and the problems in them.
At least since 2006, James has been talking about the concept of the “live oracle”. A live oracle may be the product owner, a programmer, a domain expert, a business analyst, a designer, another tester—any person who can alert us to the presence of a problem.
It’s the nature of media that they are indirect; mediate (as an adjective), rather than immediate. That is, they’re in between the observer of the problem and the person to whom the problem matters. But naturally, sometimes our recognition of a problem can be more immediate; that is, we recognize a problem based on a principle, without a specific mechanism or instrument or person to mediate it.
So: sometimes as we’re testing, we recognize a problem by a direct feeling, or through a principle that we’re aware of. Sometimes our recognition of a problem comes from a document or tool; sometimes from another person. That may entail embodied mental processes, rational principles, or mechanisms, but all that’s kind a mouthful. So lately, James and I have started to say that an oracle is simply “a way to recognize a problem”. Here’s a high-level, work-in-progress list of ways a tester might recognize a problem:
Feelings like confusion, surprise, frustration, annoyance, impatience, or amusement.
People, such as a person whose opinion matters, or the opinion of that person; or disagreements among people who matter.
Artifacts, including reference documents with useful information; sets of known good or known bad example outputs; comparable products or features or algorithms.
Mechanisms, such as processes or tools by which output can be checked, or that help a tester to identify patterns.
Principlesb)—undesirable inconsistencies between the product and something related to it; or undesirable consistency with things we don’t like.
There are two things that we encourage Rapid Software Testing students to do with this list:
1) What examples can you provide as specific instances of each of these categories? (The FEW HICCUPPS heuristics are examples of principles)
2) What are the factors or dimensions that might cause us to use or prefer one oracle over another?
I’ll try to provide some answers over the coming days.