Blog: Confusion as an Oracle

A couple of weeks back, Sylvia Killinen (@skillinen on Twitter) tweeted:

“Seems to me that much of #testing relies on noticing when one is confused rather than accepting it as Something Computer Programs Do.”

That’s a beautiful observation, near and dear to my heart since 2007 at least. The night I read Sylvia’s tweet, I wanted to blog more on the subject, but sometimes blog posts go in a different direction from where I intend them to go. At the time, I went here. And now I’m back.

Sylvia’s tweet reminded me of a story that Jon Bach tells about learning testing with his brother James. Jon had been working in a number of less-than-prestigious jobs. James suggested that Jon become a tester, and offered to train him how to be an excellent tester. Jon agreed to the idea. Everything went fine for the first couple of weeks, but one day Jon slumped into James’ office looking dejected and demoralized. The conversation went something like this.

“What’s the problem?” asked James.

“I dunno,” said Jon. “I don’t think this whole becoming-a-tester thing is going to work out.”

“Not work out? But you’re doing great!” said James.

“Well, it might look that way to you, but…” Jon paused.

“So what’s the problem?”

“Well, you gave me this program to test,” Jon began. “But I’m just so confused.”

James peered over his glasses. “When you’re confused,” he said, “that’s a sign that there’s something confusing going on. I gave you a confusing product to test. Confusion might not be fun, but it’s a natural consequence when you’re dealing with a confusing product.” James was tacitly suggesting that Jon’s confusion cound be used as an oracle—a heuristic principle or mechanism by which we recognize a problem.

This little story suggests and emphasizes a number of serious and important points.

As I mentioned before, here, feelings don’t tell us what they’re about. Confusion doesn’t come with an arrow that points directly back to its source. Jon felt confused, and thought that the confusion was about him. But that confusion wasn’t just about Jon’s internal state; it was also about the state of the product and how Jon felt about it. Feelings—internal, non-specific and ambiguous—don’t tell us what’s going on; they tell us to pay attention to what’s happening around us. When you’re a novice, you might be inclined to believe that your feelings are telling about yourself, but that’s likely not the whole story, since emotions don’t happen in isolation from everything else. It’s more probable that feelings are telling you about the relationship between you and something else, or someone else, or the situation.

Which reminds me of another story. It happened at Jerry Weinberg’s Problem Solving Leadership workshop in 2008. PSL is full of challenging and rich and tricky exercises, and one day, one team had fallen into a couple of traps and had done rather badly. During the debrief, Jerry remarked on it. “You guys handled a much harder problem than this yesterday, you know. What happened this time?”

One of the participants answered, “The problem screwed us up.”

With only the briefest pause, Jerry peered at the fellow and replied in a gently admonishing way, “Your reaction to the complexity of the problem screwed you up.”

Methodologists and process enthusiasts regularly ignore the complex human and emotional aspects of testing, and so don’t take them into account or use them as a resource. Some actively reject feelings as a rich source of information. One colleague reports that she told her boss about a presentation of mine in which I had discussed the role of emotions in software testing.

“There’s no role for emotions in software testing,” he said quietly.

“I’m not sure I agree,” she said. “I think there might be. I think at least it’s worth considering.”


She remarked he had seemed agitated—a strange reaction considered the mismatch between what he was saying and what he appeared to be feeling. What more might we learn by noting his feelings and considering possible interpretations? What inferences might we draw about the differences between his reaction and hers?

As we increasingly emphasize in the Rapid Software Testing course, recognizing and dealing with your feelings is a key self-management skill. Indeed, for testers, feelings are a kind of first-order measurement. It’s okay to be confused. The confusion is valuable and even desirable if it leads you to the right control action, which is to investigate what your emotions might be telling you and why. If we’re willing to acknowledge our feelings, we can use them productively as cues to start looking for oracles and problems in the product that trigger the feelings—before those problems lead our customers to distress.

In my article Testing Without a Map, I discuss some of the oracles that we present in the Rapid Software Testing class and methodology.

Thanks to Sylvia for the inspiration.

I’ll be bringing Rapid Testing to the Netherlands (October 31-November 2), London (November 28-30), and Oslo (December 14-16). See the right-hand panel for registration details. Join us! Spread the word! Thank you!

Want to know more? Learn about upcoming Rapid Software Testing classes here.

9 responses to “Confusion as an Oracle”

  1. Joey says:

    “Your reaction to the complexity of the problem screwed you up.”

    I’m glad no one was around to see the Terry Gilliam god cutout and trumpeting angels that appeared above my head when I read that. Because that would’ve been embarrassing.

    Great post, Michael. Thanks.

  2. Good MB, real good.

    The challenge I see here is getting this message across. It’s hard enough convincing some testers of this let alone PMs, CIOs, etc, etc.

    If testers were to raise these issues of ‘confusion’ (and I agree they should) then bug advocacy will be paramount.

    If these issues were raised more often then more products would be sold. My wife for example… will pick up a new mobile phone in the shop and have a quick play with the UI. Within 2 minutes it’s back on the shelf with a resounding “too confusing!” – Sale fail.

    Testers – get in contact with your emotional self, no matter what your boss may say.

  3. Rahul Gupta says:

    Awesome post Michael.

    I have been experiencing the same about one of the testing assignments that I have been working on these days. This post is inspiring. I now realize that I must take a second look on what is being done (on testing and with the variables that affects testing).
    Its like seeking a second opinion from a different doctor when someone is diagnosed with a life threatening decease…and the first doctor herself is not sure whether the diagnosis is correct. My analogy may not be entirely correct but it makes some sense to me.

    And here is a question. The manager in your post who said,”THERE’S NO ROLE FOR EMOTIONS IN SOFTWARE TESTING!”…how to deal with such people? Would presenting more facts help? Can we say that emotions & feelings are measurable because measurement is something QA Managers (so called) across the world think of (or pretend to present). Please guide.

  4. Sreeman says:

    “Feelings—internal, non-specific and ambiguous—don’t tell us what’s going on; they tell us to pay attention to what’s happening around us”.

    Awesome. I liked it very much.
    Though not relevant, Just to add that, This holds good for a skilled tester. Ambiguous and confusion states do exist while we lack the required skill and is not actually the state of the product 🙂

  5. […] Bolton has an interesting post regarding confusion as an oracle: James peered over his glasses. “When you’re confused,” he said, […]

  6. Dwain Olfert says:

    Great post (once again) Michael!

    Emotions definitely are a great clue towards identifying a problem with the system under test. Confusion, frustration, or even the positive emotion of being content can give you clues to the stability of the product. I always say if a tester is confused or frustrated, and they have been working with the product for x months, how will ‘real’ users feel (novice or advanced)? User experience matters! That includes the tester’s user experience. At minimum, testers should communicate these feelings to the stakeholders to determine potential ‘real-user’ impacts and how they can be addressed (code fix, design change, training, support, etc.).

    Testers – tap into your emotion Oracles!

  7. I often face the problem of confusion with my team since most of the testing we do is done to unfamiliar programs and under extreme time-constraints. I’ve often seen the confusion in the faces of my testers when we start on new project. The confusion usually passes within few hours on the new project and I’ve understood that within the first 1 or 2 days I should be more available for helping to overcome their moments of confusion. In addition I’ve understood that these confusion moments actually provide remarkable amount of additional information about the program and thus use these periods as edge.

    About “There’s no role for emotions in software testing” – As long as I consider testing as sapient social activity, then I promote people to define emotions they sense during testing and find the source of their emotions. Since emotions are one of main components of human activities, then taking out emotions removes some (if not large part) advantages of sapient human testing. Of course sometimes the emotsions are not related to the task in hand, but as time passes by and people learn to define the source of their emotions better, then both time spent defining the source of emotions and amount of non-related emotions decreases.

  8. […] • ??????? ?? ????? ????????????? ?????????????: ??????????? ???? ?????? ??? ? ??????? ??????. […]

Leave a Reply

Your email address will not be published.