Blog Posts from October, 2009

Comment on a Not-So-Good Article on Exploratory Testing

Friday, October 30th, 2009

An article from a while back on StickyMinds entitled How To Choose Between Scripted and Exploratory Testing refers to a bunch of factors in making choices between scripted testing and exploratory testing. The problems start early: “As a test manager or engineer, you may be considering whether or not to use exploratory testing on your next project.”

If you’re not planning on investigating any problems that you find, I suppose that you could choose not to use exploratory testing (investigating a bug is not a scripted or scriptable process). If you don’t allow the tester to control his or her actions, or to feed information from the last test into the thought process and action behind the next test, then you could get away without taking an exploratory approach. If the testing ideas were all declared and written down at the beginning of the project, you could successfully avoid exploratory testing. And if you used an all-checking strategy, maybe exploratory testing wouldn’t be an approach you’d take. But would you really want all of your testing to be confirmatory? All validation and verification? All checking to make sure that existing beliefs are correct, rather than probing the product to disconfirm those beliefs, with the goal of showing that they’re inaccurate or incomplete?

This isn’t the worst article ever written on exploratory testing. It is, however, typical of a certain kind of bad article. It’s full of assertions that aren’t supported in the view of exploratory testing that a bunch of us (including Jon Bach, who, at the bottom of the article provides an unusually stern comment by his usual standards) have been developing for years. Let’s have a look at some of the points. Quotes from the original article are in italics.

Tester domain knowledge…If a tester does not understand the system or the business processes, it would be very difficult for him to use, let alone test, the application without the aid of test scripts and cases.

A few years back, a former co-worker contracted me to to document a network analysis tool that was being rebranded, fixed, and upgraded by his new company. The only documentation available was not only weak but also a couple of versions out of date. In addition, I didn’t know much about network analysis tools at all. I reviewed some literature on the topic, and I interviewed the company’s lead programmer. But for me, the most rapid, relevant, and powerful learning came from interacting with the product, figuring out how it worked (and, at several points, how it appeared to fail), and building the story of the product in my mind and in the documentation that I created. Without that interaction with the product, my learning would have not have been rooted in experience, and would have been much slower. And the documentation that I produced was deemed excellent by the client.

How can I test a product in a domain that I don’t know much about? The answer is that through testing, I learn, and I learn far more rapidly than by other means. Anyone learning a complex cognitive activity learns far more, far more quickly, in the doing of it than in the reading about it. Preschool kids do the same thing. Watch them; then watch how they tend to slow down and follow the schools’ processes instead of their own.

System complexity…End-to-end testing can be accomplished with exploratory testing; however, the capabilities and skill sets required are typically those of a more experienced test engineer.

People often argue that exploratory testing needs capable, experienced, and skilled testers. I agree. The argument suggests (and often states outright) that scripted testing doesn’t need capable, experienced, and skilled testers. To some degree that’s true; when the scripted action is checking, by definition skill and sapience aren’t required for the moment of the check. But sapience and skill are required for the design and construction of the check, and for the analysis and interpretation of the results, so the argument that scripted testing doesn’t need skilled testers doesn’t hold water if you want to do scripted testing well. Indeed, good testing of any kind requires skill. If you or your testers are genuinely unskilled, training, coaching, mentoring, and a fault-tolerant environment that fosters learning will help to build skills quickly.

I’m also puzzled by the argument in another way: the (claimed) lack of a need for skilled testers is often presented as a virtue of scripted testing. This is like saying that the lack of a requirement for medical training is a virtue of quack medicine.

Level of documentation. Scripted testing generally flows from business requirements documents and functional specifications. When these documents do not exist or are deficient, it is very difficult to conduct scripted testing in a meaningful way. The shortcuts that would be required, such as scripting from the application as it is built, are accomplished as efficiently using exploratory testing.

Actually, there are several ways to do scripted testing in a meaningful way without business requirements documents or functional specifications being present or perfect. A standout example is test-driven development, in which checks are developed prior to developing the application code. Another: unit tests, in which checks are prepared at some point after the application code has been developed. Both forms of checks support refactoring. Another example: many Agile development shops use Fitnesse to develop, explore, and discover many of the requirements of the product, and to create scripted checks as development proceeds.

In fact, no one of whom I am aware has ever seen perfect requirements documents; they’re always deficient in some way for some person’s purpose, and always to some degree that’s intentional. It’s infeasible and wasteful to document every assumption; for programmers and testers with appropriate judgment and skill, it’s also somewhat insulting. Thus, whether developing either scripted or exploratory test ideas, we should treat imperfect documents as a default assumption.

Timeframes and deadlines.The lead-in time to test execution determines whether you can conduct test design before execution. When there is little or no time, you might at least need to start with exploratory testing while documented tests are prepared after the specifications become available.

Here is the implication, once again, that exploratory tests aren’t documented. Yet exploratory ideas can be documented in advance, in the form of checklists or as charters. Existing data files—a form of documentation—can be available in advance. Exploratory tests can be guided by marked-up diagrams, by user documentation, or even by a functional specification. Exploratory testing can be informed by any kind of documentation you like. One key distinction between exploratory and scripted approaches is not whether documentation is available in advance; with ET the difference is that the tester, rather than the document, is the primary agency driving the design and execution of the test. Another key distinction is that in an exploratory approach, detailed documentation of the tester’s actions tends to be de-emphasized prior to test execution, and tends to be produced during or very shortly after testing.

Available resources. The total number of person-days of effort can determine which approach you should take. Formal test documentation has significant overhead that can be prohibitive where resources and budgets are tight.

Well, at least maybe we agree on something. But the number of person-days of effort doesn’t determine which approach you choose; people do that. It’s unclear here as to whether the person-days are a given, or whether the test manager gets to choose. Moreover, I wouldn’t use the word “formal” here, since documentation associated with exploratory approaches can be quite formal (look at session-based test management as an example). The word I’d use instead of “formal” above is “excessive” or “wasteful” or “overly detailed”. I’d also suggest that it’s fairly rare for testers to feel as though resources and budgets are ample; most testers I speak with maintain that resources and budgets are always tight, for them.

Skills required. The skill sets of your team members can affect your choices. Good test analysts may not necessarily be effective exploratory testers and vice versa. A nose for finding bugs quickly and efficiently is not a skill that is easily learned…

Perhaps not. James Bach, Jon Bach, James Lyndsay, and I (to name but four) have a lot of experience training testers, and our experience is that the capacity to find bugs can be learned more quickly than many people believe. But habitutate testers to working from detailed manual test scripts and I’ll guarantee that they don’t develop skill quickly. In fact, a huge part of learning to find problems quickly lies in the tester being given the freedom and responsibility to explore and to develop his own mental models of the product.

Verification. Verification requires something to compare against. Formal test scripts describe an expected outcome that is drawn from the requirements and specifications documents. As such, we can verify compliance. Exploratory testing is compared to the test engineer’s expectations of how the application should work.

Many kinds of testing—not just verification—require something to compare against. That “something” is called an oracle, a principle or mechanism by which we recognize a problem. The claim here is that scripted testing is a good thing because we can verify “compliance” (to what?) based on requirements and specifications documents. Actually, what we’d be verifying here is consistency between some point in a test script and a specific claim made in one of the documents. There are two problems here. One is the problem of inattentional blindness; focusing a tester’s attention on a single observation drives the tester’s attention away from other possible observations that might represent an issue with the product. The second problem relates to the fact that requirements and specification documents (as above) can be presumed to contain errors, misinterpretations, and outdated information.

Exploratory approaches defend against these two problems by their emphasis on the tester’s skill set and mindset. Rather than depending upon a single oracle (that of consistency with a specification), an exploratory approach emphasizes apply several oracles, including consistency with the product’s history, with the image the development organization wants to project, with comparable products, with reasonable user expectations, with the intended purpose of the product, with other elements within the product, and with relevant standards, regulations, or laws. Claims—statements in the requirement documents—are taken seriously by the exploring tester, of course, but the other kinds of oracles provide a rich set of possible comparisons, rather than the relatively impoverished single set provided by a script. By keeping the tester cognitively engaged and under her own control, we lessen the risk of script-induced inattentional blindness. Statements like “Exploratory testing is compared to the test engineer’s expectations of how the application should work” are not only inaccurate, but also trivialize the complex set of heuristics that skilled testers bring to the game.

At this point, I’m in agreement with Jon Bach. Refuting the rest of the article would take more time than I’ve got at the moment. So, to keep the curious and eager people occupied:

Web Resources

Evolving Understanding of Exploratory Testing (Bolton)
A Tutorial in Exploratory Testing (Kaner)
The Nature of Exploratory Testing (Kaner)
The Value of Checklists and the Danger of Scripts: What Legal Training Suggests for Testers
Exploratory Testing Dynamics (James Bach, Jon Bach, Michael Bolton)
General Functionality and Stability Test Procedure (for Microsoft Windows 2000 Application Certification) (James Bach)
Experiments in Problem Solving (Jerry Weinberg)
Collaborative Discovery in a Scientific Domain (Okada and Simon)
– a study of collaborative problem-solving. Notice the subjects are testing software.


Exploring Science: The Cognition and Development of Discovery Processes (David Klahr)
Plans and Situated Actions (Lucy Suchman)
Play as Exploratory Learning (Mary Reilly)
Exploratory Research in the Behavioral Sciences
Naturalistic Inquiry
How to Solve It (George Polya)
Simple Heuristics That Make Us Smart (Gerg Gigerenzer)
Blink (Malcolm Gladwell)
Gut Feelings (Gerg Gigerenzer)
Sensemaking in Organizations (Karl Weick)
Cognition in the Wild (Edward Hutchins)
The Social Life of Information (Paul Duguid and John Seely Brown)
A System of Logic and Ratiocination (John Stuart Mill)
Sciences of the Artificial (Herbert Simon)

Yes, learning about exploratory testing in a non-trivial way might take some practice and study. That goes for anything worth doing expertly, doesn’t it?

Maturity Models Have It Backwards

Wednesday, October 28th, 2009

At a couple of recent conferences, some people have asked me about one “maturity” model or another. As one of the few people who has read the CMMI book from cover to cover, here’s what I think: In process-speak, the notion of maturity is backwards.

A mature entity, in biology, is one that can survive and thrive without parental support. A mature being is one that has achieved an age and stage where it can reproduce, mutate, and diversify. In general, diversity and sustainability come from interaction with other mature creatures. An animal or plant attains and sustains maturity either by adapting to its environment, or by being adapted to it already, but no species is adapted to all environments.

A mature person is one who is highly conscious of when it’s appropriate to follow rules and when to break them. A mature person is largely self-guided. Only in exceptional circumstances does a mature person need to refer to or appeal to a rulebook at all. In such cases, the issue is that someone believes that the rulebook isn’t working, and in such cases, consensus between mature individuals and organizations—not the rulebook itself—makes the determination as to what should happen next. Mature people know that rulebooks need to be interpreted.

We don’t consider a person mature when he says or does the same thing over and over again, when he answers by rote, when he appeals to authority, or when he goes through the motions. We consider a person mature when he is able to think and perform independently, when he behaves responsibly and respectfully towards others, and when he accepts responsibility for his actions. We also don’t mind when a mature person goofs around every once in a while, as long as little or no harm comes of it.

We generally think of mature people as being relaxed and easy-going, rather than rigid and uptight. Mature people who can’t get their way immediately don’t stamp their feet, shout, or hold their breath until they turn blue. Mature people recognize the possibility that other people’s values, actions, languages, cultures, and means of expression are worthy of respect. Mature people question themselves at least as quickly and as closely as they question others. Mature people are forgiving and fault-tolerant, recognizing that immature people often need to make mistakes in order to learn. Mature parents don’t scare the kids, don’t yell at them, don’t try to protect them from every peril. Mature parents provide a supportive environment where kids can make mistakes and learn from them.

One more thing about mature beings: they eventually get old, and they die, to be replaced by other mature beings.

So what does all this mean for “process maturity”?

If maturity means the same thing for processes as for other living things, a genuinely mature process, whether for individuals or for groups, should incorporate freedom, responsibility, diversity, adaptability, and self-sufficiency. A genuinely mature process shouldn’t emphasize repeatability as a virtue unto itself, but rather as something set up to foster appropriate variation, sustainability, and growth. A mature process should encourage risk-taking and mistakes while taking steps to limit the severity and consequence of the mistakes, because without making mistakes, learning isn’t possible.

What do you think? How do process “maturity” models stack up?

Note: This post was inspired by Jerry Weinberg’s writing, in particular the material on “maturity” in Quality Software Management, Volume 1: Systems Thinking. For an unusually sharp passage on the subject, see pages 20 and 21 (also quoted in this blog post).

The Testers’ Christmas Present

Monday, October 19th, 2009

So the holidays are coming up, and you’re wondering what to get for your tester friends, or (if you’re a tester) for your kids.

Let me be the first this season to recommend I Am A Bug, a perfectly charming little book by Robert Sabourin, and illustrated by his daughter Catherine, who was between 11 and 12 years old as the book was being published. It’s been around for several years.

The secret is that I Am A Bug is a serious testing book, cleverly disguised as a children’s book. It’s one of the wisest books on testing I’ve ever read. Each page begins with a big message, illustrated and elaborated below. Here’s a littl sample:

A bee sting may hurt a bit

The same bug can be found in different computer programs. In one program the bug may not cause much damage…

But it can kill you if you’re allergic to bees.

…but in another program it could be fatal.

The whole book is online, at Rob’s Web site, (The book is found under “Presentations”.) It’s fun to read there, but order the dead-tree version for a very reasonable price. You can get it from the usual online booksellers. Highly recommended.

When Do We Stop Testing? One More Sure Thing

Tuesday, October 13th, 2009

Not too long ago, I posted a list of stopping heuristics for testing. As usual, such lists are always subjective, subject to refinement and revision, and under scrutiny from colleagues and other readers. As usual, James Bach is a harsh critic (and that’s a compliment, not a complaint). We’re still transpecting over some of the points; eventually we’ll come out with something on which we agree.

Joe Harter, in his blog, suggests splitting “Pause That Refreshes” into two: “Change in Priorities” and “Lights are Off“. The former kicks in when we know that there’s still testing to be done, but something else is taking precedence. The latter is that we’ve lost our sense of purpose—as I suggested in the original post we might be tired, or bored, or uninspired to test and that a break will allow us to return to the product later with fresh eyes or fresh minds. Maybe they’re different enough that they belong in different categories, and I’m thinking that they are. Joe provides a number of examples of why the lights go out; one feels to me like “customary conclusion”, another looks like “Mission Accomplished”. But his third point is interesting: it’s a form of Parkinson’s Law, “work expands to fill the time available for its completion”. Says Joe, “The test team might be given more time than is actually necessary to test a feature so they fill it up with old test cases that don’t have much meaning.” I’m not sure how often people feel as though they have more time than they need, but I am sure that I’ve seen (been in) situations where people seem to be bereft of new ideas and simply going through the motions. So: if that feeling comes up, one should consider Parkinson’s Law and a Pause That Refreshes. Maybe there’s a new one there. But as Joe himself points out, “In the end it doesn’t matter if you use [Michael’s] list, my list or any list at all. These heuristics are rules of thumb to help thinking testers decide when testing should stop. The most important thing is that you are thinking about it.”

For sure, however, there is a glaring omission in the original list. Cem Kaner pointed it out to me—and that shouldn’t have been necessary, because I’ve used this heuristic myself. It focuses on the individual tester, but it might also apply to a testing or development team.

Mission Rejected. We stop testing when we perceive a problem for some person—in particular, an ethical issue—that prevents us from continuing work on a given test, test cycle, or development project.

Would you continue a test if it involved providing fake test results? Lying? Damaging valuable equipment? Harming a human, as in the Milgram Experiment or the Stanford Prison Experiment? Maybe the victim isn’t the test subject, but the client: Would you continue a test if you believed that some cost of what you were doing—including, perhaps, your own salary—were grossly disproportionate to the value it produced? Maybe the victim is you: Would you stop testing if you believed that the client wasn’t paying you enough?

The consequences of ignoring this heuristic can be dire. Outside the field of software testing, but in testing generally, a friend of mine worked in a science lab that did experiments on bone regeneration. The experimental protocol involved the surgical removal of around one inch of bone from both forelegs of a dog (many dogs, over the course of the research), treating one leg as an experiment and the other as a control. Despite misgivings, my friend was reasonably convinced of the value of the work. Later, when he found out that these experiments had been performed over and over, and that no new science was really being done, he suffered a nervous breakdown and left the field. Sometimes testing doesn’t have a happy ending.