Blog: Comment on a Not-So-Good Article on Exploratory Testing

An article from a while back on StickyMinds entitled How To Choose Between Scripted and Exploratory Testing refers to a bunch of factors in making choices between scripted testing and exploratory testing. The problems start early: “As a test manager or engineer, you may be considering whether or not to use exploratory testing on your next project.”

If you’re not planning on investigating any problems that you find, I suppose that you could choose not to use exploratory testing (investigating a bug is not a scripted or scriptable process). If you don’t allow the tester to control his or her actions, or to feed information from the last test into the thought process and action behind the next test, then you could get away without taking an exploratory approach. If the testing ideas were all declared and written down at the beginning of the project, you could successfully avoid exploratory testing. And if you used an all-checking strategy, maybe exploratory testing wouldn’t be an approach you’d take. But would you really want all of your testing to be confirmatory? All validation and verification? All checking to make sure that existing beliefs are correct, rather than probing the product to disconfirm those beliefs, with the goal of showing that they’re inaccurate or incomplete?

This isn’t the worst article ever written on exploratory testing. It is, however, typical of a certain kind of bad article. It’s full of assertions that aren’t supported in the view of exploratory testing that a bunch of us (including Jon Bach, who, at the bottom of the article provides an unusually stern comment by his usual standards) have been developing for years. Let’s have a look at some of the points. Quotes from the original article are in italics.

Tester domain knowledge…If a tester does not understand the system or the business processes, it would be very difficult for him to use, let alone test, the application without the aid of test scripts and cases.

A few years back, a former co-worker contracted me to to document a network analysis tool that was being rebranded, fixed, and upgraded by his new company. The only documentation available was not only weak but also a couple of versions out of date. In addition, I didn’t know much about network analysis tools at all. I reviewed some literature on the topic, and I interviewed the company’s lead programmer. But for me, the most rapid, relevant, and powerful learning came from interacting with the product, figuring out how it worked (and, at several points, how it appeared to fail), and building the story of the product in my mind and in the documentation that I created. Without that interaction with the product, my learning would have not have been rooted in experience, and would have been much slower. And the documentation that I produced was deemed excellent by the client.

How can I test a product in a domain that I don’t know much about? The answer is that through testing, I learn, and I learn far more rapidly than by other means. Anyone learning a complex cognitive activity learns far more, far more quickly, in the doing of it than in the reading about it. Preschool kids do the same thing. Watch them; then watch how they tend to slow down and follow the schools’ processes instead of their own.

System complexity…End-to-end testing can be accomplished with exploratory testing; however, the capabilities and skill sets required are typically those of a more experienced test engineer.

People often argue that exploratory testing needs capable, experienced, and skilled testers. I agree. The argument suggests (and often states outright) that scripted testing doesn’t need capable, experienced, and skilled testers. To some degree that’s true; when the scripted action is checking, by definition skill and sapience aren’t required for the moment of the check. But sapience and skill are required for the design and construction of the check, and for the analysis and interpretation of the results, so the argument that scripted testing doesn’t need skilled testers doesn’t hold water if you want to do scripted testing well. Indeed, good testing of any kind requires skill. If you or your testers are genuinely unskilled, training, coaching, mentoring, and a fault-tolerant environment that fosters learning will help to build skills quickly.

I’m also puzzled by the argument in another way: the (claimed) lack of a need for skilled testers is often presented as a virtue of scripted testing. This is like saying that the lack of a requirement for medical training is a virtue of quack medicine.

Level of documentation. Scripted testing generally flows from business requirements documents and functional specifications. When these documents do not exist or are deficient, it is very difficult to conduct scripted testing in a meaningful way. The shortcuts that would be required, such as scripting from the application as it is built, are accomplished as efficiently using exploratory testing.

Actually, there are several ways to do scripted testing in a meaningful way without business requirements documents or functional specifications being present or perfect. A standout example is test-driven development, in which checks are developed prior to developing the application code. Another: unit tests, in which checks are prepared at some point after the application code has been developed. Both forms of checks support refactoring. Another example: many Agile development shops use Fitnesse to develop, explore, and discover many of the requirements of the product, and to create scripted checks as development proceeds.

In fact, no one of whom I am aware has ever seen perfect requirements documents; they’re always deficient in some way for some person’s purpose, and always to some degree that’s intentional. It’s infeasible and wasteful to document every assumption; for programmers and testers with appropriate judgment and skill, it’s also somewhat insulting. Thus, whether developing either scripted or exploratory test ideas, we should treat imperfect documents as a default assumption.

Timeframes and deadlines.The lead-in time to test execution determines whether you can conduct test design before execution. When there is little or no time, you might at least need to start with exploratory testing while documented tests are prepared after the specifications become available.

Here is the implication, once again, that exploratory tests aren’t documented. Yet exploratory ideas can be documented in advance, in the form of checklists or as charters. Existing data files—a form of documentation—can be available in advance. Exploratory tests can be guided by marked-up diagrams, by user documentation, or even by a functional specification. Exploratory testing can be informed by any kind of documentation you like. One key distinction between exploratory and scripted approaches is not whether documentation is available in advance; with ET the difference is that the tester, rather than the document, is the primary agency driving the design and execution of the test. Another key distinction is that in an exploratory approach, detailed documentation of the tester’s actions tends to be de-emphasized prior to test execution, and tends to be produced during or very shortly after testing.

Available resources. The total number of person-days of effort can determine which approach you should take. Formal test documentation has significant overhead that can be prohibitive where resources and budgets are tight.

Well, at least maybe we agree on something. But the number of person-days of effort doesn’t determine which approach you choose; people do that. It’s unclear here as to whether the person-days are a given, or whether the test manager gets to choose. Moreover, I wouldn’t use the word “formal” here, since documentation associated with exploratory approaches can be quite formal (look at session-based test management as an example). The word I’d use instead of “formal” above is “excessive” or “wasteful” or “overly detailed”. I’d also suggest that it’s fairly rare for testers to feel as though resources and budgets are ample; most testers I speak with maintain that resources and budgets are always tight, for them.

Skills required. The skill sets of your team members can affect your choices. Good test analysts may not necessarily be effective exploratory testers and vice versa. A nose for finding bugs quickly and efficiently is not a skill that is easily learned…

Perhaps not. James Bach, Jon Bach, James Lyndsay, and I (to name but four) have a lot of experience training testers, and our experience is that the capacity to find bugs can be learned more quickly than many people believe. But habitutate testers to working from detailed manual test scripts and I’ll guarantee that they don’t develop skill quickly. In fact, a huge part of learning to find problems quickly lies in the tester being given the freedom and responsibility to explore and to develop his own mental models of the product.

Verification. Verification requires something to compare against. Formal test scripts describe an expected outcome that is drawn from the requirements and specifications documents. As such, we can verify compliance. Exploratory testing is compared to the test engineer’s expectations of how the application should work.

Many kinds of testing—not just verification—require something to compare against. That “something” is called an oracle, a principle or mechanism by which we recognize a problem. The claim here is that scripted testing is a good thing because we can verify “compliance” (to what?) based on requirements and specifications documents. Actually, what we’d be verifying here is consistency between some point in a test script and a specific claim made in one of the documents. There are two problems here. One is the problem of inattentional blindness; focusing a tester’s attention on a single observation drives the tester’s attention away from other possible observations that might represent an issue with the product. The second problem relates to the fact that requirements and specification documents (as above) can be presumed to contain errors, misinterpretations, and outdated information.

Exploratory approaches defend against these two problems by their emphasis on the tester’s skill set and mindset. Rather than depending upon a single oracle (that of consistency with a specification), an exploratory approach emphasizes apply several oracles, including consistency with the product’s history, with the image the development organization wants to project, with comparable products, with reasonable user expectations, with the intended purpose of the product, with other elements within the product, and with relevant standards, regulations, or laws. Claims—statements in the requirement documents—are taken seriously by the exploring tester, of course, but the other kinds of oracles provide a rich set of possible comparisons, rather than the relatively impoverished single set provided by a script. By keeping the tester cognitively engaged and under her own control, we lessen the risk of script-induced inattentional blindness. Statements like “Exploratory testing is compared to the test engineer’s expectations of how the application should work” are not only inaccurate, but also trivialize the complex set of heuristics that skilled testers bring to the game.

At this point, I’m in agreement with Jon Bach. Refuting the rest of the article would take more time than I’ve got at the moment. So, to keep the curious and eager people occupied:

Web Resources

Evolving Understanding of Exploratory Testing (Bolton)
A Tutorial in Exploratory Testing (Kaner)
The Nature of Exploratory Testing (Kaner)
The Value of Checklists and the Danger of Scripts: What Legal Training Suggests for Testers
Exploratory Testing Dynamics (James Bach, Jon Bach, Michael Bolton)
General Functionality and Stability Test Procedure (for Microsoft Windows 2000 Application Certification) (James Bach)
Experiments in Problem Solving (Jerry Weinberg)
Collaborative Discovery in a Scientific Domain (Okada and Simon)
– a study of collaborative problem-solving. Notice the subjects are testing software.

Books

Exploring Science: The Cognition and Development of Discovery Processes (David Klahr)
Plans and Situated Actions (Lucy Suchman)
Play as Exploratory Learning (Mary Reilly)
Exploratory Research in the Behavioral Sciences
Naturalistic Inquiry
How to Solve It (George Polya)
Simple Heuristics That Make Us Smart (Gerg Gigerenzer)
Blink (Malcolm Gladwell)
Gut Feelings (Gerg Gigerenzer)
Sensemaking in Organizations (Karl Weick)
Cognition in the Wild (Edward Hutchins)
The Social Life of Information (Paul Duguid and John Seely Brown)
A System of Logic and Ratiocination (John Stuart Mill)
Sciences of the Artificial (Herbert Simon)

Yes, learning about exploratory testing in a non-trivial way might take some practice and study. That goes for anything worth doing expertly, doesn’t it?

Want to know more? Learn about Rapid Software Testing classes here.

7 responses to “Comment on a Not-So-Good Article on Exploratory Testing”

  1. Simon Morley says:

    There is a suggestion/assumption that scripted testing doesn't require any more observational skills than those specified in the script/document being followed. Couldn't be further from the truth (IMHO).

    I think all testing requires what airline pilots call "situational awareness". (Blog post brewing…)

    I hadn't seen this article – but I'll definitely have a look (I don't restrict my sources of learning – whether it be the good, the bad or the ugly!)

  2. Thomas Ponnet says:

    I'd like to pick up the same point, that executing test scripts is seen as something that junior testers/inexperienced staff can do – no special skill required.
    I believe that one needs to train their observational skills to become effective in testing.
    But spreading the rumour that "anyone can execute test scripts" helps a huge area of testing companies who advocate to have one experienced test manager, some guys who are in the know who write the test scripts and the execution can be off-shored to save money.

    You wrote “But sapience and skill are required for the design and construction of the check, and for the analysis and interpretation of the results, so the argument that scripted testing doesn't need skilled testers doesn't hold water if you want to do scripted testing well. Indeed, good testing of any kind requires skill.”

    I often heard a Project/Programme Manager basically ask “How much does it cost more to test well?”. In other words, if we don’t outsource, how can you justify double the daily costs of a skilled tester?
    THAT is the problem myself and I guess many other people in our profession are facing. Good article, thanks

  3. Joey says:

    The article in question was published Feb. 25, 2003. That wasn't originally apparent on the article page itself, though the date does appear in search results on StickyMinds. We've since added it to the article for clarification. I'm not sure where the idea of its being a recent article comes from, and I hope the publishing date gives it some context.

    That doesn't mean it was accurate in 2003, but rather that there have been six years of additional thought put into the subject.

    Thanks for your response, Michael, and thanks for linking to it from the article.

    When I was a small-time film critic for a community newspaper, I discovered that I really disliked the idea of offering up my thoughts on a film if I couldn't get a response from readers. Not only because publishing criticism has an inherent vanity to it, but because I thrive on the two-way communication that comes with discussing art. It sometimes felt like shouting from the mountaintops, when I really wanted to be down in the village, chatting with differently minded folk over a beer.

    Lately, with things turning more sincerely from print to Web publishing, and with the prominence and promise of social networking, I'm starting to think that that idea has expanded beyond criticism to any area of writing. That is, most (if not all–at least nonfiction) writing benefits from a comment section.

    So, again, thank you. The publication date may (or may not?) have confused things a little, but the fact that you took the time to respond adds a lot of value to publishing in general. (At least in the opinion of this former small-time film critic.)

  4. Jim Knowlton says:

    I agree with most of what you're saying here…but don't you think there's a difference between exploratory testing (planned, scheduled, take lots of notes, defined sessions) and ad-hoc testing, which is much more unstructured? At the beginning of your critique you seem to imply that any user-driven exploration of the product is exploratory testing.

  5. Simon says:

    Thanks for the list of resources. Having been testing for a couple of years myself, I have found from personal experience that scripted testing, though it has its benefits, requires a lot of effort for not much reward. Exploratory Testing seems like a much more exciting and rewarding methodology and I for one will definitely be reviewing the articles and books you have recommended.

  6. Michael Bolton http://www.developsense.com says:

    @Jim…

    I don't use the term "ad hoc testing". You're welcome to use it if you find it helpful.

    But you're largely correct: testing is exploratory to the extent that the person doing the testing is in control of his or her process. Testing is scripted to the extent that someone else, from some time in the past, is in control of the tester's process.

    Here is exploratory testing, as I understand and teach it:

    http://www.developsense.com/2008/09/evolving-understanding-about.html

    Cheers,

    —Michael B.

  7. Immigration Services says:

    (Amusingly, this reply was provided by someone who signed him/herself as “danniseo@gmailcom”, and then plagiarized James Bach (http://www.satisfice.com/articles/what_is_et.shtml). But hey… what James has to say makes sense. —MB)

    I agree with Jim. Exploratory software testing is a powerful and fun approach to testing. In some situations, it can be orders of magnitude more productive than scripted testing. I haven’t found a tester yet who didn’t, at least unconsciously, perform exploratory testing at one time or another. Yet few of us study this approach, and it doesn’t get much respect in our field. It’s high time we stop the denial, and publicly recognize the exploratory approach for what it is: scientific thinking in real-time. Friends, that’s a good thing.

    Michael replies: Apropos of your remark, “I haven’t found a tester yet who didn’t, at least unconsciously, perform exploratory testing at one time or another,” I now have a response to those who claim they don’t do exploratory testing: Exploratory Testing is All Around You.

    Thank you for the comment. I wish you used your real name here, though.

Leave a Reply

Your email address will not be published. Required fields are marked *