Blog: Exploratory Testing and Review

The following is a lightly-edited version of something that I wrote on the software-testing mailing list, based on a misapprehension that we who advocate exploratory testing suggest that review or other forms of testing should be dropped.

Exploratory testing was, for many years, described as “simultaneous test design, test execution, and learning”. In 2006, a few of us who have been practising and studying exploratory testing got together to exchange some of what we had learned over the years, and to see if we could work on refining the definition. I did a presentation that described some of my experience at those meetings. Cem Kaner wrote a synthesis of our ideas, and several of us who were there (and many who weren’t) have since explicitly agreed with it.

Exploratory software testing is a style of software testing that emphasizes the personal freedom and responsibility of the individual tester to continually optimize the value of her work by treating test-related learning, test design, test execution, and test result interpretation as mutually supportive activities that run in parallel throughout the project.

I use that definition when I want to be explicit. Much of the time, though, I keep things shorter. I still use something close to the older defintion, with one minor change.

There’s a problem with the word “simultaneous”. When we say “simultaneous”, people seem to think that means that everything is happening at the same time and to the same degree; that is, all three of design, execution, and learning are turned up to 10, all the time. Some people believe that exploratory testing is something that happens only as direct, hands-on interaction with a working product, only after code has been written and compiled and linked. But that’s only one of the times at which something can be explored. In fact, exploratory approaches can be applied to any idea or artifact, at any stage of devlopment of the product or service. That means you can be emphasizing test execution and learning, and relaxing emphasis on test design for a while; you can be designing a test and learning from that, while not executing the test immediately. You can be designing and executing in very short cycles, but learning less now than you might learn later. So, for that reason, we’ve started to say “parallel”, rather than “simultaneous”.

One of the things that I get from Cem’s synthesis is the notion of mutually supporting activities. The traditional, more linear approaches suggest that excellent test execution depends on excellent test design. It’s hard to disagree with that. But excellent test design—and improving test design— depends on feedback from test execution, too. In general, when the loops of design, execution, and learning are shorter, the feedback from one can inform the others more quickly. But that’s not to say that you can’t design a test and then wait to act on it, if that’s the most appropriate thing to do for the moment. However, when there are very long loops (or no loops), then you’re working in a scripted way, rather than an exploratory way. Shorter loops mean that testing is more exploratory.

In addition, something is more exploratory when an individual or a group of people (rather than a process or a script) is in charge. You can do test design and test planning in a less exploratory way by mixing it with a only little test execution. You can do test design and test planning in a more exploratory way by mixing it with a lot of test execution. (Even in a heavily scripted process, that exploratory activity happens a lot without people noticing it, so it seems.)

For example, review is a testing activity—questioning a product in order to evaluate it. There are scripted and exploratory forms of review. Consider code review. A completely scripted form of code review is a static analysis tool that looks for problems that it has been programmed to identify. A more exploratory form of code review is a bunch of people looking over a couple of pages of code, looking for specific problems that have been outlined on a checklist. A still more exploratory form of code review is a bunch of people looking for problems from a checklist, but also looking for any other problems that they might see. Perhaps the most exploratory form of code review is pair programming—people looking over code that is sort-of working, creating unit checks, revising the code, running the checks, and iterating right then and there.

Other forms of technical review can take the same arc. In the most scripted form, people receive (say) a functional design document, run it through a spelling and grammar checker, and sign off on it—and that’s the only review of the document that ever happens. In a less scripted form, people receive the design document and review it, comparing it to a list of specified requirements and quality criteria. In a more exploratory form, people look at examples or a prototype of various functions, and discuss what they’ve seen; at the end of the conversation, the designer takes the notes away and goes back to build a new prototype. In an extremely exploratory form of design, people sit around a projector and work on a Fitnesse page, raising ideas and concerns, discussing them, resolving them, and updating the examples and notes on the prototype in real time.

No one who talks seriously about exploratory testing, so far as I know, talks about getting rid of review. What we do talk about is getting rid of things that waste time and mental power by introducing interruptions, needless documentation, and processes or tools that over-mediate interaction between the tester and the product. Don’t get rid of documentation; get rid of excessive amounts of documentation, or unhelpful documentation. Don’t test thoughtlessly, and don’t get rid of thinking; get rid of overthinking or freezing in the headlights. Don’t get rid of test design; shorten the feedback loops between getting an idea and acting on an idea, and then feed what you’ve learned through action back into the design. Don’t control testers’ activities though a script; guide a tester with concise documentation&charters, checklists, coverage outlines, or risk lists—that help the tester to keep focused, but that allow them to defocus and investigate using their own mindsets and skill sets when it’s appropriate to do so.

Testing is investigation of a product. Investigation can be applied at any time, to any idea or artifact. That investigation is ongoing, and it comprises design, execution, and learning. From one moment to the next, one might take precedence over the others, but which one is at the fore can flip at any instant. What distinguishes the exploratory mindset from the scripted mindset is the degree to which the tester, rather than some other agency, has the freedom and responsibility to make decisions about what he or she will do next.

8 Responses to “Exploratory Testing and Review”

  1. Lynn McKee says:

    Thanks for the clarification on Exploratory Testing particularly addressing several of the negative misconceptions. I look forward to another round of the Rapid Software Testing course in Calgary to help further the understanding of ET.

    Cheers,
    Lynn

  2. Rahul Gupta says:

    Thanks for sharing this Michael. One doubt which is lingering in my mind for quite sometime now,”Can exploratory testing be automated?” If yes then upto what extent? While I do exploration (Investigation) on a product, I do whatever comes to my mind by thinking in reverse direction as how this piece of functionality would break? I am not sure if my approach is correct but so far it’s been working for me.

    I read somewhere on Srinik’s blog that automating exploratory testing is an oxymoron, is it so? I see a general perception among junior testers (even among ignorant seniors) that in exploratory testing, there are no scripts (read test cases) to follow but first version of the definition i.e. “simultaneous test design, test execution, and learning” talks about test design also, which I have been following by writing basic test cases, building my understanding and then observing the application’s behavior once it is done, I move back to update the test cases and this continues till stakeholders agree with state of the application.

    Please guide if it is what you call exploratory testing or my understanding of exploratory testing needs modifications.

    Michael replies: Thank you for commenting. These questions are worth a blog post on their own.

    Regards,
    Rahul Gupta
    http://BugMagnate.blogspot.com

  3. Adam Knight says:

    Michael,

    Thanks for the post, I’ve asked one of my team to prepare a presentation on reviewing use of ET in our organisation and have forwarded this post on as an excellent source of clarification on some key elements.

    Regards,

    Adam.

    Michael replies: Happy to help out, Adam. There’s more to come, too.

  4. Matt says:

    Thanks Michael! This is actually very timely for me, as I’m working on embedding Exploratory Testing as part of the overall test efforts at my office.

  5. Shrini Kulkarni says:

    How about creating a Exploratory Testing manifesto? possibly have a website that is dedicated to this …

    So whenever people get confused about what ET is and what is NOT …

    Shrini

    Michael replies: I point people this blog post on evolving understanding of exploratory testing and to a list of structures of exploratory testing on my Web site. And to Cem Kaner’s Web site. And to James Bach’s Web site. And Wikipedia’s article on exploratory testing, which gets it pretty much right.

  6. Parthi says:

    Hi Michael,
    I always wondered and in attempted to see the difference between the Exploratory testing that you are talking about and the testing that I am doing. Unlike the rest of the commenter’s, this post made this question all the more valid and haunting.

    From what you have written, as long as there is a loop between the test design and execution, its exploratory testing? And the shorter the loop, exploratory nature goes up?

    Is this what you are saying? If yes, there is hardly any difference in what I do at my work and what you preach and this is true with most of my team (am talking about 600+ testers in my organization) and we simply call this Testing.

    May be these posts should be titled “Testing” than “Exploratory Testing”?

    There is already good number of groups/people taking advantage of the (confused state of the larger) testing community (like certification boards). Why to add fuel to this instead of simplifying things?

    Michael replies: Hi, Parthi. This comment deserved its own post too. Thanks.

  7. [...] particular, Michal Bolton’s excellent post http://www.developsense.com/blog/2010/09/exploratory-testing-and-review and James Bach’s much more blunt and arguably more impactful post [...]

  8. [...] particular, Michael Bolton’s excellent post http://www.developsense.com/blog/2010/09/exploratory-testing-and-review and James Bach’s much more blunt and arguably more impactful post [...]

Leave a Reply