Blog: Why Exploratory? Isn’t It All Just Testing?

The post “Exploratory Testing and Review” continues to prompt comments whose responses, I think, are worthy of their own posts. Thank you to Parthi, who provides some thoughtful comments and questions.

I always wondered and in attempted to see the difference between the Exploratory testing that you are talking about and the testing that I am doing. Unlike the rest of the commenter’s, this post made this question all the more valid and haunting.

From what you have written, as long as there is a loop between the test design and execution, its exploratory testing? And the shorter the loop, exploratory nature goes up?

Yes, that’s right. A completely linear process would be entirely scripted, with no exploratory element to it. The existence of a loop suggests that the testing is to some degree exploratory. This suggests (to me, at least) a link to one of the points of Jerry Weinberg’s Perfect Software and Other Illusions About Testing. Testing, he suggests, is gathering information with the intention of informing a decision, and he also says that if you’re not going to use that information, you might as well not test. I’ll go a little further and suggest that if you “test” with no intention of using the information in any way, you might be doing something, but you’re not really testing.

As we’ve said before, some people seem to have interpreted the fact that there’s a distinction between exploratory testing and scripted testing as meaning that you can only be doing one or the other. That’s a misconception. It’s like saying that there are only two kinds of liquid water: hot or cold. Yet there are varying gradations of water: almost freezing, extremely cold, chilly, cool, room temperature, tepid, warm, hot, scalding, boiling. To stretch the metaphor, a test is it’s being done by a machine (that is, a check) is like ice. It’s frozen and it’s not going anywhere. An investigation of a piece of software done by a tester with no purpose other than to assuage his curiosity is like steam; it’s invisible and vaporous. But testing in most cases is to some extent scripted and to some extent exploratory. No matter how exploratory, a test is to some degree informed by a mission that typically came from someone else, at some point in the past; that is, the test is to some degree scripted. No matter how scripted, a test is to some degree informed by decisions and actions that come from the individual tester in the moment—otherwise the tester would freeze and stop working, just like a machine, as soon as he or she was unable to perform some step specified in the script. That is, all testing is to some degree exploratory.

In addition to the existence of loops, there other elements too. Very generally,

  • the extent to which the tester has freedom to make his or her own choices about which step to take next, which tests to perform, which tools to use, which oracles to apply, and which coverage to obtain (more freedom means more exploratory and less scripted; more control means less exploratory and more scripted);
  • the extent to which the tester is held responsible for the choices being made and the quality of his or her work. More responsibility on the tester means more exploratory and less scripted; more responsibility on some other agency means less exploratory and more scripted.
  • the extent to which all available information (including the most recent information) informs the design and execution of the next test. The broader the scope of the information that informs the test, the more exploratory; the narrower the scope of information that informs the test , the more scripted.
  • the extent to which the mission—the search for information—is open-ended and new information is welcomed. The more new information will be embraced, the more exploratory the mission; the more new information will be ignored or rejected, the less exploratory the mission.
  • again, very generally, the length of the loops that include designing, performing, and interpreting an activity and learning from it, and then feeding that information back into the next cycle of design, performance, interpretation, and learning. I’m not talking here so much about timing and sequences of actions so much as the cognitive engagement. Timing is a factor; that’s one reason one reason that we now favour “parallel” over “simultaneous”. But more importantly, the more difficult it is to unsnarl the tangle of your interactions and your ideas, the more exploratory a process you’re in. The more rapidly you are able to shift from one heavy focus (say on executing the test) to another heavy focus (pondering the implications of what you’ve just seen) to another (running a thought experiment in your head) to yet another (revising your design), very generally, the more exploratory the process. Another way to put it: the more organic the process, the more exploratory it is; the more linear the process, the more scripted it is.

Is this what you are saying? If yes, there is hardly any difference in what I do at my work and what you preach and this is true with most of my team (am talking about 600+ testers in my organization) and we simply call this Testing.

I’d smilingly suggest that you can “simply” call it whatever you like. The more important issue is whether you want to simply call it something, or whether you want to achieve a deeper understanding of it. The risk associated with “simply” calling it something is that you’ll end up doing it simply, and that may fail to serve your clients when they are producing and using very complex products and services and systems. Which is, these days, mostly what’s happening.

For example, is there really a difference between what I’m talking about and what are your 600+ testers doing? Can you describe what they’re doing? How would you describe it? How would you frame their actions in terms of risk, cost, value, skill, diversity, heuristics, oracles, coverage, procedures, context, quality criteria, product elements, recording, reporting? Is all that stuff “simply” testing? For any one of those elements of testing, where are your testers in control of their own process, and when are they being controlled? Are all 600+ at equivalent stages of development and experience? Are they all simply testing simply, or are some testing in more complex ways?

Watch out for the magic words “simply” or “just”. Those are magic words. They cast a spell, blinding and deafening people to complexity. Yet the blindness and deafness don’t make the complexity go away. Even though these words have all the weight of snowflakes, their cumulative effect is to cover up complexity like a heavy snowfall covers up a garden.

May be these posts should be titled “Testing” than “Exploratory Testing”?

There is already good number of groups/people taking advantage of the (confused state of the larger) testing community (like certification boards). Why to add fuel to this instead of simplifying things?

There’s a set of important answers to that, in my view.

  • Testing is a complex cognitive activity comprising many other complex cognitive activities. If we want to understand testing and learn how to do it well, we need to confront and embrace that complexity, instead of trying to simplify it away.
  • If we want our clients to understand the value, the costs, the extents, and the limitations of the services we can provide for them, we need to be able to explain what we’re doing, how we’re doing it, and why we’re doing it. That’s important so that both we and they can collaborate in making better informed choices about the information that we’re all seeking and the ways we go about obtaining that information.
  • One way to “simplify” matters is to pretend that testing is “simply” the preparation and then following of a script, or that exploratory testing is “simply” fooling around with the computer. If you’re upset at all about the certification boards that trivialize testing (as I am), it’s important to articulate and demonstrate the fact that testing is not at all a simple activity, or that comprehension of it can be assessed with any validity via a 40-question multiple choice test. Such a claim, in my opinion, is false, and charging money for such a test while making such a claim is, in my opinion, morally equivalent to theft. The whole scheme is founded in the premise that testing a tester is “simply” a matter of putting the tester through 40 checks. If we really wanted to evaluate and qualify a tester, we’d use an exploratory process: interviews, auditions, field testing, long sequence tests, compatibility tests, and so on. And we wouldn’t weed people out on the basis of them failing to take a bogus exam, any more than we’d reject a program for not being run against a set of automated checks that were irrelevant to what the program was actually supposed to do.
  • Just as software development is done in many contexts, so testing is done in many contexts. As we say in the Rapid Testing class, in excellent testing, your context informs your choices and vice versa. And in excellent testing, both your context and your choices evolve over time. I would argue that a heavily scripted process is more resistant to this evolution. That might be a good thing for certain purposes and certain contexts, and a not-at-all good thing for other purposes and other contexts.

Many people say, for example, that to test medical devices, you must do scripted testing. There is indeed much in medical device testing that must be checked. Problems of a certain class yield very nicely to scripted tests (checks), such that a scripted approach is warranted. The trouble comes with the implicit suggestion that if you must do scripted testing, you must not do exploratory testing. Yet if we agree that problems in a product don’t follow scripts; if we agree that there will be problems in requirements as well as in code; if we agree that we can’t recognize incompleteness or ambiguity in advance of encountering their consequences; if we agree that although we can address the unexpected we can’t eliminate it; and if we agree that people’s lives may be at stake: isn’t it the case that we must do exploratory testing in addition to any scripted testing that we might or might not do?

The answer is, to my mind, certainly Yes. So, to what extent, from moment to moment, are we emphasising one approach or the other? That’s not a question that we can answer by saying that we’re “just” testing.

Thanks again, Parthi, for prompting this post.

12 Responses to “Why Exploratory? Isn’t It All Just Testing?”

  1. Simon Morley says:

    Interesting question! And interesting reply!

    I pondered a little over the line:
    “More responsibility on the tester means more exploratory and less scripted; more responsibility on some other agency means less exploratory and more scripted.”

    I thought this could be a generalisation and so thought about a counter-example:

    If a tester was asked to execute 1 test case and came back with the result of it, plus 2 other variants giving some additional information about the product – then although he’s been given a limited scope (and the implication was less exploratory possibilities) he’s returned with additional information. To him, the limited scope could be interpreted as his mission – and he “tested” to that scope.

    Of course, he could interpret it literally – especially if he had a script to go with the test case – and be testing with his “eyes wide shut” instead “eyes wide open”.

    I sometimes introduce the idea of exploratory approaches with non-testers – and we discuss – they usually form the opinion that it’s a challenging activity – correct, I say, any testing done well (whether it’s completely or partially scripted, whether doing standups and debriefs before starting a new test session – all of this is a challenging activity.)

    Let’s test with our “eyes wide open” I say.

  2. Smitha says:

    This is a great post on agile testing.

  3. Very interesting article Michael.

    I have been playing around with a couple of ideas about a blog article on hybrid testing – where you mix non scripted and scripted tests – this might spur me on to get it completed.

    You are correct about the confusion within the field and people taking advantage of this. It concerns me that this is happening in our craft and that people thing you either belong to the scripted or non-scripted school. I have never read anything by yourself or James in which you say that there is not a need still to carry out scripted tests or carry out document review. The one clear message I get from yourself and James is that anything that hinders testing is wrong and should be corrected.

    regards

    John

  4. Shiv says:

    I having been following ET for quite some time now and it also helps me in my testing.

    Michael replies: Hi Shiv. Thanks for writing.

    I feel that we have been doing testing from long time, but the things we lacked was order or a structured way to relate it and explain to others.When we discuss about testing there are many answers to one question or many ways of doing one thing. ET I think brings all the pieces together and gives a structure so that it gives confidence when you speak what you are doing.

    For example, I see many testers do testing but are not able to explain what they have done. If you compare with a tester who does ET you can see the difference.

    I have read about the stop heuristics in one of your blogs, when I read that I felt I do/use most of these things in my work but I use them because I was told by my lead that you should use it this way because we are successful by using this in this way. After seeing your list I feel that there are otherways as well. This is what lacks If you are a traditional tester.

    I’m glad to hear that you’re considering other factors. Have you had a conversation with your lead about it? (Have you pointed him/her to my blog?) It’s those people that we’re really trying to reach, don’t you think?

    It’s very interesting situation with the evolution of ET for few testers who might be doing ET for quite some time but they don’t know that they do ET.

    “Its kind of understanding the meaning of a song sung in language, which you don’t know, after you have sung it well and got applause” :)

    That’s a nice way of putting it. As well as you might sing it without knowing what it means, you’ll sing it even better when you do know.

    Excuse me for my eng as I am still learning to write better

    The only thing that I’d question about that is your use of “eng” instead of “English”. Other than that, there are no showstopper bugs with your English; there are easy workarounds for whatever problems there might be.

  5. Matt says:

    I really like the idea that Exploratory Testing and Automated Testing aren’t mutually exclusive. It seems like at least some people consider them (from an ideological perspective) to be polar opposites, when really all testing methodologies are different tools for different jobs/goals.

    In at least some environments, the number of people who are not familiar with the concepts and terminology of testing that have visibility and input in to the test team can be high. Could you expand on your idea about testing as a “complex cognitave activity”, specifically around communicating that complexity to audiences who lack that background?

    Michael replies: Oh, lordy. First of all, it’s not my idea, really; it’s the way things are, so far as I’m concerned. I don’t have time to give the answer I’d like to give, but at the moment, to be helpful immediately:

    Software Testing as a Social Science (Cem Kaner)
    The Case Against Test Cases (James Bach)
    Manual Tests Cannot Be Automated (James Bach)
    Is There A Problem Here? (Michael Bolton)
    How Testers Think (Michael Bolton)
    Testing Without A Map (Michael Bolton)
    Perfect Software and Other Myths About Testing (Jerry Weinberg)

    These should get you started.

  6. Shiv says:

    @Michael: Yes I agree, If we see value in something definitely it must be shared with others.

    Reg ‘eng’, I ll try to fix it so that it won’t reproduce again :)

  7. Chris says:

    “As we’ve said before, some people seem to have interpreted the fact that there’s a distinction between exploratory testing and scripted testing as meaning that you can only be doing one or the other. That’s a misconception.”

    The #1 misconception and the one that is causing so much thrash and keeping the conversation trapped in time. I suppose continuing to point this out will get us unstuck, but it is frustrating in the short term. Thanks for the post.

  8. Selim Mia says:

    some people seem to have interpreted the fact that there’s a distinction between exploratory testing and scripted testing as meaning that you can only be doing one or the other. That’s a misconception.
    With that misconception some testers are fonder to call themselves as a “Exploratory Tester”, are not all human testers engage in some level of exploratory testing? I’ve seen many testers doing some level of exploratory even don’t knowing about the term “Exploratory Tester”, even myself when i first learn/know about Exploratory Testing i applause that me and my other colleagues practicing some level of ET from the beginning of testing career though we were in the process of traditional scripting approach of testing.

    If i call myself as a Exploratory Tester does it mean that i will never do any scripted testing irrelevant in any circumstances? I don’t think so and i agree to your words “testing in most cases is to some extent scripted and to some extent exploratory”, that may depends on the certain contextual factors and/or unavoidable circumstances.

    - Selim

  9. [...] Ideas on Testing – What is “Testing” really? [...]

  10. Nilanjan says:

    For a person who is a good tester, and who doesn’t know the description/definition of exploratory testing, it is useful to point out that all testing is a combination of exploratory and scripted (in varying degrees).

    However, I think in many organizations there is a culture of scripted testing. Testers may not have the motivation to “explore”, in fact may be discouraged to do so. There are many organizations, certification boards which add to the confusion.

    Michael replies: I think you’re right about that. And I think we have to engage with those people, and persuade them that there is indeed a different way to think about things. In particular, I think it’s important to emphasize that giving a script to a human is not only inefficient, but immoral. To my mind, it’s not right to give people a task that a machine could perform. Meanwhile, giving a human a task to perform—an investigative, interactive, open-ended task—is appropriate, and affords the opportunity for discovery of problems that machines couldn’t possibly discover.

    For these groups, I think it is important to clearly define exploratory testing as has been done in the last decade (?). The importance of defining ET is more for these pathological groups rather than the target of this blog post, i.e., someone who is a good tester only doesn’t know what it’s called.

    I agree. So we need your help, and the help of the rest of the community. We need you to help us recruit the good testers to whom you refer, and we need you to recruit the managers who might recognize the value of excellent testing. That might mean any number of things: doing a presentation for them, encouraging them to purchase Lessons Learned in Software Testing or Perfect Software, pointing them to our blogs and to the work of the skilled testing community. Rest assured that you can contribute.

    In a group I worked with – there were 10 testers – 1 of them was great – he never knew there was something called ET. 9 others never really got it. That didn’t prevent them from having lengthy careers in testing. If the 9 had got some mentoring/coaching on ET, maybe a few could have redeemed their careers in test.

    Again, you can help them with that. Please welcome them to the conversation!

    My 2 cents.

    Don’t underestimate your value! Cheers!

  11. Matt says:

    Thanks Michael! Those will definitely help me state my case with an audience that doesn’t do testing.

Leave a Reply