Don't get me wrong: I could point to the study and say "Even with all the problems, the exploratory approach still beat the scripted approach by a factor of five." I'd love to do that. It would better than claiming that the approaches are equally efficient, which the study manifestly does not show. The strong points in the story are very favourable to exploratory approaches and the weaknesses unfavourable. My primary goal was to point out the misrepresentation of the study by B.J. in his recent writings. To be fair, some of that misinterpretation is founded in the idea that the authors themselves appear to have misinterpreted the results, claiming equal efficiency in test execution but missing the fact that testing incorporates not only execution, but also (at least) design and reporting.

So that raises a set of ethical dilemmas. Should I challenge the misrepresentations that are being spread in presentations and in articles and in the blogosphere? Should I provide an alternative view, cherry-picking only the bits of the paper that support my argument? Or should I do a critical review, in which I point out the good news but also acknowledge the flaws? In my view, B.J.'s fallacies shouldn't be ignored. The cherry-picking doesn't fit for me ethically; that option sounds like a sales pitch. The third is the only option that my ethics permit. Plus, questioning a product and finding problems with it is at the centre of being a tester, even if we'd be inclined to like that product. That's a credibility thing.

I'm really sorry, but I was very happy to find something (the only thing I found) that provided me with some ammunition.

But it's not the only thing you found, unless I misunderstand. You used something far more valuable than a questionable study. You say you tested and delivered outstanding results. If I read it right, you had some time, and you put it to use in exploration. I infer that you got some experience, and some results; probably you did some experimentation—which delivered more results that you used to negotiate a little more time, for even more and even better results. A virtuous cycle—right?

That's what we do, too. We don't appeal to academic studies that don't reflect real teams and testers well. We haven't found a good study. Instead, we appeal to our own experience; we appeal to your experience (that is, to the experience of our clients generally); and we appeal to your own experiments (ditto). I'll have more to say about that in my next blog post.

Just as you say,

I give you results, you give me the ok. I show you numbers, you give me the go ahead.

That's what we do. And for all of our work, if you don't like what we've done, we offer a money-back guarantee.

In your case, perhaps citing impressive results from an academic study helped; maybe the manager didn't notice the weaknesses in the study, or maybe s/he did notice and didn't care. As Cem points out, "The hallmark of useful research is whether it is useful." If it was useful to you, great. But as Cem also says, the research bugs in this study are, to me, showstoppers. Some products work for people even when the product has what the testers would consider a showstopper. Quality is value to some person, and you and I are different people. If it worked for you, I'm happy. Really.

Meanwhile, you also say…

"Give us $20,000, and we'll show you the promised land."

If you know of someone who is charging $20,000 for a three-day class and a day of consulting, and is getting it, please let me know; I'd love to find out how they're doing it. Better yet, if you know a client who's willing to part with that kind of money, please send contact information immediately.

—Michael B.

First of all, there is a massive difference between "unskilled" testers as it might apply to someone off the street, where you have absolutely no clue what their background is, and someone who is a computer science student. The foundation of software testing is in understanding computer application and understanding how an application SHOULD work. So let's not discount this aspect.

Secondly, I have seen no other attempt to bring "valid" data to the table. So aside from the, "Give us $20,000, and we'll show you the promised land." arguement, I have seen very little as far as empirical data from those who are the biggest proponents of this "approach". So, not for nothing, but a lot of what you argue here sounds like a sales pitch on your part, sorry.

I'm really sorry, but I was very happy to find something (the only thing I found) that provided me with some ammunition. In my situation, if I hadn't, literally, had time to kill on a project which ended up returning outstanding results, then ET would not be an approach available to me today. I mean we can't just sit down, smoke some pot and hit them with it when they're good and stoned and hope they say, "cool". The notion is, I give you results, you give me the ok. I show you numbers, you give me the go ahead.

I understand this is, again, what you guys are there to teach, for $20,000, but at the very least Juha's report provides a couple bullets. Now battle-hardened, by the book test vets who love their scripts might be able to shoot it apart. I mean blow it to pieces! However, it's something. Give us something! I have a much greater respect for someone who tries and fails than someone who says it cannot be done. In Juha's case, I don't view it as failing, though. I think it does provide some nuggets, as I call them. Some ammo that can be used for us who want to progress testing in a more agile way. Sure, there may be issues with it. I'm sure there can be issues with any research report, but everything we DO is subjective. So who cares?! If you don't like it, then give us something. ANYTHING!

As far as I'm concerned, it may not be the silver bullet, but nothing is. At least it's a bullet and not a guy telling me he has a bullet.

The hallmark of useful research is not that it is quantitative (some of the best research is qualitative).

Nor is the hallmark that it presents "facts." After all, "facts" to the original researcher are merely anecdotes to a person who reads about them. They are stories about what someone did, what happened, and what he thinks about it.

"Facts" presented in a research paper are observations of a researcher, recorded in way decided by the researcher (the research and recording methods are rarely well disclosed in a technical paper, partially because publication page limits don't allow enough room for details) which were then analyzed using methods chosen by the researcher (whose reasoning is not fully disclosed in the paper). A subset (chosen by the researcher) of the analyses are presented as “the facts.”

Sadly, many alleged facts in scientific papers are intentional fictions or distortions. Many other factual-seeming assertions are erroneous or misleading even though they are written in good faith, because of misperceptions or misrecollections by the author. For more information on this, read the extensive literature on experimenter effects in science.

Like anecdotes, scientific writing is often treated as credible on the reputation of the author for skill and integrity and on the extent to which the presentation of "the facts" is compelling.

In other words: facts, shmacts.

To a more mature researcher, there are more interesting questions than whether data are "facts" or the data collection and analysis techniques were quantitative rather than mixed-method or qualitative.

The hallmark of useful research is whether it is useful.

And for that, we need validity and generalizability, not quantitative "facts".

Have you ever watched kittens explore? Guided a kitten’s exploration with some string or a laserpen dot? I bet someone could do a quantitative research project on the relative effectiveness of different stimuli for guiding the exploration of 100 kittens.
We could publish the results in a book, so as to include all the pertinent data and methods. Methodologically and quantitatively, the book could be perfectly sound.

Let’s give this book a catchy title: "How We Study the Management and Training of Exploratory Testing at Microsoft."

Perhaps, though, this population studied might not reflect the characteristics of what the professional software testing community might consider the population of interest. That is, if we want to draw conclusions about software testers, then kittens (even though they explore and test soft things) might not provide a suitable basis.

Michael's assertion is that Juha studied the equivalent of kittens.

The response to Michael's critique appears to be that Juha's paper is well-written, it reports facts and numbers, and that the authors said "more research is necessary" so this is good science.

I have not read Juha's paper and am not expressing any opinion about the underlying paper. I am commenting on the exchange in this blog, and other instances I have noticed elsewhere recently of what seems to be a wielding of "fact"-based research as a bludgeon.

When I teach empirical methods in computing, my students critique research papers. An answer like "this paper is nicely written and it illustrates good science because it has facts and numbers and says 'more research is necessary'" would get an "F".

A more mature vision of scientific research would recognize that Michael was raising one of the classic "threats to validity" of empirical research. If a study does not credibly speak to the phenomena it purports to study, it is invalid. On the scale of research bugs, that would be a P1 showstopper.

Thank you for the comments.

Also, I think your assertion that the authors regard exploratory testing as a 'technique' is misleading.

I'm confused. In reading the text, I can't find such an assertion. Can you point me to it?

And, yes the studies use a lot of quantifiable data. Interesting thing about research; it supports its conclusions with facts rather then emotion.

That's the idea, at least. Is it your intention to suggest that quantifiable data is the only form of fact? Do you suggest that qualitative analysis is naturally or automatically non-factual?

Meanwhile, I hope that this post has been helpful in identifying areas that might inform your own studies. I'd be very interested in seeing some of Microsoft's work in exploratory and heuristic approaches, both from a training perspective and a practice perspective. I know about the General Functionality and Stability Test Procedure, Michael Hunter's You Are Not Done Yet list, and James Whittaker's Tour Stuff, but I'm not aware of anything from Engineering Excellence specifically. Any pointers?

Cheers,

—Michael B.

This is a well-written piece that provides perspective to on-going research by myself and other people in the area of exploratory testing in order to better understand its value and limitaions (or rephrased, which types of issues it is most effective at exposing, and which types of issues it is not as effective as exposing as compared to other approaches). It's that whole pesticide paradox thing, or usng the right tool for the job.

Also, I think your assertion that the authors regard exploratory testing as a 'technique' is misleading. Non-of the published papers by Juha or myself ever refered to exploratory testing as a technique. That would be incorrect because a technique is 'a systematic process to help solve a specific type of problem.'

I agree with you that techniques or patterns are often the foundation of how we design tests either from an exploratory or pre-defined test case (scripted) approach.

And, yes the studies use a lot of quantifiable data. Interesting thing about research; it supports its conclusions with facts rather then emotion. But, you are also quite right that data may not present the whole story or may slant the picture which is why we both suggest further research is required.