Subject: Another Sporadic Newsletter from DevelopSense
Things Can Be Different
Another in a Series of Sporadic Newsletters From DevelopSense
November 2008
In this issue...
It's Another Newsletter!
My First Book Contribution
A Different Approach to Process Improvement
Off to EuroSTAR!
Perfect Software Released!
Evolving Understanding About Exploratory Testing
It's Another Newsletter!
Every now and again, people ask me, in email or in person, "Are you still doing the newsletter?  I haven't received one in a long time."  The answer is that this is to be expected; I don't put this newsletter out on a schedule.  I really only prepare one when I've got a number of newsworthy things to say all at the same time.  I blog at   I also write a monthly column for Better Software Magazine, which I try to keep up to date at  And I write altogether too much on a several online forums, particularly the Agile Testing mailing list and the Indian testing forum  This newsletter is sporadic precisely because I don't know what's going to come up, and when.  In addition, I don't know how people like to get their news.  Still, it occurs to me that I should ask, and maybe you'd like to tell me:  would you prefer a regular schedule?
My First Book Contribution
I'm delighted to announce that my first book publication has been released.  A Gift of Time, published by Dorset House, is a book, edited by my colleague Fiona Charles honouring the life and work of Jerry Weinberg, who recently celebrated his 75th birthday and 50th year in the computing business.  (He also published a book himself this year, about which more below.)  Fiona collected essays from a number of Jerry's colleagues and students, including Robert Glass, James Bach, Sherry Heinze, Sue Petersen, Esther Derby, Willem van den Ende, Judah Mogilensky, Naomi Karten, James Bullock, Tim Lister, Johanna Rothman, Jonathan Kohl, Bent Adsersen, Jerry's wife Dani Weinberg... and me.  I've not seen all of the essays yet, but the ones that I've seen are wonderful and a fitting tribute to a man who has been so influential for all of us.

The book made its first public appearance today, November 4 2008, at the AYE Conference in Phoenix.  It's so new that it doesn't yet appear on the Dorset House web site, but it will be up and available for ordering soon.

Many thanks to Fiona Charles and to Wendy Eakin at Dorset House for their work in producing the book.

And to Jerry:  Happy birthday, arrigato, and namaste!

Positive Deviance

The most noteworthy thing that I've heard of recently is a set of parallels between our world of software development and the history of medicine.  The doctoring business has big, expensive, health- and life-critical problems to solve.  One might expect that, considering the current attention, the problems are new.  One might assume that the solutions are complex.  And one might assume that the optimal approach would be to have armies of experts descending on the profession and telling the underlings what to do.  According to Atul Gawande-a surgeon, a staff writer for the New Yorker, and the author of Better, a book on performance in medicine-none of this need be the case.
Ignace Semmelweis was an obstetrician and an empiricist in Vienna.  Around 1847, he observed a problem:  that puerperal (or "childbed") fever was killing over 20% of the mothers in his hospital's maternity ward, where for women who delivered babies at home the death rate was closer to 1%.  He came up with a simple solution to the problem:  he ordered every doctor and nurse in his ward to scrub with chlorine and with a nail brush between each patient visit.  This one practice dropped the death rate on his ward down to 1%.  Simple approaches can have a huge impact.
You might think that other doctors would have paid attention, but they didn't want to play.  Part of the reason was that Semmelweis's approach was spectacularly abrasive; he harangued people who forgot to scrub, and repeatedly accused doctors of killing their patients when they didn't comply.  This reminded me of Jerry Weinberg's "team-breaking exercise" at CAST 2008-set people to the task of telling other people exactly what's wrong with them and exactly what they have to fix.  Quality police may have good intentions, but neglect the niceties of human contact, and the idea-no matter how compelling-dies before it can do good.
Contrast this with the approach developed by Jerry and Monique Sternin to the address the problem of malnutrition in villages in Viet Nam.  They asked mothers in the community note whose children were best-fed and healthy, and then encouraged those mothers to visit and observe what the Sternins call positive deviations from the norm, or positive deviance.   The mothers with healthy kids were breaking with the conventional wisdom in minimal, easy-to-accomplish ways, making slight changes to diets and feeding patterns.  In each village, the incidence of malnutrition decreased 65% to 85% within two years.
In 1860, Joseph Lister identified the reason for Semmelweis' success and made a more successful case for prevention of infection, including handwashing.  Even so, almost 150 years later, infections in hospitals are still a problem, particularly with respect to "superbugs"-antibiotic-resistant bacteria-because doctors, nurses, and attendants are still not washing their hands as often as they might.  In 2005, the Sternins were asked to consult at a hospital in Pittsburgh, PA, where a number of initiatives had been tried to reduce the transmission of Methicillin-Resistant Staphylococcus Aureus (MRSA), a superbug.  Rather than prescribe a set of rules, the hospital administration followed the positive deviance approach, asking the workers what they reckoned should be done.   There were some logistical suggestions-making more antiseptic dispensers available, and in more convenient locations, for example-and some behaviour modification, such as persuading recalcitrant colleagues to don gloves and wash hands, and setting up mutual pacts to speak up to colleagues who didn't follow the protocols.
The bulk of the ideas were not terribly innovative, but they came from the people who needed to implement them, and they were practical and achievable.  And they were overwhelmingly successful. You can see the level of the ideas yourself.  For an example, look at the wonderful video of Jasper Palmer at  In less than two minutes, he demonstrates a safe and easy method that he developed himself for removing a hospital gown and gloves that are potentially contaminated.
One key to success was that the workers agreed to speak out when they saw others engaged in risky practices; another was that the workers were being consulted and encouraged to innovate for themselves, learning from one another; a third was that the improvements were based on practical things to do, rather than on abstract mandates or principles; a fourth was that results and progress were publicized within the department.  Within a year, the rate of MRSA transmission in the hospital had dropped to zero.
As I read this, it reminded me of John Seely Brown and Paul Duguid's book, The Social Life of Information.  To improve its field service, Xerox created training courses, detailed product documentation, and a clearly-defined process for the field technicians to follow.  One researcher, though, decided to follow the technicians around and observe what they actually did.  He found that the diagnostic and repair problems were much messier and more complex than the experts anticipated, and the simple, step-by-step approaches devised by the process gurus didn't work.  The copiers were quirky, the office environments were diverse, and the problems were often unexpected-and therefore not covered by the field manuals.  Moreover, the documentation gave them step-by-step procedures to follow, but didn't provide the motivation-why certain steps were worth the effort.  The technicians were forced to make sense of their situations, even when their maps didn't help.  So what did they do?  Mostly they chatted.  They ate together, meeting for breakfast, or coffee, or lunch; they played cards; and even when they appeared to be doing other things, they talked about work.  They exchanged knowledge, where the process and repair manuals only gave them information.
When Quarterdeck was at its best-the years in which I was a support person, a tester, a program manager, and a programmer-the company became rich enough to buy other companies and absorb their product lines.  In addition programmers were consistently developing new features and technologies, and adding them to existing products.  We had training sessions for those, but the real magic of the company was the way in which the culture supported knowledge, not just information.  Programmers and support people worked very closely together; sales people asked questions-and often provided answers-in the internal technical support forums.  People tended to know what each others' specialties were.  In my later years, we established science fairs, in which anyone-whether a programmer, support person, or tester-showed off some toy or technology that they had developed, discovered, or mastered.  Although supported from the top, the culture was fostered at all levels.
So:  if you want to improve processes in your own shop, consider an alternative to "maturity" and heavyweight process improvements.  Maturity isn't about following rigid processes; it's about adapting flexibly and responsibly to changing and unpredicted situations.  Plus, as Jerry Weinberg says, no one is going to read a 17-volume process manual anyway.  Instead, consider watching what people actually do, listening to what they say to one another, and learning from them.  Set up science fairs or peer conferences in your own shop or in your local testing community (look at the LAWST model, described at  Notice that no one who is effective follows their process from a script or a recipe, and notice that most of what's going on is in their actions and in their heads.  As we've long said in the Rapid Software Testing course, only a fraction of a skilled tester's work can be put onto paper.  It's what you-and your colleagues-think and what you do that counts.
Off to EuroSTAR!
EuroSTAR Logo I'm going to be at EuroSTAR November 10-13.  I'll be presenting at three sessions.

On Tuesday, November 11, I'll be leading a session titled Questioning Testing Myths: Critical Thinking Skills for Testers.  In this, we're going to examine and question some common testing bromides:  "Every test must have an expected, predicted result." "Effective testing requires complete, clear, consistent, and unambiguous specifications."  "Bugs found earlier cost less to fix than bugs found later." "Testers are the quality gatekeepers for a product."  "Testing at boundary values is the best way to find bugs."  These are often touted as Great Truths about testing, when they're really heuristics.  We get into trouble as testers when we take such statements uncritically.  In the session, we're going to explore some common cognitive errors and biases, and some thinking tools-questioning skills, critical thinking, context-driven thinking, general systems thinking-that can help testers deal confidently and thoughtfully with difficult testing situations.

On Wednesday morning, I'll be leading a session called Heuristics: Solving Problems Rapidly. Heuristics are fallible methods for solving a problem or making a decision.  People often make-and follow-economical, "fast and frugal" approaches to solving problems, rather than using more deliberate or thorough reasoning. We base our decisions on rapid cognition, snap judgments, and imprecise decision rules. The surprising fact is that these approaches can often lead to better decisions because of limited information, rather than in spite of it.   For one thing, heuristics lead us to more rapid thinking and activity, which can generate the information that we need at a decreased level of noise, where more complex analysis delays action and adds noise to the critical information.  Why is this important for testers?  Among other things, heuristics inform our test oracles-principles or mechanisms by which we recognize a problem.  In the session, we'll talk about heuristics and do an exercise or two to help us to learn about heuristic approaches.

The theme of this year's EuroSTAR conference is "The Future of Software Testing".  Thursday afternoon, I'll be presenting a talk I've given in various forms before:  Two Futures of Software Testing.  One vision that I have of the future is decidely a dystopia, with testers as quality gatekeepers, with testing following a rigourously controlled, bureaucratic, and essentially clerical process where exploration and investigation is frowned upon and considered "immature". Senior testers work from extensively detailed specifications, creating plans, drawing state diagrams, and compiling test scripts.  Less experienced testers are relegated to following the scripts. Changes to the product are resisted, and testers drive that resistance. The worst thing about this vision of the future is that it's so much like today, and that so many people appear to desire it.  In the other view of the future, testers are active investigators and critical thinkers who neither accept nor desire responsibility for releasing or otherwise managing the product; instead, they provide important, timely, credible information to managers so that they can make sound and informed business decisions. Testers work collaboratively not only with the developers, but with the entire project community, and are able to report at any time on product and project risks. Testers have an extensive understanding of tools and automation, and decide when-and when not-to use them. Most importantly, testers embrace challenge and change, adapting practices and strategies thoughtfully to fit the testing mission and its context.

I'm looking forward to meeting colleagues at the conference, especially some of the new friends that I made last year.  If you're there, please make sure to say Hello!
Perfect Software Released!
Perfect Software Cover Made you look!

The perfect software in question isn't a software product; it's Jerry Weinberg's latest release, Perfect Software and Other Illusions About Testing.  This is a fabulous book, and a very important one for the community of skilled testers.  It joins Lessons Learned in Software Testing in my list of books that every tester should own.  Moreover, it's a book that we should should pass on to our clients.

Jerry has been involved in software development longer than many of us have been alive.  In fact, his first book was released in 1961, the topsy-turvy year in which I was born.  Since the 1960s, he's been drawing attention to the human aspects of software development, writing or co-authoring more than 40 books.  His definition of quality--value to some person--is a notion that strongly informs context-driven testing and thinking.  One of his descriptions of a tester--a tester is someone who knows that things can be different--gives this newsletter its title.

Forests have been felled to tell us about testing techniques that reveal information about our products.  Too few testing books talk sensibly about value.  Too few books mention anything at all about the sheer volume and quality of information that we can obtain without a finger even hitting a keyboard.  We can learn critically important things by paying attention to people--what they say and how they behave.  An example:  consider the resistant response that we get when we even propose the idea of technical reviews.  I believe that Jerry would argue that the proposal is a kind of test, and that the response is a test result.  What might that resistance tell us about the state of the project?

The book was launched at the CAST conference in July, and made available to the general public near the end of August.  Okay, so this isn't exactly a breaking story.  Dorset House is a small publisher, and the quality of their catalog greatly exceeds the reach of their marketing--so it's up to us.  Grab a copy of the book for yourself and spread the word!
Evolving Understanding About Exploratory Testing

Compass This is an update of a blog post that I wrote in September 2008.

One of the highlights of CAST 2008 was Cem Kaner's talk "The Value of Checklists and the Danger of Scripts: What Legal Training Suggests for Testers."A big part of the talk was the contrast between exploratory and scripted processes, wherein Cem contrasted scripts-canned, step-by-step instructions that are executed more or less automatically, whether by an automaton or a human acting like one-and checklists-sets of things that we might want to consider, where the decision to consider them is under the control of the person using the list. For me, what was most valuable about the the talk was the evolving story of the nature of exploration and exploratory testing. So, as of October 31, 2008, here's a report on my version (as usual, strongly entwined with Cem's and with James Bach's) of the story so far. One goal of this exercise is to be able to point people to this post instead of repeating myself in testing forums.

Testing is questioning the product in order to evaluate it; that's James' definition. For Cem, exploratory testing these days is (deep breath, now) "a style of testing that emphasizes the freedom and responsibility of the individual tester to continually optimize the quality of her work by treating test design, test execution, test result interpretation, and learning as activities that continue in parallel through the course of the project." This was the definition that he synthesized around the time of the Workshop on Heuristic and Exploratory Techniques in Palm Bay, FL, May 2006. (James Bach, Jonathan Bach, Scott Barber, Tim Coulter, Rebecca Fiedler, David Gilbert, Marianne Guntow, James Lyndsay, Robert Sabourin, Adam White, Cem, and I were participants in that workshop.) The short version-James'-is "simultaneous test design, test execution, and learning." The two definitions are intended to mean the same thing. One is more explicit; the other is quicker to say.

The opposite of exploratory testing is scripted testing. Neither of these is a technique; they're both approaches to testing. Irrespective of any other dimension of it, we call a test more exploratory and less scripted to the extent that
  • elements of design, execution, interpretation, and learning are performed by the same person;
  • the design, execution, interpretation, and learning happen together, rather than being separated in time;
  • the tester is making her own choices about what to test, when to test it, and how to test it;
  • the tester makes her own choices about any automation or tools in support of her testing, from depending entirely upon tools to using none at all, as she sees fit;
  • everything that has been learned so far, including the result of the last test, informs the tester's choices about the next test;
  • the tester is focused on revealing new information, rather than confirming existing knowledge about the product;
  • in general, the tester is varying aspects of her tests rather than repeating them, except where the repeating aspects of the test are intended to support the discovery of new information.
Whether a test is a black-box test (performed with less knowledge of the internals of the code) or a glass-box test (performed with more knowledge of the internals of the code) is orthogonal to the exploratory or scripted dimension. A white-box test can be done either in an exploratory or a scripted way; a black-box test can be done either way too. The key here is the cognitive engagement of the tester, and the extent to which she manages her choices and her time.

Another common misconception is that exploratory approaches can only be applied as we're interacting directly with the product.  As Jerry Weinberg points out in Perfect Software, many important tests can happen without fingers hitting a keyboard or automation being run. Exploratory approaches can be applied to any aspect of the development process, and not just to test execution (that is, configuring, operating, observing, and evaluating the running product).  We can explore during the process of review of test plans, requirements documents, functional specifications, code, end-user documentation, or marketing materials.  In any of these circumstances, what we discover contributes to our decisions about what to do next.  Review can be done in exploratory way or in a scripted way, to the extent that the reviewer controls his own choices about what to observe. For example, a review may be done freestyle (in which case it might be highly exploratory); under the guidance of a set of ideas, perhaps in the form of a checklist (in which case it may be somewhat more scripted and somewhat less exploratory); or under the control of a strict set of conditions such as those used by static code analyzers (in which case it's entirely scripted).

Automation ("any use of tools to support testing") can be used in a scripted way or in an exploratory way. If the tool is being used to reveal new information, or if the tester is using the tool such that activities or data are varying, then the test tends to be more exploratory.  If the tools are being used to confirm what we already know, or if the tool is supporting repetition of tests, then the test is tending towards the scripted.  Note that these are heuristics.  Exploration may be served by repeating a test, especially if we're exploring state-related problems, and tools can help us with that.  A test that we think of as heavily scripted may have some variable elements in it--consider randomized high-volume tests.  In this case, the scripted dimension is dominant during test execution, but the exploratory dimension might become more prominent in the evaluation and interpretation of the result.

For the programmers, testing (questioning the product in order to evaluate it, remember) is a natural element of pair programing, and since the design, execution, interpretation, and learning are highly integrated, pair programming is an exploratory process. TDD has a strong exploratory element for the same reason. Indeed, Elisabeth Hendrickson quotes one programmer's "aha!" moment about ET: "I get it! It's test-driven testing!" I had a similar "aha!" moment when I realized that TDD, done well, was a highly exploratory style of development.

For the inevitable people who will inevitably ask "How is exploratory testing different from ad hoc testing?", I'll start by replying that I can't make the distinction until you've provided me with a notion of what "ad hoc" means to you. Some people believe that "ad hoc" is a synonym for "sloppy" or "slapdash". Exploratory testing done well is neither sloppy nor slapdash, of course. When I go to the dictionary, I find that "ad hoc" means literally "to this", and by convention "to this purpose". The Rogers Commission on the Challenger was an ad hoc commission-that is, it was a commission set up for a purpose, and dissolved after its purpose was fulfilled. In that sense, "ad hoc" and "exploratory" are in different categories. Almost all testing worth doing, whether exploratory or scripted, is ad hoc; it's done in service of some purpose, and it stops after that purpose is fulfilled. So I can't be sure what you mean by "ad hoc" until you tell me. I'm providing my definition of exploratory testing here; you can contrast it with your notions of "ad hoc" if you like.

Is exploratory testing something that you want to do when you want fast results? Exploratory approaches tend to be faster than scripted approaches, since in an exploratory approach there are fewer gaps or lags in passing learning from person to person, and since learning tends to be faster when people are in control of their own processes. In an exploratory mode, the tester
  • tends to be more cognitively engaged;
  • dynamically manages her focus;
  • makes many observations simultaneously-some consciously and some not;
  • makes rapid evaluations-again, some consciously and some not; and
  • makes instant decisions as to what to do next.
Even in a scripted mode, it's hard to conceive of the notion of a single, atomic test; even a terribly disengaged human will make an insightful, off-script observation every now and again. At least in part because of its tendency towards repetition and disengagement, human scripted test execution often feels like plodding. Scripted test execution by a machine tends to take less time than scripted test execution performed by a human, but preparing a script (whether it is to be performed by a machine or a human) tends to take longer than not preparing a script.  Development time is a cost of automation that is sometimes hidden in plain sight.

Why are checklists okay when scripted tests are not?  Cem's point was that checklists leave the tester in control of his or her process.  They help to organize test ideas without rigorously specifying how testing should be done or what steps should be followed.  This can lead to variation in sequence, in timing, in focus, in models, and in observations.  Some suggest that this will cause testers to miss bugs.  That may be so, but we figure the risk is low; if people use diversified models, tactics, and teams, the diversity is more likely to help us find more bugs in more places.  Meanwhile, if we absolutely need to observe something specific, checklist it; or if we're concerned about detecting unwanted changes, put those change detectors in at the unit test level or fix the process problems that leave us exposed to unwelcome surprises.

Here's the most important thing:  learning is fundamental to testing, and exploratory processes are fundamental to learning.  Everything we know about educational theory says so, but our own experiences should tell us the same kinds of things.  Those of us who have kids know that kids learn most quickly when they're engaged in what they do, when they're in control of their own processes, and when they're steered rather than programmed.  Grownups are the same. 

Exploration, discovery, investigation and learning are what testing is all about.  Confirmation, verification, and validation-that's just checking.

So that's the story so far. Both testing and learning are highly open-ended and exploratory processes, so there will likely be more as we explore, learn, and test these ideas.
Quick Links...
Contact Information
Need to get in touch?
Phone: +1 (416) 656-5160

Join Our Mailing List
Safe Unsubscribe
This email was sent to by
DevelopSense | 61 Ashburnham Road | Toronto | Ontario | M6H 2K4 | Canada