It's Another
Newsletter!
Every now and again, people ask me, in email
or in person, "Are you still doing the
newsletter? I haven't received one in a long
time." The answer is that this is to be
expected; I don't put this newsletter out on a
schedule. I really only prepare one when
I've got a number of newsworthy things to say all
at the same time. I blog at http://www.developsense.com/blog.html.
I also write a monthly column for Better
Software Magazine, which I try to keep up to
date at http://www.developsense.com/articles.
And I write altogether too much on a several
online forums, particularly the Agile Testing
mailing list and the Indian testing forum
TechRepublic.com. This newsletter is
sporadic precisely because I don't know what's
going to come up, and when. In addition, I
don't know how people like to get their
news. Still, it occurs to me that I should
ask, and maybe you'd like to tell me: would
you prefer a regular
schedule? |
My First Book
Contribution
I'm delighted
to announce that my first book publication has
been released. A Gift of Time,
published by Dorset
House, is a book, edited by my colleague Fiona
Charles honouring the life and work of Jerry
Weinberg, who recently celebrated his 75th
birthday and 50th year in the computing
business. (He also published a book himself
this year, about which more below.) Fiona
collected essays from a number of Jerry's
colleagues and students, including Robert Glass,
James Bach, Sherry Heinze, Sue Petersen, Esther
Derby, Willem van den Ende, Judah Mogilensky,
Naomi Karten, James Bullock, Tim Lister, Johanna
Rothman, Jonathan Kohl, Bent Adsersen, Jerry's
wife Dani Weinberg... and me. I've not seen
all of the essays yet, but the ones that I've seen
are wonderful and a fitting tribute to a man who
has been so influential for all of us.
The
book made its first public appearance today,
November 4 2008, at the AYE Conference in
Phoenix. It's so new that it doesn't yet
appear on the Dorset House web site, but it will
be up and available for ordering soon.
Many
thanks to Fiona Charles and to Wendy Eakin at
Dorset House for their work in producing the
book.
And to Jerry: Happy birthday,
arrigato, and
namaste!
|
Positive
Deviance
The most
noteworthy thing that I've heard of recently is a
set of parallels between our world of software
development and the history of medicine. The
doctoring business has big, expensive, health- and
life-critical problems to solve. One might
expect that, considering the current attention,
the problems are new. One might assume that
the solutions are complex. And one might
assume that the optimal approach would be to have
armies of experts descending on the profession and
telling the underlings what to do. According
to Atul Gawande-a surgeon, a staff writer for the
New Yorker, and the author of Better, a book
on performance in medicine-none of this need be
the case.
Ignace Semmelweis was an
obstetrician and an empiricist in Vienna.
Around 1847, he observed a problem: that
puerperal (or "childbed") fever was killing over
20% of the mothers in his hospital's maternity
ward, where for women who delivered babies at home
the death rate was closer to 1%. He came up
with a simple solution to the problem: he
ordered every doctor and nurse in his ward to
scrub with chlorine and with a nail brush between
each patient visit. This one practice
dropped the death rate on his ward down to
1%. Simple approaches can have a huge
impact.
You might think that other
doctors would have paid attention, but they didn't
want to play. Part of the reason was that
Semmelweis's approach was spectacularly abrasive;
he harangued people who forgot to scrub, and
repeatedly accused doctors of killing their
patients when they didn't comply. This
reminded me of Jerry Weinberg's "team-breaking
exercise" at CAST 2008-set people to the task of
telling other people exactly what's wrong with
them and exactly what they have to fix.
Quality police may have good intentions, but
neglect the niceties of human contact, and the
idea-no matter how compelling-dies before it can
do good.
Contrast this with the
approach developed by Jerry and Monique Sternin to
the address the problem of malnutrition in
villages in Viet Nam. They asked mothers in
the community note whose children were best-fed
and healthy, and then encouraged those mothers to
visit and observe what the Sternins call positive
deviations from the norm, or positive
deviance. The mothers with healthy
kids were breaking with the conventional wisdom in
minimal, easy-to-accomplish ways, making slight
changes to diets and feeding patterns. In
each village, the incidence of malnutrition
decreased 65% to 85% within two
years.
In 1860, Joseph Lister
identified the reason for Semmelweis' success and
made a more successful case for prevention of
infection, including handwashing. Even so,
almost 150 years later, infections in hospitals
are still a problem, particularly with respect to
"superbugs"-antibiotic-resistant bacteria-because
doctors, nurses, and attendants are still not
washing their hands as often as they might.
In 2005, the Sternins were asked to consult at a
hospital in Pittsburgh, PA, where a number of
initiatives had been tried to reduce the
transmission of Methicillin-Resistant
Staphylococcus Aureus (MRSA), a superbug.
Rather than prescribe a set of rules, the hospital
administration followed the positive deviance
approach, asking the workers what they reckoned
should be done. There were some
logistical suggestions-making more antiseptic
dispensers available, and in more convenient
locations, for example-and some behaviour
modification, such as persuading recalcitrant
colleagues to don gloves and wash hands, and
setting up mutual pacts to speak up to colleagues
who didn't follow the protocols.
The
bulk of the ideas were not terribly innovative,
but they came from the people who needed to
implement them, and they were practical and
achievable. And they were overwhelmingly
successful. You can see the level of the ideas
yourself. For an example, look at the
wonderful video of Jasper Palmer at
http://www.positivedeviance.org. In less
than two minutes, he demonstrates a safe and easy
method that he developed himself for removing a
hospital gown and gloves that are potentially
contaminated.
One key to success was
that the workers agreed to speak out when they saw
others engaged in risky practices; another was
that the workers were being consulted and
encouraged to innovate for themselves, learning
from one another; a third was that the
improvements were based on practical things to do,
rather than on abstract mandates or principles; a
fourth was that results and progress were
publicized within the department. Within a
year, the rate of MRSA transmission in the
hospital had dropped to zero.
As I
read this, it reminded me of John Seely Brown and
Paul Duguid's book, The Social Life of
Information. To improve its field
service, Xerox created training courses, detailed
product documentation, and a clearly-defined
process for the field technicians to follow.
One researcher, though, decided to follow
the technicians around and observe what they
actually did. He found that the diagnostic
and repair problems were much messier and more
complex than the experts anticipated, and the
simple, step-by-step approaches devised by the
process gurus didn't work. The copiers were
quirky, the office environments were diverse, and
the problems were often unexpected-and therefore
not covered by the field manuals. Moreover,
the documentation gave them step-by-step
procedures to follow, but didn't provide the
motivation-why certain
steps were worth the effort. The technicians
were forced to make sense of their situations,
even when their maps didn't help. So what
did they do? Mostly they chatted. They
ate together, meeting for breakfast, or coffee, or
lunch; they played cards; and even when they
appeared to be doing other things, they talked
about work. They exchanged knowledge, where
the process and repair manuals only gave them
information.
When Quarterdeck was at
its best-the years in which I was a support
person, a tester, a program manager, and a
programmer-the company became rich enough to buy
other companies and absorb their product
lines. In addition programmers were
consistently developing new features and
technologies, and adding them to existing
products. We had training sessions for
those, but the real magic of the company was the
way in which the culture supported knowledge, not
just information. Programmers and support
people worked very closely together; sales people
asked questions-and often provided answers-in the
internal technical support forums. People
tended to know what each others' specialties
were. In my later years, we established
science fairs, in which anyone-whether a
programmer, support person, or tester-showed off
some toy or technology that they had developed,
discovered, or mastered. Although supported
from the top, the culture was fostered at all
levels.
So: if you want to
improve processes in your own shop, consider an
alternative to "maturity" and heavyweight process
improvements. Maturity isn't about following
rigid processes; it's about adapting flexibly and
responsibly to changing and unpredicted
situations. Plus, as Jerry Weinberg says, no
one is going to read a 17-volume process manual
anyway. Instead, consider watching what
people actually do, listening to
what they say to one another, and learning from
them. Set up science fairs or peer
conferences in your own shop or in your local
testing community (look at the LAWST
model, described at
http://www.kaner.com/lawst.htm). Notice that
no one who is effective follows their process from
a script or a recipe, and notice that most of
what's going on is in their actions and in their
heads. As we've long said in the Rapid
Software Testing course, only a fraction of a
skilled tester's work can be put onto paper.
It's what you-and your colleagues-think and what
you do
that counts.
|
Off to
EuroSTAR!
 I'm going to be at EuroSTAR
November 10-13. I'll be presenting at three
sessions.
On Tuesday, November 11, I'll be
leading a session titled Questioning Testing
Myths: Critical Thinking Skills for Testers.
In this, we're going to examine and question some
common testing bromides: "Every test must
have an expected, predicted result." "Effective
testing requires complete, clear, consistent, and
unambiguous specifications." "Bugs found
earlier cost less to fix than bugs found later."
"Testers are the quality gatekeepers for a
product." "Testing at boundary values is the
best way to find bugs." These are often
touted as Great Truths about testing, when they're
really heuristics. We get into trouble as
testers when we take such statements
uncritically. In the session, we're going to
explore some common cognitive errors and biases,
and some thinking tools-questioning skills,
critical thinking, context-driven thinking,
general systems thinking-that can help testers
deal confidently and thoughtfully with difficult
testing situations.
On Wednesday morning,
I'll be leading a session called Heuristics:
Solving Problems Rapidly. Heuristics are fallible
methods for solving a problem or making a
decision. People often make-and
follow-economical, "fast and frugal" approaches to
solving problems, rather than using more
deliberate or thorough reasoning. We base our
decisions on rapid cognition, snap judgments, and
imprecise decision rules. The surprising fact is
that these approaches can often lead to better
decisions because of limited information, rather
than in spite of it. For one thing,
heuristics lead us to more rapid thinking and
activity, which can generate the information that
we need at a decreased level of noise, where more
complex analysis delays action and adds noise to
the critical information. Why is this
important for testers? Among other things,
heuristics inform our test oracles-principles or
mechanisms by which we recognize a problem.
In the session, we'll talk about heuristics and do
an exercise or two to help us to learn about
heuristic approaches.
The theme of this
year's EuroSTAR conference is "The Future of
Software Testing". Thursday afternoon, I'll
be presenting a talk I've given in various forms
before: Two Futures of Software
Testing. One vision that I have of the
future is decidely a dystopia, with testers as
quality gatekeepers, with testing following a
rigourously controlled, bureaucratic, and
essentially clerical process where exploration and
investigation is frowned upon and considered
"immature". Senior testers work from extensively
detailed specifications, creating plans, drawing
state diagrams, and compiling test scripts.
Less experienced testers are relegated to
following the scripts. Changes to the product are
resisted, and testers drive that resistance. The
worst thing about this vision of the future is
that it's so much like today, and that so many
people appear to desire it. In the other
view of the future, testers are active
investigators and critical thinkers who neither
accept nor desire responsibility for releasing or
otherwise managing the product; instead, they
provide important, timely, credible information to
managers so that they can make sound and informed
business decisions. Testers work collaboratively
not only with the developers, but with the entire
project community, and are able to report at any
time on product and project risks. Testers have an
extensive understanding of tools and automation,
and decide when-and when not-to use them. Most
importantly, testers embrace challenge and change,
adapting practices and strategies thoughtfully to
fit the testing mission and its
context.
I'm looking forward to meeting
colleagues at the conference, especially some of
the new friends that I made last year. If
you're there, please make sure to say
Hello!
|
Perfect Software
Released!
 Made you look!
The
perfect software in question isn't a software
product; it's Jerry Weinberg's latest release, Perfect
Software and Other Illusions About
Testing. This is a fabulous book, and a
very important one for the community of skilled
testers. It joins Lessons
Learned in Software Testing in my list
of books that every tester should own.
Moreover, it's a book that we should should pass
on to our clients.
Jerry has been involved
in software development longer than many of us
have been alive. In fact, his first book was
released in 1961, the topsy-turvy year in which I
was born. Since the 1960s, he's been drawing
attention to the human aspects of software
development, writing or co-authoring more than 40
books. His definition of quality--value to
some person--is a notion that strongly informs
context-driven testing and thinking. One of
his descriptions of a tester--a tester is someone who
knows that things can be different--gives
this newsletter its title.
Forests have
been felled to tell us about testing techniques
that reveal information about our products.
Too few testing books talk sensibly about value. Too
few books mention anything at all about the sheer
volume and quality of information that we can
obtain without a finger even hitting a
keyboard. We can learn critically important
things by paying attention to people--what
they say and how they behave. An
example: consider the resistant response
that we get when we even propose the idea
of technical reviews. I believe that Jerry
would argue that the proposal is a kind of test,
and that the response is a test result. What
might that resistance tell us about the state of
the project?
The book was launched at the
CAST conference in July, and made available to the
general public near the end of August. Okay,
so this isn't exactly a breaking story.
Dorset House is a small publisher, and the quality
of their catalog greatly exceeds the reach of
their marketing--so it's up to us. Grab a
copy of the book for yourself and spread the
word!
|
Evolving Understanding
About Exploratory Testing
 This
is an update of a blog post that I wrote in
September 2008.
One of the
highlights of CAST 2008 was Cem Kaner's talk "The
Value of Checklists and the Danger of Scripts:
What Legal Training Suggests for Testers."A big
part of the talk was the contrast between
exploratory and scripted processes, wherein Cem
contrasted scripts-canned, step-by-step
instructions that are executed more or less
automatically, whether by an automaton or a human
acting like one-and checklists-sets of things that
we might want to consider, where the decision to
consider them is under the control of the person
using the list. For me, what was most valuable
about the the talk was the evolving story of the
nature of exploration and exploratory testing. So,
as of October 31, 2008, here's a report on my
version (as usual, strongly entwined with Cem's
and with James Bach's) of the story so far. One
goal of this exercise is to be able to point
people to this post instead of repeating myself in
testing forums.
Testing is questioning the
product in order to evaluate it; that's James'
definition. For Cem, exploratory
testing these days is (deep breath, now) "a style
of testing that emphasizes the freedom and
responsibility of the individual tester to
continually optimize the quality of her work by
treating test design, test execution, test result
interpretation, and learning as activities that
continue in parallel through the course of the
project." This was the definition that he
synthesized around the time of the Workshop on
Heuristic and Exploratory Techniques in Palm Bay,
FL, May 2006. (James Bach, Jonathan Bach, Scott
Barber, Tim Coulter, Rebecca Fiedler, David
Gilbert, Marianne Guntow, James Lyndsay, Robert
Sabourin, Adam White, Cem, and I were participants
in that workshop.) The short version-James'-is
"simultaneous test design, test execution, and
learning." The two definitions are intended to
mean the same thing. One is more explicit; the
other is quicker to say.
The opposite of
exploratory testing is scripted testing. Neither
of these is a technique; they're both approaches
to testing. Irrespective of any other dimension of
it, we call a test more exploratory and less
scripted to the extent that
- elements of design, execution,
interpretation, and learning are performed by
the same person;
- the design, execution, interpretation, and
learning happen together, rather than being
separated in time;
- the tester is making her own choices about
what to test, when to test it, and how to test
it;
- the tester makes her own choices about any
automation or tools in support of her testing,
from depending entirely upon tools to using none
at all, as she sees fit;
- everything that has been learned so far,
including the result of the last test, informs
the tester's choices about the next test;
- the tester is focused on revealing new
information, rather than confirming existing
knowledge about the product;
- in general, the tester is varying aspects of
her tests rather than repeating them, except
where the repeating aspects of the test are
intended to support the discovery of new
information.
Whether a test is a
black-box test (performed with less knowledge of
the internals of the code) or a glass-box test
(performed with more knowledge of the internals of
the code) is orthogonal to the exploratory or
scripted dimension. A white-box test can be done
either in an exploratory or a scripted way; a
black-box test can be done either way too. The key
here is the cognitive engagement of the tester,
and the extent to which she manages her choices
and her time.
Another common misconception
is that exploratory approaches can only be applied
as we're interacting directly with the
product. As Jerry Weinberg points out in
Perfect
Software, many important tests can happen
without fingers hitting a keyboard or automation
being run. Exploratory approaches can be applied
to any aspect of the development process, and not
just to test execution (that is, configuring,
operating, observing, and evaluating the running
product). We can explore during the process
of review of test plans, requirements documents,
functional specifications, code, end-user
documentation, or marketing materials. In
any of these circumstances, what we discover
contributes to our decisions about what to do
next. Review can be done in exploratory way
or in a scripted way, to the extent that the
reviewer controls his own choices about what to
observe. For example, a review may be done
freestyle (in which case it might be highly
exploratory); under the guidance of a set of
ideas, perhaps in the form of a checklist (in
which case it may be somewhat more scripted and
somewhat less exploratory); or under the control
of a strict set of conditions such as those used
by static code analyzers (in which case it's
entirely scripted).
Automation ("any use of tools to
support testing") can be used in a scripted
way or in an exploratory way. If the tool is being
used to reveal new information, or if the tester
is using the tool such that activities or data are
varying, then the test tends to be more
exploratory. If the tools are being used to
confirm what we already know, or if the tool is
supporting repetition of tests, then the test is
tending towards the scripted. Note that
these are heuristics. Exploration may be
served by repeating a test, especially if we're
exploring state-related problems, and tools can
help us with that. A test that we think of
as heavily scripted may have some variable
elements in it--consider randomized high-volume
tests. In this case, the scripted dimension
is dominant during test execution, but the
exploratory dimension might become more prominent
in the evaluation and interpretation of the
result.
For the programmers, testing
(questioning the product in order to evaluate it,
remember) is a natural element of pair programing,
and since the design, execution, interpretation,
and learning are highly integrated, pair
programming is an exploratory process. TDD has a
strong exploratory element for the same reason.
Indeed, Elisabeth Hendrickson quotes one
programmer's "aha!" moment about ET: "I get it!
It's test-driven testing!" I had a similar "aha!"
moment when I realized that TDD, done well, was a
highly exploratory style of
development.
For the inevitable people who
will inevitably ask "How is exploratory testing
different from ad hoc testing?", I'll start by
replying that I can't make the distinction until
you've provided me with a notion of what "ad hoc"
means to you. Some people believe that "ad hoc" is
a synonym for "sloppy" or "slapdash". Exploratory
testing done well is neither sloppy nor slapdash,
of course. When I go to the dictionary, I find
that "ad hoc" means literally "to this", and by
convention "to this purpose". The Rogers
Commission on the Challenger was an ad hoc
commission-that is, it was a commission set up for
a purpose, and dissolved after its purpose was
fulfilled. In that sense, "ad hoc" and
"exploratory" are in different categories. Almost
all testing worth doing, whether exploratory or
scripted, is ad hoc; it's done in service of some
purpose, and it stops after that purpose is
fulfilled. So I can't be sure what you mean by "ad
hoc" until you tell me. I'm providing my
definition of exploratory testing here; you can
contrast it with your notions of "ad hoc" if you
like.
Is exploratory testing something that
you want to do when you want fast results?
Exploratory approaches tend to be faster than
scripted approaches, since in an exploratory
approach there are fewer gaps or lags in passing
learning from person to person, and since learning
tends to be faster when people are in control of
their own processes. In an exploratory mode, the
tester
- tends to be more cognitively engaged;
- dynamically manages her focus;
- makes many observations simultaneously-some
consciously and some not;
- makes rapid evaluations-again, some
consciously and some not; and
- makes instant decisions as to what to do
next.
Even in a scripted mode, it's hard
to conceive of the notion of a single, atomic
test; even a terribly disengaged human will make
an insightful, off-script observation every now
and again. At least in part because of its
tendency towards repetition and disengagement,
human scripted test execution often feels like
plodding. Scripted test execution by a machine
tends to take less time than scripted test
execution performed by a human, but preparing a
script (whether it is to be performed by a machine
or a human) tends to take longer than not preparing a
script. Development time is a cost of
automation that is sometimes hidden in plain
sight.
Why are checklists okay when
scripted tests are not? Cem's point was that
checklists leave the tester in control of his or
her process. They help to organize test
ideas without rigorously specifying how testing
should be done or what steps should be
followed. This can lead to variation in
sequence, in timing, in focus, in models, and in
observations. Some suggest that this will
cause testers to miss bugs. That may be so,
but we figure the risk is low; if people use
diversified models, tactics, and teams, the
diversity is more likely to help us find more bugs
in more places. Meanwhile, if we absolutely
need to observe something specific, checklist it;
or if we're concerned about detecting unwanted
changes, put those change detectors in at the unit
test level or fix the process problems that leave
us exposed to unwelcome surprises.
Here's
the most important thing: learning is
fundamental to testing, and exploratory processes
are fundamental to learning. Everything we
know about educational theory says so, but our own
experiences should tell us the same kinds of
things. Those of us who have kids know that
kids learn most quickly when they're engaged in
what they do, when they're in control of their own
processes, and when they're steered rather than
programmed. Grownups are the same.
Exploration, discovery, investigation and
learning are what testing is all
about. Confirmation, verification, and
validation-that's just checking.
So
that's the story so far. Both testing and learning
are highly open-ended and exploratory processes,
so there will likely be more as we explore, learn,
and test these ideas.
| |
