Blog Posts for the ‘Uncategorized’ Category

Deeper Testing (1): Verify and Challenge

Thursday, March 16th, 2017

What does it mean to do deeper testing? In Rapid Software Testing, James Bach and I say:

Testing is deep to the degree that it has a probability of finding rare, subtle, or hidden problems that matter.

Deep testing requires substantial skill, effort, preparation, time, or tooling, and reliably and comprehensively fulfills its mission.

By contrast, shallow testing does not require much skill, effort, preparation, time, or tooling, and cannot reliably and comprehensively fulfill its mission.

Expressing ourselves precisely is a skill. Choosing and using words more carefully can sharpen the ways we think about things. In the next few posts, I’m going to offer some alternative ways of expressing the ideas we have, or interpreting the assignments we’ve been given. My goal is to provide some quick ways to achieve deeper, more powerful testing.

Many testers tell me that their role is to verify that the application does something specific. When we’re asked to that, it can be easy to fall asleep. We set things up, we walk through a procedure, we look for a specific output, and we see what we anticipated. Huzzah! The product works!

Yet that’s not exactly testing the product. It can easily slip into something little more than a demonstration—the kinds of things that you see in a product pitch or a TV commercial. The demonstration shows that the product can work, once, in some kind of controlled circumstance. To the degree that it’s testing, it’s pretty shallow testing. The product seems to work; that is, it appears to meet some requirement to some degree.

If you want bugs to survive, don’t look too hard for them! Show that the product can work. Don’t push it! Verify that you can get a correct result from a prescribed procedure. Don’t try to make the product expose its problems.

But if you want to discover the bugs, present a challenge to the product. Give it data at the extremes of what it should be able to handle, just a little beyond, and well beyond. Stress the product out; overfeed it, or starve it of something that it needs. See what happens when you give the product data that it should reject. Make it do things more complex than the “verification” instructions suggest. Configure the product (or misconfigure it) in a variety of ways to learn how it responds. Violate an explicitly stated requirement. Rename or delete a necessary file, and observe whether the system notices. Leave data out of mandatory fields. Repeat things that should only happen once. Start a process and then interrupt it. Imagine how someone might accidentally or maliciously misuse the product, and then act on that idea. While you’re at it, challenge your own models and ideas about the product and about how to test it.

We can never prove by experiment—by testing—that we’ve got a good product; when the product stands up to the challenge, we can only say that it’s not known to be bad. To test a product in any kind of serious way is to probe the extents and limits of what it can do; to expose it to variation; to perform experiments to show that the product can’t do something—or will do something that we didn’t want it to do. When the product doesn’t meet our challenges, we reveal problems, which is the first step towards getting them fixed.

So whenever you see, or hear, or write, or think “verify”, try replacing it with “challenge“.

The Test Case Is Not The Test

Thursday, February 16th, 2017

A test case is not a test.

A recipe is not cooking. An itinerary is not a trip. A score is not a musical performance, and a file of PowerPoint slides is not a conference talk.

All of the former things are artifacts; explicit representations. The latter things are human performances.

When the former things are used without tacit knowledge and skill, the performance is unlikely to go well. And with tacit knowledge and skill, the artifacts are not central, and may not be necessary at all.

The test case is not the test. The test is what you think and what you do. The test case may have a role, but you, the tester, are at the centre of your testing.

Further reading:

Throwing Kool-Aid on the Burning Books

Sunday, November 2nd, 2014

Another day, and another discovery of an accusation of Kool-Aid drinking or book burning from an ISO 29119 proponent (or an opponent of the opposition to it; a creature in the same genus, but not always the same species).

Most of the increasingly vehement complaints come from folks who have not read [ISO 29119], perhaps because they don’t want to pay for the privilege but also (and I’d guess mainly) because they are among the now possibly majority of folks who don’t read anything except from their favorite few Kool Aid pushers and don’t want their opinions muddled by actual information, especially any which might challenge their views.

The charge that the opponents of 29119 don’t read anything other than their favourite Kool-Aid pushers is almost—but not quite—as ludicrous as the idea that a complex, investigative, intellectual, activity like testing can be standardized. Does this look like a book-burner to you? One opponent to 29119 is the author of two of the best-selling (in, in my view, best-written) books on software testing in its relatively short history—does this look a book burner? (In the unlikely event that it does, drop in to his web site and have a look at his publications and the references within.) Here’s a thoughtful opponent of 29119; book burner? How about this—book burner? And here’s a relatively recent snapshot of my own library.

For some contrast, have a look at the standard itself. As a matter of fact, other than the standards that it replaces, along with the ISTQB Foundation Syllabus, the standard’s bibliographies include references to no works at all; neither in testing nor in any of the other domains that relate to testing—programming, psychology, mathematics, history, measurement, anthropology, critical thinking, economics, philosophy, computer science, sociology, systems thinking, qualitative research, software development… The ISTQB syllabus includes a handful of books about testing, and only about testing. The most recent reference is to Lee Copeland’s A Practitioner’s Guide to Software Testing, which—although a quite worthy book for new testers—was published in 2004, a full seven years before the syllabus was published in 2011.

Update, November 4: Sharp-eyed reader Nathalie van Delft points out that Part One of the Standard contains references to two books that are not prior standards or ISTQB material: Crispin and Gregory’s Agile Testing (2009), and Koen’s Definition of the Engineering Method (1985). So, one book since 2004, and one book on engineering, in the Concepts and Definitions section of the standard.

Where are the references to other books, old or new, that would be genuinely helpful to new testers, like Petzold’s Code: The Hidden Language of Computer Hardware and Software, or Weinberg’s Perfect Software and Other Illusions About Testing, or Marick’s Everyday Scripting in Ruby? Why is the syllabus not updated with important new books like Kahneman’s Thinking Fast and Slow, or Kaner and Fiedler’s Foundations of Software Testing, or Elisabeth Hendrickson’s Explore It!, even if the rest of the syllabus remains static? Worried that things might get too heady for foundation-level testers? Why not refer to The Cartoon Guide to Statistics or a first-year college book on critical thinking, like Levy’s Tools of Critical Thinking, or introductory books on systems thinking like Meadows’ Thinking in Systems: A Primer or Weinberg and Gause’s Are Your Lights On?

See the difference? Our community encourages testers to study the craft; to read; to import new ideas from outside the field; to argue and debate; to learn from history; to think independently. We also cop to errors, when someone points them out; thanks, Nathalie. Some of the books above are by intellectual or commercial competitors, or contain material on which there is substantial disagreement between individuals and clans in the wider community. Big deal; those books are useful and important, and they’re part of the big conversation about testing.

You could only believe that the thoughtful opponents to 29119 are book-burners or Kool-Aid drinkers… well, if you haven’t read what they’ve been writing.

So, to those who answer the opposition to 29119 with calumny… drink up. And know that no smoke detectors were activated in the preparation of this blog post.

Facts and Figures in Software Engineering Research

Monday, October 20th, 2014

On July 23, 2002, Capers Jones, Chief Scientist Emeritus of a company called Software Productivity Research, gave a presentation called “SOFTWARE QUALITY IN 2002: A SURVEY OF THE STATE OF THE ART”. In this presentation, he provided the sources for his data on the second slide:

SPR clients from 1984 through 2002
• About 600 companies (150 clients in Fortune 500 set)
• About 30 government/military groups
• About 12,000 total projects
• New data = about 75 projects per month
• Data collected from 24 countries
• Observations during more than a dozen lawsuits

(Source:, accessed September 5, 2014)

On May 2, 2005, Mr. Jones, this time billed as Chief Scientist and Founder of Software Quality Research, gave a presentation called “SOFTWARE QUALITY IN 2005: A SURVEY OF THE STATE OF THE ART”. In this presentation, he provided the source for his data, again on the second slide:

SPR clients from 1984 through 2005
• About 625 companies (150 clients in Fortune 500 set)
• About 35 government/military groups
• About 12,500 total projects
• New data = about 75 projects per month
• Data collected from 24 countries
• Observations during more than 15 lawsuits

(Source:, accessed September 5, 2014)

Notice that 34 months have passed between the two presentations, and that the “total projects number” has increased by 500. At 75 projects a month, we should expect that 2550 projects have been added to the original tally; yet only 500 projects have been added.

On January 30, 2008, Mr. Jones (Founder and Chief Scientist Emeritus of Software Quality Research), gave a presentation called “SOFTWARE QUALITY IN 2008: A SURVEY OF THE STATE OF THE ART”. This time the sources (once again on the second slide) looked like this:

SPR clients from 1984 through 2008
• About 650 companies (150 clients in Fortune 500 set)
• About 35 government/military groups
• About 12,500 total projects
• New data = about 75 projects per month
• Data collected from 24 countries
• Observations during more than 15 lawsuits

(Source:, accessed September 5, 2014)

This is odd. 32 months have passed since the May 2005 presentation. With new data being added at 75 projects per month, there should have been 2400 projects new since the prior presentation. Yet there has been no increase at all in the number of total projects.

On November 2, 2010, Mr. Jones (now billed as Founder and Chief Scientist Emeritus and as President of Capers Jones & Associates LLC) gave a presention called “SOFTWARE QUALITY IN 2010: A SURVEY OF THE STATE OF THE ART”. Here are the sources, once again from the second slide:

Data collected from 1984 through 2010
• About 675 companies (150 clients in Fortune 500 set)
• About 35 government/military groups
• About 13,500 total projects
• New data = about 50-75 projects per month
• Data collected from 24 countries
• Observations during more than 15 lawsuits

(Source:, accessed September 5, 2014)

Here three claims about the data have changed: 25 companies have been added to the data sources, 13,500 projects comprises the total set, and “about 50-75 projects” have been added (or are being added; this isn’t clear) per month. 21 full months have passed since the January presentation (which came at the end of the month). To be fair, the claim of an increase of 1,000 projects almost fits the lower bound of the claimed number of per-month increases (which would be 1,050 new projects since the last presentation), but not the claim of 75 per month (1,575 new projects). What does it mean to claim “new data = about 50-75 projects per month”, when the new data appears to be coming in a rate below the lowest rate claimed?

On May 1, 2012, Mr. Jones (CTO of Namcook Analytics LLC) gave a talk called “SOFTWARE QUALITY IN 2012: A SURVEY OF THE STATE OF THE ART”. Once again, the second slide provides the sources.

Data collected from 1984 through 2012
• About 675 companies (150 clients in Fortune 500 set)
• About 35 government/military groups
• About 13,500 total projects
• New data = about 50-75 projects per month
• Data collected from 24 countries
• Observations during more than 15 lawsuits

(Source:, accessed September 5, 2014)

Here there has been no change at all in any of the previous claims (except for the range of time over which the data has been collected). The claim that 50-75 projects per month has been added remains. At that rate, extrapolating from the claims in the November 2010 presentation, there should be between 14,400 and 14,850 projects in the data set. Yet the claim of 13,500 total projects also remains.

On August 18, 2013, Mr. Jones (now VP and CTO of Namcook Analytics LLC) gave a presentation “SOFTWARE QUALITY IN 2013: A SURVEY OF THE STATE OF THE ART”. Here are the data sources (from page 2)

Data collected from 1984 through 2013
• About 675 companies (150 clients in Fortune 500 set)
• About 35 government/military groups
• About 13,500 total projects
• New data = about 50-75 projects per month
• Data collected from 24 countries
• Observations during more than 15 lawsuits

(Source:, accessed September 5, 2014)

Once again, no change in the total number of projects, but the claim of 50-75 new projects remains. Again, based on the 2012 claim, 15 months in time passed (more like 16, but we’ll be generous here), and the growth claims in these presentations, there should be between 14,250 and 14,625 projects in the data set.

Based on the absolute claim of 75 new projects per month in the period 2002-2008, and 50 per month in the remainder, we’d expect 20,250 projects at a minimum by 2013. But let’s be conservative and generous, and base the claim of new projects per month at 50 for the entire period from 2002 to 2013. That would be 600 new projects per year over 11 years; 6,600 projects added to 2002’s 12,000 projects, for a total of 18,600 by 2013. Yet the total number of projects went up by only 1,500 over the 11-year period—less than one-quarter of what the “new data” claims would suggest.

In summary, we have two sets of figures in apparent conflict here. In each presentation,

1) the project data set is claimed to grow at a certain rate (50-75 per month, which amounts to 600-900 per year).
2) the reported number of projects grows at a completely different rate (on average, 136 per year).

What explains the inconsistency between the two sets of figures?

I thank Laurent Bossavit for his inspiration and help with this project.

Dramatis Personae

Thursday, September 18th, 2014

On September 8, Stuart Reid, the convenor of the Working Group, posted a response to the Stop 29119 petition that included this paragraph:

(The petitioners claim that) “The standards ‘movement’ is politicized and driven by big business to the exclusion of others. A large proportion of the members of WG26 are listed at our About WG26 page along with their employers. This list does not support the assertion in the petition. The seven editors (who do the majority of the work) are from a government department, a charity, two small testing consultancies, a mid-size testing consultancy, a university and one is semi-retired.”

I believe Dr. Reid misinterprets the position of the petition’s authors and signers as objecting to the standards process “driven by big business to the exclusion of others”. Here I can only speak for myself. My concern is not about the size of the businesses involved. Instead, it is this: if a handful of consultancies of any size were to use the ISO standards process to set the terms for “the only internationally-recognised and agreed standards”, it would raise a plausible perception of conflict of interest. If those standards were lengthy, complex, open to interpretation, and paywalled, and if those consultancies were to offer services related to standards compliance, the possibility of motivations other than altruism could loom. So, as the convenor of the working group, Dr. Reid is right to attempt to make affiliations clear and transparent.

As of September 3, the roster for the ISO 29119 working group looked like this:

The convenor of ISO/IEC JTC1/SC7 WG26 is:

Dr Stuart Reid – Testing Solutions Group, United Kingdom
The co-editors of ISO/IEC/IEEE 29119 Software Testing and members of WG26 are:

Anne Mette Hass (editor of ISO/IEC/IEEE 29119-3) – KOMBIT, Denmark
Jon Hagar (product editor of ISO/IEC/IEEE 29119) – USA
Matthias Daigl (editor of ISO/IEC/IEEE 29119-5) – Koln University, Germany
Prof Qin Liu (co-editor of ISO/IEC/IEEE 29119-4) – School of Software Engineering, Tongji University, China
Sylvia Veeraraghavan (editor of ISO/IEC/IEEE 29119-2) – Mindtree, India
Dr Tafline Murnane (editor of ISO/IEC/IEEE 29119-4) – K. J. Ross & Associates, Australia
Wonil Kwon (ISO/IEC 33063 Process Assessment Model for Software testing processes) – Software Testing Alliances, South Korea


As of September 12, that page had changed:

The convenor of ISO/IEC JTC1/SC7 WG26 is:

Dr Stuart Reid – Testing Solutions Group, United Kingdom
The co-editors of ISO/IEC/IEEE 29119 Software Testing and members of WG26 are:

Anne Mette Hass (editor of ISO/IEC/IEEE 29119-3) – KOMBIT, Denmark
Jon Hagar (product editor of ISO/IEC/IEEE 29119) – USA
Matthias Daigl (editor of ISO/IEC/IEEE 29119-5) – Koln University, Germany
Prof Qin Liu (co-editor of ISO/IEC/IEEE 29119-4) – School of Software Engineering, Tongji University, China
Sylvia Veeraraghavan (editor of ISO/IEC/IEEE 29119-2) – Janaagraha, India
Dr Tafline Murnane (editor of ISO/IEC/IEEE 29119-4) – K. J. Ross & Associates, Australia
Wonil Kwon (ISO/IEC 33063 Process Assessment Model for Software testing processes) – Software Testing Alliances, South Korea


Anne Mette Hass‘ affiliation has been listed as KOMBIT for several years. Her LinkedIn history suggests other possible connections. There, as of September 14, 2014, she is listed as a Compliance Consultant for NNIT. A search for “29119” in NNIT’s web site leads quickly to an events page (, retrieved September 14, 2014) that features a promotion for “Webinar – The Core of Testing, Dynamic Testing Process, According to ISO 29119.” Prior to this, once again according to LinkedIn, Ms. Hass worked for Delta, a Danish test consultancy. A search for “29119” on Delta’s site leads quickly to a page that begins “DELTA’s experts participate as key players in a variety of national and international norms and standardization groups”. ISO 29119 is listed as one of those international standards.

I presume that Jon Hagar, with no affiliation listed, is the “semi-retired” editor to whom Dr. Reid refers. Per LinkedIn, he is currently an independent consultant. Prior to this, Jon was an Engineer-Manager of Software Testing at Lockheed Martin.

Matthias Daigl‘s affiliation is listed as on the Working Group’s roster as “Koln University”, yet his profile on LinkedIn lists him as “Managing Consultant at imbus”. It makes no mention of Koln University. On the site, you can find this page, which includes this paragraph: “Represented by our managing consultant Matthias Daigl we take an active part in the development of the series 29119. Matthias Daigl is a member of the DIN norm commission 043-01-07 ‘Software und System-Engineering’ and he is one of the two Germans who belong to the international ISO/IEC JTC 1 subcommittee 07 workgroup 26 ‘Software testing’ working on the standard 29119 with test experts from the whole world. There the imbus consultant has the editor role for the part 29119-5.”

There are several listings for Qin Liu on LinkedIn. One of them refers to an associate professor at Tongji University.

Dr. Tafline Murname‘s affiliation is with KJ Ross and Associates. “KJ Ross provide you with independent software quality expertise, either in-house, fully outsourced or a blend of both. With 100 local and 3000 offshore trained test consultants on hand, our service is carried out to national and international standards, including ISO/IEC 29119 and ISO/IEC 17025.” It is worth noting that KJ Ross does not explicitly offer 29119 consulting services on its Web site; if it is marketing such services, it is not doing so aggressively.

Wonil Kwon is listed as “Software Testing Alliances, South Korea”. LinkedIn shows this affiliation, along with one for STEN, a testing consultancy.

Sylvia Veeraraghavan‘s affiliation according to the Working Group roster page suddenly changed on or about September 3, 2014. She is now with Janaagraha, a charity. Prior to that, though, she was with Mindtree, a company that assertively touts its part in developing 29119, and which sells related consulting services.

So, let’s review the claim of affiliations for the seven editors as currently listed on the page.

A government department. Dr. Reid apparently refers to Matt Mansell, affiliated with the Department of Internal Affairs, New Zealand. This description is consistent with Mr. Mansell’s current and past affiliations on LinkedIn. Oddly, Mr. Mansell’s name is no longer listed among the editors; he was formerly given credit as the editor of 29119-1.

A charity. Technically true, but Ms. Veeraraghavan very recently resigned from a large testing consultancy; a lucky break for Dr. Reid in terms of the timing of his response to the petition.

Two small testing consultancies and a mid-size consultancy. Ms. Hass, Mr. Daigl, and Mr. Kwon currently work for testing consultancies, and until very recently, Ms. Veeraraghavan did too. Ms. Murname also works for a consultancy that touts her work on ISO 29119, and notes that its services “are carried out to national and international standards, including ISO/IEC 29119”.

A university. Two per the roster, but of these only one claim—Qin Liu’s—is supported by LinkedIn. Why is Mr. Daigl listed as being with Koln University? Could it be because of this?

One semi-retired. True.

Finally, note that when Dr. Reid lists the editors of the standards, he does not refer to himself, even though he is arguably the most publicly prominent member of the Working Group, and its convenor. Dr. Reid is the CTO of Testing Solutions Group, a testing services consultancy. From TSG’s Web site: “For companies looking to make the switch to ISO 29119, TSG can provide help with implementation or measure how closely existing processes conform to the standard and an action plan to being about compliance.” (sic) (Source:

Six of nine core members of the working group appear to be affiliated with consultancies. Why does Dr. Reid offer a different assessment? Would it be to distract from the appearance of a conflict of interest?

In addition, Dr. Reid states:

There is also no link between the ISO/IEC/IEEE Testing Standards and the ISTQB tester certification scheme.

That’s interesting. No link, eh?

(All links retrieved 18 September, 2014.)

I observe that, as Santayana said, “Those who cannot remember the past are condemned to repeat it”. It will be very interesting to watch what happens over the next few years.

So, has Dr. Reid been transparent, forthcoming, and credible about affiliations between himself and the editors (who, in his words, do the majority of the work) and organizations that are positioned to benefit from the standard? Has the Working Group diligently upheld and documented its compliance with ISO’s Codes of Conduct?

Those who support the standard, when you can find them, often tout the advantages of an international language for testing. I’ve written against this idea in the past. However, it is true that Latin was used as an international language for a long time. To this day, some phrases survive in common parlance: Cui bono? Quis custodiet ipsos custodes?

One final note: Investigation is not well served by standardization. I did not follow a standardized process in preparing this report.

Weighing the Evidence

Friday, September 12th, 2014

I’m going to tell you a true story.

Recently, in response to a few observations, I began to make a few changes in my diet and my habits. Perhaps you’ll be impressed.

  • I cut down radically on my consumption of sugar.
  • I cut down significantly on carbohydrates. (Very painful; I LOVE rice. I LOVE noodles.)
  • I started drinking less alcohol. (See above.)
  • I increased my intake of tea and water.
  • I’ve been reducing how much I eat during the day; some days I don’t eat at all until dinner. Other days I have breakfast, lunch, and dinner. And a snack.
  • I reflected on the idea of not eating during the day, thinking about Moslem friends who fast, and about Nassim Taleb’s ideas in Antifragile. I decided that some variation of this kind in a daily regimen is okay; even a good idea.
  • I started weighing myself regularly.

    Impressed yet? Let me give you some data.

    When I started, I reckon I was just under 169 lbs. (That’s 76.6 kilograms, for non-Americans and younger Canadians. I still use pounds. I’m old. Plus it’s easier to lose a pound than a kilogram, so I get a milestone-related ego boost more often.)

    Actually, that 169 figure is a bit of a guess. When I became curious about my weight, the handiest tool for measuring it was my hotel room’s bathroom scale. I kicked off my shoes, and then weighed myself. 173 lbs., less a correction for my clothes and the weight of all of the crap I habitually carry around in my pockets: Moleskine, iPhone, Swiss Army knife, wallet stuffed with receipts, pocket change (much of it from other countries). Sometimes a paperback.

    Eventually I replaced the batteries on our home scale (when did bathroom scales suddenly start needing batteries? Are there electronics in there? Is there software? Has it been tested?—but I digress). The scale implicitly claims a certain level of precision by giving readings to the tenth of a pound. These readings are reliable, I believe; that is, they’re consistent from one measurement to the next. I tested reliability by weighing myself several times over a five-minute period, and the results were consistent to the tenth of a pound. I repeated that test a day or two later. My weight was different, but I observed the same consistency.

    I’ve been making the measurement of my actual weight a little more precise by, uh, leaving the clothes out of the measurement. I’ve been losing between one and two pounds a week pretty consistently. A few days ago, I weighed myself, and I got a figure of 159.9 lbs. Under 160! Then I popped up for a day or two. This morning, I weighed myself again. 159.4! Bring on the sugar!

    That’s my true story. Now, being a tester, I’ve been musing about aspects of the measurement protocol.

    For example, being a bathroom scale, it’s naturally in the bathroom. The number I read from the scale can vary depending on whether I weigh myself Before or After, if you catch my meaning. If I’ve just drunk a half litre of water, that’s a whole pound to add to the variance. I’ve not been weighing myself at consistent times of the day, either. In fact, this afternoon I weighed myself again: 159.0! Aren’t you impressed!

    Despite my excitement, it would be kind of bogus for me to claim that I weigh 159.0 lbs, with the “point zero”. I would guess my weight fluctuates by at least a pound through the day. More formally, there’s natural variability in my weight, and to be perfectly honest, I haven’t measured that variability. If I were trying to impress you with my weight-loss achievement, I’d be disposed to report the lowest number on any given day. You’d be justified in being skeptical about my credibility, which would make me obliged to earn it if I care about you. So what could I do to make my report more credible?

    • I could weigh myself several times per day (say, morning, afternoon, and night) at regular times, average the results, and report the average. If I wanted to be credible, I’d tell you about my procedure. If I wanted to be very credible, I’d tell you about the variances in the readings. If I wanted to be super credible, I’d let you see my raw data, too.

      All that would be pretty expensive and disruptive, since I would have to spend few minutes going through a set procedure (no clothes, remember?) at very regular times, every day, whether I was at home or at a business lunch or travelling. Few hotel rooms provide scales, and even if they did, for consistency’s sake, I’d have to bring my own scale with me. Plus I’d have to record and organize and report the data credibly too. So…

    • Maybe I could weigh myself once a day. To get a credible reading, I’d weigh myself under very similar and very controlled conditions; say, each morning, just before my shower. This would be convenient and efficient, since doffing clothes is part of the shower procedure anyway. (I apologize for my consistent violation of the “no disturbing mental images” rule in this post.) I’d still have to bring my own scale with me on business trips to be sure I’m using consistent instrumentation.
    • Speaking of instrumentation, it would be a good idea for me to establish the reliability and validity of my scale. I’ve described its reliability above; it produces a consistent reading from one measurement to the next. Is it a valid reading, though? If I desired credibility, I’d calibrate the scale regularly by comparing its readings to a reference scale or reference weight that itself was known to be reliable (consistent between observations) and valid (consistent with some consensus-based agreement on what “a pound” is). If I wanted to be super-credible, I’d report whatever inaccuracy or variability I observed in the reading from my scale, and potential inconsistencies in my reference instruments, hoping that both were within an acceptable range of tolerance. I might also invite other people to scrutinize and critique my procedure.
    • If I wanted to be ultra-scientific, I’d also have to be prepared to explain my metric—the measurement function by which I hang a number on an observation. and the manner in which I operationalized the metric. The metric here is bound into the bathroom scale: for each unit pound placed on the scale, the figure display should increase by 1.0. We could test that as I did above. Or, more whimsically, if I were to put 159 one-pound weights on one side of Sir Bedevere’s largest scales, and me on the other, the scales would be in perfect balance (“and therefore… A WITCH!”), assuming no problems with the machinery.
    • If I missed any daily observations, that would be unfortunate and potentially misleading. Owning up to the omission and reporting it would probably preferable to covering it up. Covering up and getting caught would torpedo my credibility.
    • Based on some early samples, and occasional resampling, I could determine the variability of my own weight. When reporting, I could give a precise figure and along with the natural variation in the measurement: 159.4 lbs, +/- 1.2 lbs.
    • Unless I’m wasting away, you’d expect to see my weight stabilize after a while. Stabilize, but not freeze. Considering the natural variance in my weight, it would be weird and incredible if I were to report exactly the same weight week after week. In that case, you’d be justified to suspect that something was wrong. It could be a case of quixotic reliability—Kirk and Miller’s term for an observation that is consistent in a trivial and misleading way, as a broken thermometer might yield. Such observations, they say, frequently prove “only that the investigator has managed to observe or elicit ‘party line’ or rehearsed information. Americans, for example, reliably respond to the question ‘How are you?’ with the knee-jerk ‘Fine.” The reliability of this answer does not make it useful data about how Americans are.” Another possibility, of course, is that I’m reporting faked data.
    • It might be more reasonable to drop the precision while retaining accuracy. “About 160 lbs” is an accurate statement, even if it’s not a precise one. “About 160, give or take a pound or so” is accurate, with a little patina of precision and a reasonable and declared tolerance for imprecision.
    • Plus, I don’t think anyone else cares about a daily report anyhow. Even I am only really interested in things in the longer term. Having gone this far watching things closely, I can probably relax. One weighing a week, on a reasonably consistent day, first thing in the morning before the shower (I promise; that was the last time I’ll present that image) is probably fine. So I can relax the time and cost of the procedure, too.
    • I’m looking for progress over time to see the effects of the changes I’m made to my regimen. Saying “I weigh about 160. Six weeks ago, I weighed about 170” adds context to the report. I could provide the raw data:

      Plotting the data against time on a chart would illustrate the trend. I could show display the data in a way that showed impressive progress:

      But basing the Y-axis at 154.0 (to which Excel defaulted, in this case) wouldn’t be very credible because it exaggerates the significance of the change. To be credible, I’d use a zero base:

      Using a zero-based Y-axis on the chart would show the significance of change in a more neutral way.

    • To support the quantitative data, I might add other observations, too: I’ve run out of holes on my belt and my pants are slipping down. My wife has told me that I look trimmer. Given that, I could add add these observations to the long-term trend in the data, and could cautiously conclude that the regimen overall was having some effect.
    • All this is fine if I’m trying to find support for the hypothesis that my new regimen is having some effect. It’s not so good for two other things. First, it does not prove that my regimen change is having an effect. Maybe it’s having no effect at all, and I’ve been walking and biking more than before; or maybe I acquired some kind of wasting disease just as I began to cut down on the carbs. Second, it doesn’t identify specific factors that brought about weight loss and rule out other factors. To learn about those and to report on them credibly, I’d have to go back to a more refined approach. I would have to vary aspects of my diet while controlling others and make precise observations of what happened. I’d have to figure out what factors to vary, why they might be important, and what effects they might have. In other words, I’d be developing a hypothesis tied to a model and a body of theory. Then I’d set up experiments, systematically varying the inputs to see their effects, and searching for other factors that might influence the outcomes. I’d have to control for confounding factors outside of my diet. To make the experiment credible, I’d have to show that the numbers were focused on describing results, and not on attaining a goal. That’s the distinction between inquiry metrics and control metrics: an inquiry metric triggers questions; a control metric influences or drives decisions.

    When I focus on the number, I set up the possibility of some potentially harmful effects. To make the number look really good on any given day, I might cut my water intake. To make the number look fabulous over a prolonged period (say, as long as I was reporting my weight to you), I could simply starve myself until you stopped paying attention. Then it’d be back to lots of sugar in the coffee, and yes, I will have another beer, thank you.) I know that if I were to start exercising, I’d build up muscle mass, and muscle weighs more than flab. It becomes very tempting to optimize my weight in pounds, not only to impress you, but also to make me feel proud of myself. Worst of all: I might rig the system not consciously, but unconsciously. Controlling the number is reciprocal; the number ends up controlling me.

    Having gone through all of this, it might be a good idea to take a step back and line up the accuracy and precision of my measurement scheme with my goal—which I probably should have done in the first place. I don’t really care how much I weigh in pounds; that’s just a number. No one else should care how much I weigh every day. And come to think of it, even if they did care, it’s none of their damn business. The quantitative value of my weight is only a stand-in—a proxy or an indirect measurement—for my real goal. My real goal is to look and feel more sleek and trim. It’s not to weigh a certain number of pounds; it’s to get to a state where my so-called “friends” stop patting my belly and asking me when the baby is due. (You guys know who you are.)

    That goal doesn’t warrant a strict scientific approach, a well-defined system of observation, and precise reporting, because it doesn’t matter much except to me. Some data might illustrate or inform the story of my progress, but the evidence that matters is in the mirror; do I look and feel better than before?

    In a different context, you may want to persuade people in a professional discipline of some belief of some course of action, while claiming that you’re making solid arguments based on facts. If so, you have to marshal and present your facts in a way that stands up to scrutiny. So, over the next little while, I’ll raise some issues and discuss things that might be important for credible reporting in a professional community.

    This blog post was strongly influenced by several sources.

    Cem Kaner and Walter P. Bond, “Software Engineering Metrics: What Do They Measure and How Do We Know“. In particular, I used the ten questions on measurement validity from that paper as a checklist for my elaborate and rigourous measurement procedures above. If you’re a tester and you haven’t read the paper, my advice is to read it. If you have read it, read it again.

    Shadish, Cook, and Campbell, Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Snappy title, eh? As books go, it’s quite expensive, too. But if you’re going to get serious about looking at measurement validity, it’s a worthwhile investment, extremely interesting and informative.

    Jerome Kirk and Mark L. Miller, Reliability and Validity in Qualitative Research. This very slim book raises lots of issues in performing, analyzing, and reporting if your aim is to do credible research. (Ultimately, all research, whether focused on quantitative data or not, serves a qualitative purpose: understanding the nature of things at least a little better.)

    Gerald M. (Jerry) Weinberg, Quality Software Management, Vol. 2: First Order Measurement, (also available as two e-books, “How to Observe Software” and “Responding to Significant Software Events”)

    Edward Tufte’s Presenting Data and Information (a mind-blowing one-day course) and his books The Visual Display of Quantitative Information; Envisioning Information; Visual Explanations; and Beautiful Evidence.

    Prior Art Dept.: As I was writing this post, I dimly recalled Brian Marick posting something on losing weight several years ago. I deliberately did not look at that post until I was finished with this one. From what I can see, that material ( was not related to this. On the other hand, I hope Brian continues to look and feel his best. 🙂

    I thank Laurent Bossavit and James Bach for their reviews of earlier drafts of this article.

Construct Validity

Tuesday, September 9th, 2014

A construct, in science, is (informally) a pattern or a means of categorizing something you’re talking about, especially when the thing you’re talking about is abstract.

Constructs are really important in both qualitative and quantitative research, because they allow us to differentiate between “one of these” and “not one of these”, which is one of the first steps in measurement and analysis. If you want to describe something or count it such that other people find you credible, you’ll need to describe the difference between “one” and “not-one” in a way that’s valid. (“Valid” here means that you’ve provided descriptions, explanations, or measurements for your categorization scheme while managing or ruling out alternatives, such that other people are prepared to accept your construct, and your definition can withstand challenges successfully.)

If you’re familiar with object-oriented programming, you might think of a construct as being like a class, in that objects have an “is a” relationship to a class. In an object-oriented program, things tend to be pretty tidy; an object is either a member of a certain class or it isn’t. For example, in Ruby, an object will respond to a query of the kind_of?() method with a binary true or false. In the world, not under the control of nice, neat models developed by programmers armed with digital computers, things are more messy.

Supposing that someone asks you to identify vehicles and pedestrians passing by a little booth that he’s set up. It seems pretty obvious that you’d count cars and trucks without asking him for clarification. However, what about bicycles? Tricycles? A motor scooter? An electric motor scooter? If a unicyclist goes by, do we count him? A skateboarder? A pickup truck towing a wagon with two ATVs in it? A recreational vehicles towing a car? An ATV? A tractor, pulling a wagon? A diesel truck pulling a trailer? How do you count a tow-truck, towing another vehicle, with the other vehicle’s driver riding in the tow truck? As one vehicle or two? A bus? A car transporter—a truck with nine vehicles on it? Who cares, you ask?

Well, the booth is at the entrance to a ferry boat, and the fee is $60 per vehicle, $5 per passenger, and $10 for pedestrians. Lots of people (especially those self-righteous cyclists)(relax; I’m one of them too) will gripe if they’re charged sixty bucks. Yet where I live, a bicycle is considered a vehicle under the Highway Traffic Act, which would suit the ferry owner who wants to maximize the haul of cash. He’d like especially like to see $600 from the car transporter. So in regular life, categorization schemes count, and the method for determining what fits into what category counts too.

How many vehicles?

If the problem is tricky for physical things—widgets—it’s super-tricky for abstractions in science that pertains to humans. You’ve decided to study the effect of a new medicine, and you want to try it out on healthy people to check for possible side effects. What is a healthy person? Health is an abstraction; a construct. If someone is in terrific shape but happens to have a cold today, does that person count as healthy? Over the last few summers, I’ve met a kid who’s a friend of a friend. He’s fit, strong, capable, active… and he does kidney dialysis ever couple of days or so. Healthy? A transplant patient who is in great shape, but who needs a daily dose of anti-rejection drugs: healthy?

If your country gives extra points to potential immigrants who are bilingual (as mine does), what level of fluency constitutes competence in a language to the degree that you can decide, “bilingual or not”? Note that I’m not referring to a test of whether someone is bilingual or not; I’m talking about the criteria that we’re going to test for; our sorting rules. Economists talk about “the economy” growing; what constitutes “the economy”? People speak of “events”; when airplanes hit the World Trade Center, was that one event or two? Who cares? Property owners and insurance companies cared very deeply indeed.

Construct validity is important in the “hard” physical sciences. “Temperature” is a construct. “To discuss the validity of a thermometer reading, a physical theory is necessary. The theory must posit not only that mercury expands linearly with temperature, but that water in fact boils at 100°. With such a theory, a thermometer that reads 82° when the water breaks into a boil can be reckoned inaccurate. Yet if the theory asserts that water boils at different temperatures under different ambient pressures, the same measurement may be valid under different circumstances — say at one half an atmosphere.” (Kirk and Miller, Reliability and Validity in Qualitative Research) Atmosopheric pressure varies from day to day, from hour to hour. So what is the temperature outside your window right now? The “correct” answer is surprisingly hard to decide.

In the “soft” social sciences and qualitative research, the measurement problem is even harder. Kirk and Miller go on, “In the case of qualitative observations, the issue of validity is not a matter of methodological hairsplitting about the fifth decimal point, but a question of whether the researcher sees what he or she thinks he or she sees.” (Kirk and Miller, Reliability and Validity in Qualitative Research)

When we come to the field of software development, there are certain constructs that people bandy about as though they were widgets, instead of idea-stuff: requirements; defects; test cases; tests; fixes; discoveries. What is a “programmer”? What is a “tester”? Is a programmer who spends a couple of days writing a test framework a programmer or a tester? Questions like these raise problems for anyone who wants a quantitative answer to the question, “How many testers per developer?” Kaner, Hendrickson, and Smith-Brock go into extensive detail on the subject. I’ve written about what counts before, too.

There’s a terrible difficulty in our craft: those who seem most eager to measure things seem not to pay very much attention to the problem of construct validity, as Cem Kaner and Walter P. Bond point out in this landmark paper, “Software Engineering Metrics: What Do They Measure and How Do We Know”). (I’m usually loath to say “All testers should do X”, but I think anyone serious about measurement in software development should read this paper. It’s not hard. Do it now. I’ll wait.)

If you’re doing research into software development, how do you define, describe, and justify your notion of “defects” such that you count all the things that are defects, and leave out all the things that aren’t defects, and such that your readers agree? If you’re getting reports and aggregating data from the field, how do you make sure that other people are counting the same way as you are? Does “defect” have the same meaning in a game development shop as it does for the makers of avionics software? If you’re attempting to prove something in a quantitative, rigourous and scientific way, how do you answer objections when you say something is a defect and someone else says it isn’t? How do you respond when someone wants to say that “there’s more to defects than coding errors”?

Those questions will become very important in the days to come. Stay tuned.

For extra reading: See Shadish, Cook, and Campbell, Experimental and Quasi-Experimental Designs for Generalized Causal Inference. This book is unusually expensive, but well worth it if you’re serious about measurement and validity.

Frequently-Asked Questions About the 29119 Controversy

Tuesday, September 2nd, 2014

This is a first stab at a frequently-asked questions list about the movement to stop ISO 29119. Here I speak for myself, and not for the community. If you see “we”, it refers to my perception of the community at large, but not necessarily to the whole community; your mileage may vary. There is plenty of discussion in the community; Huib Schoots is curating a collection of resources on the controversy. If you’ve got a different perception or a different opinion, please share it and let me know. Meanwhile, I hasten to point out that absolutely everyone is welcome and encouraged to share my opinions.

Q. Why bother with a community attack on ISO 29119? Isn’t it irrelevant to most testers? And why now?

To start with, we believe that ISO 29119 is irrelevant to all testers, in the sense that it seems to be an overstructured process model, focused on relentless, ponderous, wasteful bureaucracy and paperwork, with negligible content on actual testing. If your organization is in the business of producing pointless documentation, so be it, but that’s not what we call testing. The approaches suggested by 29119 might be useful to people who are more interested in ass coverage than in test coverage.

Originators and supporters of the petition are trying to establish a pattern of opposition to the standard. This becomes important when lawyers or auditors ask “Why didn’t you follow ‘an internationally agreed set of standards for software testing that can be used within any software development life cycle or organisation’?” Loud voices of opposition—not only to the standard, but also to the process by which it was created and by which it will be marketed—will help to show that the suggestion of “international agreement” is meaningless; that the standard misrepresents testing as many prominent testers see it; that the standard is overly complex and opaque; that it is both too vague here and too specific there to be useful in “any” organisation; and that radically different contexts for testing—quite appropriately—require radically different approaches for testing.

As to the “why now” question, there’s another reason for the groundswell that I think we’re discovering as we go: over the years, in fits and starts, the context-driven community has become much larger and more capable of acting like a community. And that despite the fact that people who aspire to be fiercely independent thinkers can be a fairly fractious bunch. A community that welcomes serious disagreement will have serious disagreements, and there have been some. Yet it seems that, every now and then, there are some things that are just odious enough to unite us. Personally, I’m treating this as a trial run and a learning experience to prepare for something seriously important.

Q. The promoters of the standard insist that it’s not mandatory, so what’s the fuss?

The promoters of the standard who say that the standard is not mandatory are being disingenuous. They are well aware of this idea:

“Still another classification scheme distinguishes between voluntary standards, which by themselves impose no obligations regarding use, and mandatory standards. A mandatory standard is generally published as part of a code, rule or regulation by a regulatory government body and imposes an obligation on specified parties to conform to it. However, the distinction between these two categories may be lost when voluntary consensus standards are referenced in government regulations, effectively making them mandatory standards.”


The 29119 promoters begin by using appeal to authority (in this case, the reputation of ISO) to declare a standard. If it so happens that a regulator or bureaucrat, uninformed about testing, happens upon “an internationally agreed set of standards for software testing that can be used within any software development life cycle or organisation” and refers to them in government regulations, well, then, so much the better for aspiring rent-seekers who might have been involved in drafting the standard.

Q. If ISO 29119 is so terrible, won’t it disappear under its own weight?

Yes, it probably will in most places. But for a while, some organizations (including public ones; your tax dollars at work, remember) will dally with it at great cost—including the easily foreseeable costs of unnecessary compliance, goal displacement, misrepresentation of testing, and yet another round of marketing of bogus certifications, whereby rent-seekers obtain an opportunity to pick the pockets of the naïve and the cynical.

Q. Aren’t you just griping because you’re worried that your non-standard approach to testing will put you out of business?

Here’s how I answered this question on one blog (with a couple of minor edits for typos and clarity):

“In one sense, it won’t make any difference to my business if 29119-1, 29119-2, and 29119-3 are left to stand, and if 29119-4 and 29119-5 move from draft to accepted. Rapid Software Testing is about actual testing skills—exploration, experimentation, critical thinking, scientific thinking, articulate reporting, and so forth. That doesn’t compete with 29119, in the same kind of way that a fish restaurant doesn’t compete with the companies that make canned tuna. We object to people manipulating the market and the ISO standards development process to suggest to the wider world that canned tuna is the only food fit for people to eat. I discuss that here:

“In another sense, 29119 could be fantastic for my business. It would offer me a way to extend the brand: how to do excellent, cost-effective testing that stands up to scrutiny in contexts where some bureaucrat, a long way away from the development project, was fooled into believing that 29119 was important. At the moment, I’m happy to refer that kind of business to colleagues of mine, but I suspect that it would be something of a gold mine for me. Yet still I oppose 29119, because what’s in my interest may not be in the interests of my clients and of society at large.

“Let me be specific: There are existing standards for medical devices, for avionics, and the like. Those standards matter, and many of them are concise and well-written, and were created by genuine collaboration among interested parties. Testers who are working on medical devices or on avionics software have a limited number of minutes in the working day. As someone who flies a lot, and as someone who is likely to require the help of medical devices in the foreseeable future, I would prefer that those testers spend as many minutes as humanly possible actually investigating the software, rather than complying (authentically, pathetically, or maliciously) to an unnecessary standard for process modeling, documentation, and strategising (a standard for developing a strategy—imagine that!).”

Q. You just don’t like standards. Isn’t that it?

Nope. I love standards when they’re used appropriately.

As I emphasized in a 2011 PNSQC presentation called “Standards and Deviations“, it is possible and often desirable to describe and standardize widgets—tangible, physical things that have quantifiably measurable attributes, and that must interface, interact, or fit with other things. Thank goodness for standardized screws and screwdrivers, CDs, and SATA hard drives! Bravo to the EU for mandating that power supplies for smartphones standardize on USB! Yet even with widgets, there are issues related to the tension between standards and an advancing state of the art. Here’s one of the best-ever articles on problems with standards: Joel Spolsky on Martian Headsets.

It is more difficult and to describe processes, since the description is, by necessity, a model of the process. It’s difficult for many people to avoid reifying the model—that is, to avoid treating the model—idea-stuff—as though it were a thing. For an example of reification of testing, take a few moments to reflect on the notion of representing testing work in terms of test cases; then read “Test Cases Are Not Testing: Toward a Culture of Test Performance” by James Bach & Aaron Hodder. More generally, 29119’s focus on the artifacts and the process model displace and de-centre the most important part of any testing effort: the skill set and the mindset of the individual tester.

Q. Do you really believe that ISO 29119 can be stopped?

No, of course we don’t. Curtis Stuehrenberg puts it perfectly in a discussion on LinkedIn: “The petition is not about stopping the publication any more than an anti-war march is about a reasonable expectation of ending a war through a parade. The point of the petition and the general chatter is to make sure at least some people hear there is a significant portion of the testing community who was not represented and who espouse different viewpoints and practices for software testing as a professional discipline.” If we can’t get the standard stopped by the ISO’s mechanisms, at least we can show that there is an absence of consensus outside of the 29119 working groups.

Q. The standard has been in development for the last seven years; why have you waited so long?

Some of us haven’t been waiting. For example, I gave this presentation in 2011. Some of us have been busy objecting to certification schemes. (There’s only so much rent-seeking one can oppose at once.) Several of us have argued at length and in public with some of the more prominent figures promoting the standard at conferences. They sometimes seem not to understand our objections. However, as Upton Sinclair said, “It is difficult to get a man to understand something, when his salary depends upon his not understanding it!” ( Whether through terrible argumentation or deliberate guile, the responses in those discussions usually took the form of non-sequiturs: “The standard is optional; something is better than nothing; many people were involved; the perfect is the enemy of the good; we’re trying to help all those poor people who don’t know what to do.” The standards promoters should (and probably do) know that those statements would apply to any process model that any person or group could offer. Constructing bogus authority into the ISO, and then appealing to that authority, looks an awful lot like rent-seeking to me.

Moreover, it strains believe that the standard has undergone serious development when some of the basic models (for example, 29119’s model for the test planning process) have gone essentially unchanged over four years—a period that included the rise of smartphones and mobile technology, the aftermath of the financial crisis, and the emergence of tablet computing. Testing in an Agile context reportedly garners little more than a few hand-waving references. I can’t say I’m surprised that testing and checking don’t appear on 29119’s radar either.

Q. Why didn’t you object using the formal process set up by ISO?

As James Bach points out, the real question there has been begged: why should the craft have to defend itself against a standards process that is set up to favour the determined and the well-funded? ISO is a commercial organization; not an organ of the United Nations, emanating from elected representative governments; not an academic institution; not a representative group of practitioners; not ordained by any deity. The burden is on ISO to show the relevance of the standard, even under its own terms. Simon Morley deconstructs that.

Q. Wouldn’t it be good to have an international common language for software testing?

Great idea! In fact, it would be good to have an international common language for everything. And in order to be truly international and to represent the majority of people in the world, let’s make that language Mandarin, or Hindi.

There are many arguments to show that a common language for software testing is neither desirable nor possible. I’ve blogged about a few of them, and I’ve done that more than once.

Q. Why are you always against stuff? Don’t you want to be for something?

You don’t have to be for something to be against something that’s odious. But as a matter of fact, I am for something that is more important than any standard: freedom and responsibility for the quality of my work (as I hope all testers are for freedom and responsibility for the quality of their own work). That includes the responsibility to make my work capable, credible, open to scrutiny, and as cost-effective as possible. I must be responsible to my clients, to my craft, and to society as a whole. In my view, those responsibilities do not and should not include compliance with unnecessary, time-consuming, unrepresentative standards created by self-appointed documentation and process-model enthusiasts.

Some other things I’m for: the premises of Rapid Software Testing; the Rapid Testing framework; studying the structures of exploratory testing; the Heuristic Test Strategy Model; a set of commitments for testers to make to their clients; practicing the skill of test framing; excellent reporting; and a host of other things. This is unrepresentative of the wider testing community… so I bet you’re glad that compliance with standards set by James and me is voluntary. In fact, compliance with our standards requires you to invent testing for yourself; to adopt standards that help; and to resist the ones that don’t, including ours. But if you find something else that works for you, tell us. Tell everybody.

Q. What about the poor people who need guidance on how to test?

My (free) offerings to those poor people include those just above. Those poor people are welcome to use these suggestions and to investigate the alternatives that anyone else offers. That may be harder than referring to an ISO standard and appealing to its authority. (It may be considerably easier, too.) But my first piece of guidance on how to test is this: learn about testing, and learn how to test, through study and practice. I argue that ISO 29119 will not help you with that.

Q. Okay, despite the Quixotic nature of the petition, I’m convinced. Where do I sign?

Go to And thank you.

An Example of Progress in the Drafting of ISO 29119

Monday, September 1st, 2014

The proponents of ISO Standard 29119 proudly claim that they have received and responded to “literally thousands” of comments during the process of drafting the standard. So I thought it might be interesting to examine how one component of the basic model has changed or evolved through the course of its development.

Here’s a screenshot of a diagram that illustrates the test planning process, taken from a presentation given in 2009.

(Source:, accessed September 1, 2014)

Here’s another diagram illustrating the test planning process, presumably reflecting input from all of those thousands of comments (plus changes due to the rise of smartphone and mobile technology, the consequences of the financial crisis, and the emergence of tablet computing) from a presentation from 2013:

(Source:, accessed September 1, 2014)

But maybe that second presentation was based on an interim draft. Let’s look at something that should reflect the published standard, as it came from Stuart Reid, the convenor of the standard working group, in March 2014, after the standard was published.

Source:, accessed August 21, 2014 by Huib Schoots)

Perhaps the model developed and presented by someone in 2009 was so universally representative of software test planning that it has been able to withstand the critique and feedback from a wide and diverse community of thousands of testers over four years, with only minimal and inconsequential changes to the text in the diagram. Or perhaps substantial disagreement with this model was ignored, suppressed, or subjected to a process of consensus based on attrition, as I alluded to in an earlier post.

Which do you think is more likely?

Please, sign the petition to stop ISO 29119.

The Sock Puppets of Formal Testing

Monday, July 21st, 2014

Formal testing is testing that must be done in a specific way, or to check specific facts. In the Rapid Software Testing methodology, we map the formality of testing on a continuum. Sometimes it’s important to do testing in a formal way, and sometimes it’s not so important.

Formality Continuum

From Rapid Software Testing. See

People sometimes tell me that they must test their software using a particular formal approach—for example, they say that they must create heavily documented and highly prescriptive test cases. When I ask them why, they claim it’s because of “standards” or “regulations”. For example, Americans often refer to Public Law 107-204, also known as the Sarbanes-Oxley Act of 2002, or SOX. I ask if they’ve read the Act. “Well… no.”

If you’re citing Sarbanes-Oxley compliance as a reason for producing test scripts, I worry that someone is beating you—or you’re beating yourself, perhaps—with a rumour. Sarbanes-Oxley says nothing about software testing. In fact, it says nothing about software. It does not say that you have to do software testing in any particular way. It certainly does not talk about test cases or test scripts. It does talk about testing, but only of “the auditor’s testing of the internal control structure and procedures of the issuer (of reports required by the SEC —MB)”. In this context, the word “testing” could easily be replaced by “scrutiny” or “evaluation” or “assessment”. Don’t believe me? Read it: Search for the words “test cases”, or “test scripts”, or even “test”. What do you find?

All testing comes with several kinds of cost, including development cost, maintenance cost, execution cost, transfer cost, and—perhaps most importantly—opportunity cost. (Opportunity cost means, basically, that we won’t be able to do that potentially valuable thing because we’re doing this potentially valuable thing.) Formality may provide a certain kind of value, but we must remain aware of its cost. Every moment that we spend on representing our test ideas as documentation, or on preparing to test in a specific way, or on encoding a testing activity, or on writing about something that we’ve done is a moment in which we’re not interacting directly with the product. More formality can be valuable if it’s helping us learn, or if it’s helping us structure our work, or if it helps us to tell the testing story—all such that the value of the activity meets its cost. The key question to ask, it seems to me, is “is it helping?” If it’s not helping, is it necessary?—and what is the basis of the claim that it’s necessary? Asking such questions helps to underscore that many things we “have to” do are not obligatory, but choices.

Formal testing is often said to involve comparing the product to the requirements for it, based on the unspoken assumption that “requirements” means “documented requirements”. Certainly some requirements can be understood and documented before we begin testing the product. Other requirements are tacit, rather than explicit. (As a trivial example, I’ve never seen a requirements document that notes that the product must be installed on a working computer.) Still other requirements are not well-understood before development begins; the actual requirements are discovered, revealed, and refined as we develop the product and our understanding of the problems that the product is intended to solve. As my colleague James Bach has said, “Requirements are ideas at the intersection between what we want, what we can have and how we might decide that we got it. In complex systems that requires testing. Testing is how we learn difficult things.” That learning happens in tangled loops of development, experimentation, investigation, and decision-making. Our ideas may or may not be documented formally—or at all—at any specific point in time. Indeed, some things are so important that we never document them; we have to know them.

So it goes with testing. Some aspects of our testing will be clear from the outset of the project; other aspects of our testing will be discovered, invented, and refined as we develop and test the product. Excellent formal testing, as James says, must therefore rooted in excellent informal testing. And who agrees with that? The Food and Drug Administration, for one, in the Agency’s “Design Considerations for Pivotal Clinical Investigations for Medical Devices“.

Few (in my experience, very few) of those people who insist on highly formal testing to comply with standards or regulations—or even with requirements documents—have read the relevant documents or examined them closely. In such cases, the justification for the formality is based on rumours and on going through the motions; the formality is premature, or excessively expensive, or unnecessary, or a distraction; and (in)consistency with the documentation and (non-)compliance with the regulations becomes accidental. At its worst, the testing falls into what James calls “pathetic compliance”, following “the rules” so we don’t “get in trouble”—a pre-school level of motivated behaviour.

As responsible testers, it is our obligation to understand what we’re testing, why we’re testing it, and why we’re testing it in a particular way. If we’re responsible for testing to comply with a particular standard, we must learn that standard, and that means studying it, understanding it, and relating it to our product and to our testing work. That may sound like a lot of work, and often it is. Yet our choice is to do that work, or to run the risk of letting our testing be structured and dictated to us by a sock puppet. Or a SOX puppet.

Sock Puppet

Photo credit: iStockPhoto