Blog Posts for the ‘Standards’ Category

A Response to Anne Mette Hass

Saturday, September 20th, 2014

In response to my earlier blog post, I received this comment from Anne Mette Hass. I’m going to reproduce it in its entirety, and then I’ll respond to each point.

I think this ‘war’ against the ISO standard is so sad. Nobody has set out to or want to harm anybody else with this work. The more variation we have in viewpoints and the more civilized be can debate, the wiser we get as times go by.

There is no way the standard ever wanted to become or ever will become a ‘kalifat’.

So why are you ‘wasting’ so much time and energy on this? What are you afraid of?

Best regards,
Anne Mette

PS. I’m likely not to answer if you answer me – I have better things to do.

And now my response:

Anne Mette,

It may surprise you to hear that I agree with many of the conclusions that you’ve given here. The trouble is that I don’t agree with your premises.

I think this ‘war’ against the ISO standard is so sad.

I agree that it’s sad.

It’s sad that a small group of people and/or organizations have decided unilaterally to proclaim an “internationally-recognised and agreed standard”, hiding behind ISO processes and implicitly claiming it to be based on consensus of the affected stakeholders, when it is manifestly not.

It’s sad that the working group has proceeded to declare a “standard” when the convenor of the working group has admitted that it has no evidence of efficacy. Those who claim to be expert testers would raise the alarm about an inefficacious product, would gather evidence, and would investigate. The “standard” has not been tested with actual application in the field.

It’s sad when the convenor of the working group treats “craftsmen” as a word with a negative connotation.

It’s sad that those people producing the “standard” have so stubbornly and aggressively ignored the breadth of ideas in the craft of testing, preferring to adopt a simplistic and shallow syllabus that was developed through a similar process. (“The ISO 29119 is based on the ISTQB syllabi, and, as far as I understand, it is the intention that the ISO 29119 testing process and testing documentation definitions will be adopted by ISTQB over time.” —Anne Mette Hass, https://www.facebook.com/permalink.php?story_fbid=10152731672549009&id=144926394008)

It’s sad that members of the Working Group would issue blatantly contradictory and false statements about their intentions (“There is also no link between the ISO/IEC/IEEE Testing Standards and the ISTQB tester certification scheme.” —Stuart Reid, http://softwaretestingstandard.org/29119petitionresponse.php).

It is sad that ISO’s reputation stands a chance of being tarnished by this debacle. It’s not that the opponents of 29119 are opposed to standards. There is a place for standards in physical things that need to be interoperable. There is a place for standardization in communication protocols. There is no place for standardization of testing when the business of technology development requires variation and deviation from the norm.

War is a predictable response when people attempt to game political systems and invade territories. Wars happen when one group of people imposes its beliefs and its way of life over another group of people. War is what happens when politics fails. And war is always sad.

Nobody has set out to or want to harm anybody else with this work.

I’m aware of several people who worked on the ISTQB certification scheme. They entered into that with good will and the best of intentions, hoping to contribute new ideas, alternative views, and helpful critique. They have reported, both publicly and privately, that their contributions were routinely rejected or ignored until eventually they gave up in frustration. This was a pattern that carried over from the Software Engineering Body of Knowledge, the CMM, IEEE 829 and other standards.

Some people have asked “If you don’t like the way the standard turned out, why didn’t you get involved with the development of the standard?” This is the equivalent of saying “If you don’t like where the hijacked plane landed, why didn’t you put on a mask and join us when we stormed the cockpit?”

Even for people of good will who stuck with the effort, the road to hell is paved with good intentions. Whatever your motives, it is important to consider reasonably foreseeable consequences of your actions. On Twitter, Laurent Bossavit has said “ISO 29119 may be software testing’s Laetrile. Never proven, toxic side effects, sold as ‘better than nothing’ to the desperate and unwary. And I do mean ‘sold’, at $200 each chapter. That’s de rigueur when selling miracle cures. (More on Laetrile: http://t.co/R6J9V6OMIs)” Similarly, I’m sure that Jenny McCarthy meant to harm no one with her ill-informed and incorrect claims that vaccinations caused autism. But Jenny McCarthy is neither doctor, nor scientist, nor tester.

Here’s what we’ve seen from the community’s experience with the ISTQB: whatever good intentions anyone might have had, a lot of money has been transferred from organizations and individuals into the pockets of people who have commercialized the certification. Testers have been not only threatened with unemployment, but prevented from working unless or until they have become certified. And ISO 29119 plows the soil for another crop of certification schemes.

The more variation we have in viewpoints and the more civilized be can debate, the wiser we get as times go by.

I agree with that too. I’m all for variation in viewpoints. The trouble is, by definition, the purpose of a standard is to suppress variation. That’s what makes it “standard”. I, for one, would enthusiastically join a civilized debate (please inform me of any point at which you believe my discourse in this matter has been uncivilized), but it appears that you are dismissing the idea out of hand: I refer readers to your postscript.

There is no way the standard ever wanted to become or ever will become a ‘kalifat’.

I agree there too. The “standard” doesn’t want anything. My concern is about the people who develop and promote the standard‐what they want.

So why are you ‘wasting’ so much time and energy on this? What are you afraid of?

I don’t think that anyone who is opposing the “standard” is wasting time at all. I’m a tester. It’s my job and my passion. When someone is attempting to claim a “standard” approach to my craft, I’m disposed to investigate the claim. Notice that the opponents of the “standard” are the ones doing vigourous investigative work and looking for bugs; the development team is showing no sign of doing it. When you ask “why are you wasting so much time and energy on this?” it reminds me of a developer who doesn’t believe that his product should be tested.

I’m not worried about the “standard” becoming a caliphate; I’m concerned about it becoming anything more than a momentary distraction. And I’m not afraid; I’m anticipating and I’m protesting. Specifically, I’m anticipating and protesting

  • another round of dealing with uninformed managers and human resource people requiring candidates to have experience in or credentials for yet another superficial body of “knowledge”;

  • another round of bogus certification schemes that pick the pockets of naïve or vulnerable testers, especially in developing countries;

  • another several years of having to persuade innocent managers that intellectual work cannot and should not be “standardised”, turned into bureaucracy and paperwork;

  • another several years of explaining that, despite what some “standard” says, a linear process model for testing (even one that weasels out and says that some iteration may occur) is deeply flawed and farcical;

  • the gradual drift of the “voluntary” “standard” into mandatory compliance, as noted by the National Institute of Standards and Technology (the second last paragraph here) and as helpfully offered by ISO.

  • waste associated with having to decide whether to follow given points in the standard or reject them. (Colleagues who have counted report that there are over 100 points at which a “standard-compliant” organization must identify its intention to follow or deviate from the “standard”. That’s overhead and extra effort for any organization that wants simply to do a good job of testing on its own terms.)

  • goal displacement as organizations orient themselves towards complying to the letter of the standard rather than, say, testing to help make sure that their products don’t fail, or harm people, or kill people.

Best regards,
Anne Mette

PS. I’m likely not to answer if you answer me – I have better things to do.

Since Anne Mette evinces no intention of responding, I will now address the wider community.

There’s an example of a response from the 29119 crowd, folks. This one makes no attempt to address any of the points raised in my post; presents not a single reasoned argument; nor any supporting evidence. Mind, you don’t need supporting evidence when you don’t present an argument. But at least we get a haughty dismissal from someone who has “better things to do” than to defend the quality of the work.

Rising Against the Rent-Seekers

Monday, August 25th, 2014

At CAST 2014, a quiet, modest, thoughtful, and very experienced man named James Christie gave a talk called “Standards: Promoting Quality or Restricting Competition?”. The talk followed on from his tutorial at EuroSTAR 2013 on working with auditors—James is a former auditor himself—and from his blogs on software standards over the years.

James’ talk introduced to our community the term rent-seeking. Rent-seeking is the act of using political means—the exercise of power—to obtain wealth without creating wealth; see http://www.econlib.org/library/Enc/RentSeeking.html and http://en.wikipedia.org/wiki/Rent-seeking. One form of rent-seeking is using regulations or standards in order to create or manipulate a market for consulting, training, and certification.

James’ CAST presentation galvanized several people in attendance to respond to ISO Standard 29119, the most recent rent-seeking scheme by a very persistent group of certificationists and standards promoters. Since the ISO standard on standards requires—at least in theory—consensus from industry experts, some people proposed a petition to demonstrate opposition and the absence of consensus amongst skilled testers. I have signed this petition, and I urge you to read it, and, if you agree, to sign it too.

Subsequently, a publication named Professional Tester published—under an anonymous byline—a post about the petition, with the provocative title “Book burners threaten (old) new testing standard”. Presumably such (literally) inflammatory language was meant as clickbait. Ordinarily such things would do little to foster thoughtful discussion about the issues, but it prompted some quite thoughtful reactions. Here’s one example; here’s another. Meanwhile, if the author wishes to characterize me as a book burner, here are (selected) contents of my library relevant to software testing. Even the lamest testing books (and some are mighty lame) have yet to be incinerated.

In the body text, the anonymous author mischaracterises the petition and its proponents, of which I am one. “Their objection,” (s)he says, “is that not everyone will agree with what the standard says: on that criterion nothing would ever be published.” I might not agree with what the standard says, but that’s mostly a side issue for the purposes of this post. I disagree with what the authors of the standard attempt to do with it.

1) To prescribe expensive, time-consuming, and wasteful focus on bloated process models and excessive documentation. My concern here is that organizations and institutions will engage in goal displacement: expending money, time and resources on demonstrating compliance with the standard, rather than on actually testing their products and services. Any kind of work presents opportunity cost; when you’re doing something, most of the time it prevents you from doing something else. Every minute that a tester spends on wasteful documentation is a minute that the tester cannot fulfill the overarching mission of testing: learning about the product, with an emphasis on discovering important problems that threaten value or safety, so that our clients can make informed decisions about problems and risks.

I am not objecting here to documentation, as the calumny from Professional Tester suggests. I am objecting to excessive and wasteful documentation. Ironically, the standard itself provides an example: the current version of ISO 29119-1 runs to 64 pages; 29119-2 has 68 pages; and 29119-3 has 138 pages. If those pages follow the pattern of earlier drafts, or of most other ISO documents, you have a long, pointless, and sleep-inducing read ahead of you. Want a summary model of the testing process? Try this example of what the rent-seekers propose as their model of of testing work. Note the model’s similarity to that of a (overly complex and poorly architected) computer program.

2) To set up an unnecessary market for training, certification, and consultancy in interpreting and applying the standard. The primary tactic here is to instill the fear of being de-certified. We’ve been here before, as shown in this post from Tom DeMarco (date uncertain, but it seems to have been written prior to 2000).

Rent-seeking is of the essence, and we’ve been here before in another sense: this was one of the key goals of the promulgators of the ISEB and ISTQB. In the image, they’ve saved the best for last.

The well-informed reader will note that the list of organizations behind those schemes and the members of the ISO 29119 international working group look strikingly similar.

If the working group happens to produce a massive and opaque set of documents, and you’re in an environment that claims conformance to the 29119 standards, and you want to get some actual testing work done, you’ll probably find it helpful to hire a consultant to help you understand them, or to help defend you from charges that you were not following the standard. Maybe you’ll want training and certification in interpreting the standard—services that the authors’ consultancies are primed to offer, with extra credibility because they wrote the standards! Good thing there are no ethical dilemmas around all of this.

3) To use the ISO’s standards development process to help suppress dissent. If you want to be on the international working group, it’s a commitment to six days of non-revenue work, somewhere in the world, twice a year. The ISO/IEC does not pay for travel expenses. Where have international working group meetings been held? According to the http://softwaretestingstandard.org/ Web site, meetings seem to have been held in Seoul, South Korea (2008); Hyderabad, India (2009); Niigata, Japan (2010); Mumbai, India (2011); Seoul, South Korea (2012); Wellington New Zealand (2013). Ask yourself these questions:

  • How many independent testers or testing consultants from Europe or North America have that kind of travel budget?

  • What kinds of consultants might be more likely to obtain funding for this kind of travel?

  • Who benefits from the creation of a standard whose opacity demands a consultant to interpret or to certify?

Meanwhile, if you join one of the local working groups, there are two ways that the group arrives at consensus.

  • By reaching broad agreement on the content. (Consensus, by the way, does not mean unanimity—that everyone agrees with the the content. It would be closer to say that in a consensus-based decision-making process, everyone agrees that they can live with the content.) But, if you can’t get to that, there’s another strategy.

  • By attrition. If your interest is in promulgating an unwieldy and opaque standard, there will probably be objectors. When there are, wait them out until they get frustrated enough to leave the decision-making process. Alan Richardson describes his experience with ISEB in this way.

In light of that, ask yourself these questions:

  • How many independent consultants have the time and energy to attend local working groups, often during otherwise billable hours?

  • What kinds of consultants might be more likely to support attendance at local working groups?

  • Who benefits from the creation of a standard that needs a consultant to interpret or to certify?

4) To undermine the role of skill in testing, and the reputations of people who discuss and promote it. “The real reason the book burners want to suppress it is that they don’t want there to be any standards at all,” says the polemicist from Professional Tester. I do want there to be standards for widgets and for communication protocols, but not for complex, cognitive, context-sensitive intellectual work. There should be standards for designed things that are intended to work together, but I’m not at all sure there should be mandated standards for how to do design. S/he goes on: “Effective, generic, documented systematic testing processes and methods impact their ability to depict testing as a mystic art and themselves as its gurus.” Far from treating testing as a mystic art, appealing to things like “intuition” and “experienced-based techniques”, my community has been trying to get to the heart of testing skills, flexible and responsive coverage reporting, tacit and explict knowledge, and the premises of the way we do testing. I’ve seen no such effort to dig deeper into these subjects—and to demystify them—from the rent-seekers.

Unlike the anonymous author at Professional Tester, I am willing to stand behind my work, my opinions, and my reputation by signing my name and encouraging comments. Feel free.

—Michael B.