DevelopsenseLogo

Frequently-Asked Questions About the 29119 Controversy

This is a first stab at a frequently-asked questions list about the movement to stop ISO 29119. Here I speak for myself, and not for the community. If you see “we”, it refers to my perception of the community at large, but not necessarily to the whole community; your mileage may vary. There is plenty of discussion in the community; Huib Schoots is curating a collection of resources on the controversy. If you’ve got a different perception or a different opinion, please share it and let me know. Meanwhile, I hasten to point out that absolutely everyone is welcome and encouraged to share my opinions.

Q. Why bother with a community attack on ISO 29119? Isn’t it irrelevant to most testers? And why now?

To start with, we believe that ISO 29119 is irrelevant to all testers, in the sense that it seems to be an overstructured process model, focused on relentless, ponderous, wasteful bureaucracy and paperwork, with negligible content on actual testing. If your organization is in the business of producing pointless documentation, so be it, but that’s not what we call testing. The approaches suggested by 29119 might be useful to people who are more interested in ass coverage than in test coverage.

Originators and supporters of the petition are trying to establish a pattern of opposition to the standard. This becomes important when lawyers or auditors ask “Why didn’t you follow ‘an internationally agreed set of standards for software testing that can be used within any software development life cycle or organisation’?” Loud voices of opposition—not only to the standard, but also to the process by which it was created and by which it will be marketed—will help to show that the suggestion of “international agreement” is meaningless; that the standard misrepresents testing as many prominent testers see it; that the standard is overly complex and opaque; that it is both too vague here and too specific there to be useful in “any” organisation; and that radically different contexts for testing—quite appropriately—require radically different approaches for testing.

As to the “why now” question, there’s another reason for the groundswell that I think we’re discovering as we go: over the years, in fits and starts, the context-driven community has become much larger and more capable of acting like a community. And that despite the fact that people who aspire to be fiercely independent thinkers can be a fairly fractious bunch. A community that welcomes serious disagreement will have serious disagreements, and there have been some. Yet it seems that, every now and then, there are some things that are just odious enough to unite us. Personally, I’m treating this as a trial run and a learning experience to prepare for something seriously important.

Q. The promoters of the standard insist that it’s not mandatory, so what’s the fuss?

The promoters of the standard who say that the standard is not mandatory are being disingenuous. They are well aware of this idea:

“Still another classification scheme distinguishes between voluntary standards, which by themselves impose no obligations regarding use, and mandatory standards. A mandatory standard is generally published as part of a code, rule or regulation by a regulatory government body and imposes an obligation on specified parties to conform to it. However, the distinction between these two categories may be lost when voluntary consensus standards are referenced in government regulations, effectively making them mandatory standards.”

(Source: http://www.nist.gov/standardsgov/definestandards.cfm)

The 29119 promoters begin by using appeal to authority (in this case, the reputation of ISO) to declare a standard. If it so happens that a regulator or bureaucrat, uninformed about testing, happens upon “an internationally agreed set of standards for software testing that can be used within any software development life cycle or organisation” and refers to them in government regulations, well, then, so much the better for aspiring rent-seekers who might have been involved in drafting the standard.

Q. If ISO 29119 is so terrible, won’t it disappear under its own weight?

Yes, it probably will in most places. But for a while, some organizations (including public ones; your tax dollars at work, remember) will dally with it at great cost—including the easily foreseeable costs of unnecessary compliance, goal displacement, misrepresentation of testing, and yet another round of marketing of bogus certifications, whereby rent-seekers obtain an opportunity to pick the pockets of the naïve and the cynical.

Q. Aren’t you just griping because you’re worried that your non-standard approach to testing will put you out of business?

Here’s how I answered this question on one blog (with a couple of minor edits for typos and clarity):

“In one sense, it won’t make any difference to my business if 29119-1, 29119-2, and 29119-3 are left to stand, and if 29119-4 and 29119-5 move from draft to accepted. Rapid Software Testing is about actual testing skills—exploration, experimentation, critical thinking, scientific thinking, articulate reporting, and so forth. That doesn’t compete with 29119, in the same kind of way that a fish restaurant doesn’t compete with the companies that make canned tuna. We object to people manipulating the market and the ISO standards development process to suggest to the wider world that canned tuna is the only food fit for people to eat. I discuss that here: http://www.developsense.com/blog/2014/08/rising-against-the-rent-seekers/

“In another sense, 29119 could be fantastic for my business. It would offer me a way to extend the brand: how to do excellent, cost-effective testing that stands up to scrutiny in contexts where some bureaucrat, a long way away from the development project, was fooled into believing that 29119 was important. At the moment, I’m happy to refer that kind of business to colleagues of mine, but I suspect that it would be something of a gold mine for me. Yet still I oppose 29119, because what’s in my interest may not be in the interests of my clients and of society at large.

“Let me be specific: There are existing standards for medical devices, for avionics, and the like. Those standards matter, and many of them are concise and well-written, and were created by genuine collaboration among interested parties. Testers who are working on medical devices or on avionics software have a limited number of minutes in the working day. As someone who flies a lot, and as someone who is likely to require the help of medical devices in the foreseeable future, I would prefer that those testers spend as many minutes as humanly possible actually investigating the software, rather than complying (authentically, pathetically, or maliciously) to an unnecessary standard for process modeling, documentation, and strategising (a standard for developing a strategy—imagine that!).”

Q. You just don’t like standards. Isn’t that it?

Nope. I love standards when they’re used appropriately.

As I emphasized in a 2011 PNSQC presentation called “Standards and Deviations“, it is possible and often desirable to describe and standardize widgets—tangible, physical things that have quantifiably measurable attributes, and that must interface, interact, or fit with other things. Thank goodness for standardized screws and screwdrivers, CDs, and SATA hard drives! Bravo to the EU for mandating that power supplies for smartphones standardize on USB! Yet even with widgets, there are issues related to the tension between standards and an advancing state of the art. Here’s one of the best-ever articles on problems with standards: Joel Spolsky on Martian Headsets.

It is more difficult and to describe processes, since the description is, by necessity, a model of the process. It’s difficult for many people to avoid reifying the model—that is, to avoid treating the model—idea-stuff—as though it were a thing. For an example of reification of testing, take a few moments to reflect on the notion of representing testing work in terms of test cases; then read “Test Cases Are Not Testing: Toward a Culture of Test Performance” by James Bach & Aaron Hodder. More generally, 29119’s focus on the artifacts and the process model displace and de-centre the most important part of any testing effort: the skill set and the mindset of the individual tester.

Q. Do you really believe that ISO 29119 can be stopped?

No, of course we don’t. Curtis Stuehrenberg puts it perfectly in a discussion on LinkedIn: “The petition is not about stopping the publication any more than an anti-war march is about a reasonable expectation of ending a war through a parade. The point of the petition and the general chatter is to make sure at least some people hear there is a significant portion of the testing community who was not represented and who espouse different viewpoints and practices for software testing as a professional discipline.” If we can’t get the standard stopped by the ISO’s mechanisms, at least we can show that there is an absence of consensus outside of the 29119 working groups.

Q. The standard has been in development for the last seven years; why have you waited so long?

Some of us haven’t been waiting. For example, I gave this presentation in 2011. Some of us have been busy objecting to certification schemes. (There’s only so much rent-seeking one can oppose at once.) Several of us have argued at length and in public with some of the more prominent figures promoting the standard at conferences. They sometimes seem not to understand our objections. However, as Upton Sinclair said, “It is difficult to get a man to understand something, when his salary depends upon his not understanding it!” (http://en.wikiquote.org/wiki/Upton_Sinclair) Whether through terrible argumentation or deliberate guile, the responses in those discussions usually took the form of non-sequiturs: “The standard is optional; something is better than nothing; many people were involved; the perfect is the enemy of the good; we’re trying to help all those poor people who don’t know what to do.” The standards promoters should (and probably do) know that those statements would apply to any process model that any person or group could offer. Constructing bogus authority into the ISO, and then appealing to that authority, looks an awful lot like rent-seeking to me.

Moreover, it strains believe that the standard has undergone serious development when some of the basic models (for example, 29119’s model for the test planning process) have gone essentially unchanged over four years—a period that included the rise of smartphones and mobile technology, the aftermath of the financial crisis, and the emergence of tablet computing. Testing in an Agile context reportedly garners little more than a few hand-waving references. I can’t say I’m surprised that testing and checking don’t appear on 29119’s radar either.

Q. Why didn’t you object using the formal process set up by ISO?

As James Bach points out, the real question there has been begged: why should the craft have to defend itself against a standards process that is set up to favour the determined and the well-funded? ISO is a commercial organization; not an organ of the United Nations, emanating from elected representative governments; not an academic institution; not a representative group of practitioners; not ordained by any deity. The burden is on ISO to show the relevance of the standard, even under its own terms. Simon Morley deconstructs that.

Q. Wouldn’t it be good to have an international common language for software testing?

Great idea! In fact, it would be good to have an international common language for everything. And in order to be truly international and to represent the majority of people in the world, let’s make that language Mandarin, or Hindi.

There are many arguments to show that a common language for software testing is neither desirable nor possible. I’ve blogged about a few of them, and I’ve done that more than once.

Q. Why are you always against stuff? Don’t you want to be for something?

You don’t have to be for something to be against something that’s odious. But as a matter of fact, I am for something that is more important than any standard: freedom and responsibility for the quality of my work (as I hope all testers are for freedom and responsibility for the quality of their own work). That includes the responsibility to make my work capable, credible, open to scrutiny, and as cost-effective as possible. I must be responsible to my clients, to my craft, and to society as a whole. In my view, those responsibilities do not and should not include compliance with unnecessary, time-consuming, unrepresentative standards created by self-appointed documentation and process-model enthusiasts.

Some other things I’m for: the premises of Rapid Software Testing; the Rapid Testing framework; studying the structures of exploratory testing; the Heuristic Test Strategy Model; a set of commitments for testers to make to their clients; practicing the skill of test framing; excellent reporting; and a host of other things. This is unrepresentative of the wider testing community… so I bet you’re glad that compliance with standards set by James and me is voluntary. In fact, compliance with our standards requires you to invent testing for yourself; to adopt standards that help; and to resist the ones that don’t, including ours. But if you find something else that works for you, tell us. Tell everybody.

Q. What about the poor people who need guidance on how to test?

My (free) offerings to those poor people include those just above. Those poor people are welcome to use these suggestions and to investigate the alternatives that anyone else offers. That may be harder than referring to an ISO standard and appealing to its authority. (It may be considerably easier, too.) But my first piece of guidance on how to test is this: learn about testing, and learn how to test, through study and practice. I argue that ISO 29119 will not help you with that.

Q. Okay, despite the Quixotic nature of the petition, I’m convinced. Where do I sign?

Go to http://www.ipetitions.com/petition/stop29119. And thank you.

14 replies to “Frequently-Asked Questions About the 29119 Controversy”

  1. Hi Michael,

    I’m also part of the movement, as I happily signed the petition against the standard. During the last 2 years, I have emerged into doctoral studies in software engineer, particularly in testing. One of the biggest challenges is to try to understand both sides IEEE guys and the Professional Testers community as both parties have very different approaches in documenting and promoting testing.

    Michael replies: It remains an enduring mystery to me why this should be.

    Even though I’m a big fan of RST & CDT I think at some point the purpose of standardization has been deflected. Hearing and getting involved in the petition I started to grasp through SWEBOK as promoted by the Computer Society. It got my attention since Foreword, stating that “the work is partial fulfillment to promote both theory and practice for the profession of software engineering. Also, it does not represent the entire body of knowledge, but rather serves as a guide to what has been developed over more than four decades”

    Or, more accurately, what was developed four decades ago.

    I’m fully aware of the gap between academia and industry and there’s a long way to bridge these two, still we shouldn’t take things very literally since the world hasn’t been built on standards, but experience and hard work. I strongly believe that there’s plenty of space to put effort in making academia aware on our practices and vice-versa.

    Remember that academia is one thing; the IEEE is another; the practitioner community yet another. Part of the problem is that, in academia, testing is treated as a branch of software development, which in turn is treated as a branch of computer science, which in turn is treated as a branch of mathematics. From the perspective of computer science, testing is theorem-proving. Nothing wrong with that, as far as it goes—but it doesn’t go very far. Software development is more like engineering: multi-disciplinary, with mathematics as an important component. But engineering done well also includes design; human factors; systems thinking; questions of cost, value, and risk; even anthropology and philosophy. Testing is suffused with philosophical questions: what do we know? How do we know that we know it? What sorts of things matter? What doesn’t matter so much?

    Academics would benefit from observing expert testers at work, and studying testing as a social science.

    Thanks!

    You’re welcome. Thank YOU for the comment.

    Reply
  2. Ok, so is it a craft or is it a profession? By “it” I mean testing. How can we say we have a profession? In the absence of standardization we are left with a craft. Do you agree with this statement?

    Michael replies: Beware the binary-thinking trap; it doesn’t have to be one or the other exclusively. Let’s look at what Oxford says. A profession is 1) a vocation or calling, especially one that involves some branch of of advanced learning or science; 2) a body of people engaged in a profession. Those are the two definitions that refer to something that we might otherwise call a craft.

    Rent-seeking standardizers and certificationists love to insist that you can’t have a profession without formal, exclusionary (and, typically, pricey) “professional standards”. Also, “left with a craft”, as you’ve said it, suggests something pejorative about “craft”, of which Oxford says 1) skill, especially in practical arts a) (especially in combination with) a trade or an art b) the members of a craft. I aspire to practice the craft and profession of software testing professionally, as a craftsman, and many colleagues do too. So, no, I don’t agree with the statement.

    I read your FAQ and the title is a bit misleading. Allow me to explain: this blog post is more of a why M.B. is against the standard rather than a “Frequently Asked Questions about the 29119 Controversy”. Do you agree with that statement?

    It’s the binary thinking trap again here too. Some of post certainly explains why I oppose 29119 (as I make clear immediately in the body of the text), but it’s also true that these are questions that people have asked me and other colleagues who oppose 29119 and support the petition. Moreover, I have not answered questions like “Do you really believe that ISO 29119 can be stopped?” with a reason why I oppose the standard; to do so would be a non-sequitur. So the title is fine, in my view, and no, I don’t agree with the statement.

    Reply
  3. Great, I can live with that. So one final question why oppose something? Why not be for something instead. It takes a lot of energy to be against something and it’s also a negative activity.

    When folks are for something, rather than against it, its a positive activity, it feels good to be for something while it feels bad to be against something. “what you resist, persists”.

    Mother Theresa always said don’t invite me to an anti war rally, invite me to a pro peace rally. I think she fully understood the meaning of “what you resist, persists”.

    Michael replies: First of all, you don’t have to be for something to be against something else. You don’t have to be for anything in particular to be against genital mutilation. You don’t have to demand an alternative to anti-cancer medication with life-threatening and unpredictable side effects ever, and especially if you don’t have cancer.

    But in spite of that, here’s something I’m for: Primum non nocere is a principle in medicine; first, do no harm. Taleb, in Antifragile, refers to Via Negativa, “the way of nothing”, the approach of doing nothing where no action is necessary. I wish the certificationists were able to recognize this principle. So I’m for telling people to take a hike when they’re messing with the craft.

    But here’s something I’m also for: reading the whole post. I think you might have missed it, so here’s a link to where I answered that question already.

    Thanks for sharing your space for my feedback.
    Warm Regards.

    You’re welcome.

    Reply
  4. “This becomes important when lawyers or auditors ask “Why didn’t you follow ‘an internationally agreed set of standards for software testing that can be used within any software development life cycle or organisation’?”

    In my 16 year career I’ve been asked a lot of things by auditors, but I’ve never had or heard of an auditor (or customer for that matter) asking that question about ISO/IEC 9126, IEEE 12207, ISO/IEC 25000 or any other standard that has pretended to cover software testing. But if they did, putting such a line of inquiry to rest would be trivial. If the testing community needs a “pattern of opposition” to deal with an unlikely and unchallenging question, then the testing community has bigger problems than ISO 29119.

    Michael replies: In your 16-year career, how much time have you spent in organizations such as banks, big finance, government, big pharma, where standards play a significant role? Have you avoided such questions because you’ve been lucky enough not to work in such organizations, or to have sympathetic auditor?

    The testing community does have bigger problems than ISO 29119. For example, there’s the common pattern, “I haven’t experienced a certain problem, therefore it’s not a problem”. Here’s another one: “I’ve managed to avoid dealing with demands that I’ve been certified, therefore it’s not a problem.” Here’s a third: “I think of testing solely in terms of automated checking, therefore I’m testing well.”

    “But for a while, some organizations (including public ones; your tax dollars at work, remember) will dally with it at great cost”

    So? Arguably only inept organisations would adopt ISO 29119. Inept organisations dally with everything at great cost. It’s possible that adopting ISO 29119 may cost less than what they’d do without it. The threat of ISO 29119 comes down to who is likely to adopt it and whether that makes a material difference to what they would do otherwise. This point deserves more than an unsubstantiated assertion sprinkled with FUD…if ISO 29119 was a serious threat…but it’s not.

    Here’s another pattern: “Arguably only inept organizations would adopt ISO 29119”, begging the question of who pays for ineptness—and not only in the public sector, but also in the private sector. If the point deserves more than that, you’re welcome and encouraged to make a point. You’re also encouraged to think like a tester. What I’m seeing instead is an overly optimistic developer who, when confronted by the possibility of a problem in the product, says “FUD… if that bug were a serious bug… but it’s not.” Asserting that there’s no risk because you don’t see a risk is very untesterly of you, I must say.

    As you said;

    “Personally, I’m treating this as a trial run and a learning experience to prepare for something seriously important.”

    But there are many seriously important issues and challenges facing testing and software development (that testers could help address) today. At a time when companies are achieving billion dollar revenues and market capitalisations without any testers, when startups view testers as anathema, when Microsoft is laying off thousands of testers because they believe the role is an impediment to quality, it is disconcerting to see the CDT community choose opposing a glass jawed anachronism like ISO 29119 as its priority.

    The CDT tends to pay attention when thoughtful people raise serious problems, as the community did when James Christie gave his talk. On the one hand, it seems that you’re suggesting that I should be writing about other things—except if you look around, you’ll notice that I’ve been doing that about a bunch of subjects over the last 10 years or so. Meanwhile, if you think there are many seriously important issues and challenges facing testing and software development (that testers could help address) today, again: you are welcome and encouraged to help address them. And on your own blog, as well as this one.

    Reply

Leave a Comment