Continuing from yesterday…

As you may recall, my correspondent remarked

“To be honest, I don’t care what these types of verification are called be it automated checking or manual testing or ministry of John Cleese walks. What I would like to see is investment and respect being paid to testing as a profession rather than arguing with ourselves over semantics.”

Here’s an example of the importance of semantics in testing. When someone says, “it works”, what do they mean? In the Rapid Testing class, we say that “it works” really means “it appears to meet some requirement to some degree”.

That expanded statement should raise a bunch of questions for a tester, a test manager, or a client: What observations gave rise to that appearance? Observations by whom? When did that person observe? Under what conditions? Did he induce any variation into those conditions? Did he observe repeatedly, or just the once? For a long time, or just a glance? If the observation was made somewhere, what might be different somewhere else? Has anything changed since the last observation? Which requirements were considered? Which requirement specifically seems to have been met? Only that requirement? Are there other explicit or implicit requirements that might have not been met? To some degree–to what degree? Have the right people been consulted on the degree to which the requirement seems to have been met? Do they agree? Is the requirement and the evidence that “it works” well-understood by and acceptable to other people who might matter? Yes, that’s a lot of questions—and testers who ask them find nests of bugs living underneath the questions, and in the cracks between the answers.

Semantics has its parallel in science and measurement, in the form of a concept called “construct validity”. In measurement, construct validity centres around the issue of what counts as an instance of something you’re measuring, and what doesn’t count as such an instance. In their book Experimental and Quasi-Experimental Designs for Generalized Causal Inference (I have no idea why a book with such a catchy title isn’t flying off the shelves), Shadish, Cook and Campbell say,

“The naming of things is a key problem in all sciences, for names reflect category memberships that themselves have implications about relationships to other concepts, theories, and uses. This is true even for seemingly simple labeling problems. For example, a recent newspaper article reported a debate among astronomers over what to call 18 newly discovered celestial objects… The Spanish astronomers who discovered the bodies called them planets, a choice immediately criticized by some other astronomers… At issue was the lack of a match between some characteristics of the 18 objects (they are drifting freely through space and are only about 5 million years old) and some characteristics that are prototypical of planets (they orbit a star and require tens of millions of years to form). Critics said these objects were more reasonably called brown dwarfs, objects that are too massive to be planets but not massive enough to sustain the thermonuclear processes in a star. Brown dwarfs would drift freely and be young, like these 18 objects. The Spanish astronomers responded that these objects are too small to be brown dwarfs and are so cool that they could not be that young.” (p. 66)

Well, tomayto-tomahto, right? There are these objects out there, and they’re out there no matter what we call them. Brown dwarfs, planets… why bother quibbling? Shadish, Cook, and Campbell answer: “All this is more than just a quibble: if these objects really are planets, then current theories of how planets form by condensing around a star are wrong!” (my emphasis)

They end the passage by noting that construct validity is a much more difficult problem in social science field experiments—and I would argue that most of software testing is far closer to the field sciences than to astrophysics. More on that in future blog posts.

(Shadish, Cook and Campell cite the newspaper article as “Scientists quibble on calling discovery ‘planets’” (2000, October 6). The Memphis Commercial Appeal p. A5. I did an online search through all of the the Commercial Appeal’s articles for that day, and a more global search for newspaper articles with that title, but I was unable to find it. However, the controversy over what constitutes a planet continues: http://en.wikipedia.org/wiki/IAU_definition_of_planet.)

Labels are intertwined with our ontologies (our ways of categorizing the world) and our theories. Combustion used to be explained by an element called “phlogiston” that escaped when something was burned, and that had negative mass. When problems with that theory arose, phlogiston evolved from an element into a principle. The phlogiston theory had considerable explanatory power, and drove a good deal of invention and research. Eventually, though, people recognized enough problems in the phlogiston theory that Joseph Priestly’s discovery—”dephlogisticated air”—came to be called by Lavoisier’s name, the name we still use today: oxygen. Interestingly, oxidation began as a principle, before the element was identified. So theories of combustion went from element to principle to inverted principle and back to element. (See Steven Johnson’s The Invention of Air.)

In the old days, people simply got sick. Some treatments worked; others didn’t work so well. Modern doctors don’t casually confuses bacteria and viruses. They prescribe antibiotics for bacterial infections, and retroviral drugs for viruses. Labels not only represent underlying meanings; they often incorporate those meanings.

If we’re pleading for professional respect for testing, it’s worth asking ourselves what we think is worthy of respect. I believe that what’s most respectable is our special interest in dispelling illusions, demolishing unwarranted confidence, recognizing unnoticed complexity, and revealing underlying truths in a rapidly changing and developing world. If you agree, I think you’ll see a professional problem in promoting ways of speaking that create or sustain illusions, instead of telling plain truths about our work. I think you’ll see a professional problem with oversimplified models of complex cognitive processes, and I think you’ll see a problem with keeping our vocabulary static.

To my correspondent way above: I understand that dealing with all this discussion is effortful. Taking on new ideas is a pain, and so is defending old ones that are being challenged. Revising some simple, seemingly stable beliefs is hard work. Recognizing that two labels might point to the same thing might feel like a distraction, and choosing whose vocabulary to use might be laden with politics and emotion. You’re almost certainly busy with getting stuff done, and it takes time to keep up with the conversation—never mind the racket. But be sure of this: investment and respect are paid to astronomy, chemistry, and medicine as professions precisely because it is the nature of a profession to question and redefine its ideas about itself. Studying a profession involves developing distinctions and definitions to aid in the study. If we’re going to talk seriously and credibly about developing skill in testing, it’s important for us to develop clarity on the activities we’re talking about.

I thank James Bach for his contributions to this essay.

Another day has dawned on Planet Earth, so another tester has used LinkedIn to ask about the difference between severity and priority.

The reason the tester is asking is, probably, that there’s a development project, and there’s probably a bug tracking system, and it probably contains fields for both severity and priority (and probably as numbers). The tester has probably been told to fill in each field as part of his bug report; and the tester probably hasn’t been told specifically what the fields mean—or the tester is probably uncertain about how the numbers map to reality.

“Severity” is the noun associated with the adjective, “severe”. In my Concise Oxford Dictionary, “severe” has six listed meanings. The most relevant one for this context is “serious, critical”. Severity, with respect to a problem, is basically how big a problem is; how much trouble it’s going to cause. If it’s a big problem, it gets marked as high severity (oddly, that’s typically a low number), and if it’s not a big deal, it gets marked as low severity, typically with a higher number. So, severity is a simple concept. Except…

When we’re testing, and we think we see a problem, we don’t see everything about that problem. We see what some people call a failure, a symptom. The symptom we observe may be a manifestation of a coding error, or of a design issue, or of a misunderstood or mis-specified requirement. We see a symptom; we don’t see the cause or the underlying fault, as the IEEE and others might call it.

Whatever we’re observing may be a terrible problem for some user or some customer somewhere—or the customer might not notice or care. Here’s an example: in Microsoft Word 2010′s Insert Page Number feature, choose small Roman numerals as your format, and use the value 32768 (rendered in Roman numerals). Word hangs on my machine, and on every machine I’ve tried this trick on (you can try it too). Now: is this a Severity 1 bug? It certainly appears to be severe, considering the symptom. A hang is a severe problem, in terms of reliability.

But wait… considering that vanishingly few people use lower-case Roman numeral page numbers larger than, say, a few hundred, is the problem really that severe? In terms of capability, it’s probably not a big deal; there’s a very low probability that any normal user would need to use that feature and would encounter the problem.

Except… considering the fact that a problem like this could—at least in theory—present an opportunity for a hacker to bring down an application or, worse, take control of a system, maybe this is a devastatingly severe problem.

There’s yet another factor to consider here. We all suffer to some degree from a bias that can play out in testing. This might be a form of representativeness bias, or of assimilation bias, or of correspondence bias, but none of these seems to be a perfect fit. I think of it as the Heartburn Heuristic, in honour of my dad: for a year or more, he perceived minor heartburn—a seemingly trivial symptom of a seemingly minor gastric reflux problem. What my (late) dad didn’t count on was that, from the symptoms, it’s hard to tell the difference between gastric reflux and esophageal cancer.

The Heartburn Heuristic is a reminder that it’s easy to believe—falsely—that a minor symptom is naturally associated with a minor problem. It’s similarly easy to believe that a serious problem will always be immediately and dramatically obvious. It’s also easy to believe that a problem that looks like big trouble is big trouble, even when a fast one-byte fix will make the problem go away forever. We also become easily confused about the relationship between the prominence of the symptom, the impact on the customer, and the difficulty associated with fixing the problem, and the urgency of the fix relative to the urgency of releasing the product. (Look at the Challenger and Columbia incidents as canonical examples of how this plays out in engineering, emotions, and politics.) In reality, there’s no reason to believe in a strong correlation between the prominence of a problem and its severity, or the potential impact of a problem and the difficulty of a fix. A missing character in some visible field may be a design limitation or a display formatting bug, or it may be a sign of corruption in the database. Of course, since we’re fallible human beings, looking for unknown problems in an infinite space with finite time to do it, the most severe problems in a product can escape our notice entirely. So based on the symptom alone, at best we can only guess at the severity of the problem. That’s bad enough, but the problem of classifying severity gets even worse.

Just as we have biases and cognitive shortcomings, other people on the project team will tend to have them too. The tester’s credibility may be called into question if she places a high severity number on what others consider to be a low severity problem. Severity, after all, is subject to the Relative Rule: severity is not an attribute of the problem, but a relationship between the problem and some person at some time. To the end user who never uses the feature, the Roman numeral hang is not a big deal. To the end user who actually experiences a hang and possible loss of time or data, this could be a deeply annoying problem. To a programmer who takes great pride in his craft, a hang is a severe problem. To a programmer who is being evaluated on the number of Severity 1 problems in the product (a highly dubious way to measure the quality of a programmer’s work, but it happens), there is a strong motivation to make sure that the Roman numeral hang is classified as something other than a Severity 1 problem. To a program manager who has a few months of development time available before release, our Roman numeral problem might be a problem worth fixing. To a program manager who is facing a one-week deadline before the product has to ship (thanks to retail and stock market pressure), this is a trivial bug. (Trust me on that; I’ve been a program manager.)

In light of all this, what is a tester to do? My personal preference (based on experience as a tester, as a programmer, and as a program manager) is to encourage testers to stay out of the severity business if possible. By all means, I provide the project team with a clear description of the symptom, the quality criteria that could be threatened by it, and ideas on how the problem could have an effect on people who matter. I might provide a guess, based on inference, as to the underlying cause. I’ll be careful to frame it as a guess, unless I’ve seen the source code and understand the problem clearly. My default assumption is that I can’t go by appearances, and that every symptom has an unknown cause with potentially harsh consequences. I assume that every problem is guilty until proven innocent—that it’s a potentially severe problem until the code has been examined, the risk models revisited, and the team consulted. I’m especially wary of assigning a low severity on a bug report based on an apparently trivial symptom. If I haven’t seen the code, I try to avoid saying that something is a trivial problem; if pressed, I’ll say it looks like a trivial problem. If I’m forced to enter a number into a bug reporting form, I’ll set the severity of a problem at its highest level unless I have substantial understanding and reason to see the problem as being insignificant. In order to avoid the political cost of seeming like a Cassandra, I’ll make sure my clients are aware of my fundamental uncertainty about severity: the best I can provide is a guess, and if I want to err, I’d rather err or the side of overestimating severity rather than underestimating it and thereby downplaying an important problem. As a solution that feels better to me, I might also request an “unclassified” option in the Severity field, so that I can move on quickly and leave the classification to the team, to the programmers and to the program managers.

As for priority: priority is the order in which someone wants things to be done. Perhaps some people use the priority field to rank the order in which particular problems should be discussed, but my experience is that, usually, “priority” is a tester’s assessment of how important it is to fix the problem—a kind of ranking of what should be fixed first. Again based on my experience as tester, programmer, and program manager, I don’t see this as being a tester’s business at all. Deciding what should be done on a programming or business level is the job of the person with authority and responsibility over the work, in collaboration with the people who are actually doing the work. When I’m a tester, there is one exception: if I see a problem that is preventing me from doing further testing, I will request that the fix for that problem be fast-tracked (and I’ll outline the risks of not being able to test that area of the product). As tester, one of the most important aspects of my report is the set of things that make testing harder or slower, the things that give bugs more time and more opportunity to hide. Nonetheless, deciding what gets fixed first is for those who do the managing and the fixing.

In the end, I believe that decisions about severity and priority are business and management decisions. As testers, our role is to provide useful information to the decision-makers, but I believe we should let development managers manage development.

The categories “manual testing” and “automated testing” (and their even less helpful byproducts, “manual tester” and “automated tester”) were arguably never meaningful, but they’ve definitely outlived their sell-by date. Can we please put them in the compost bin now? Thank you.

Here’s a long and excellent rant by pilot Patrick Smith, who for years has been trying to address a similar problem in the way people talk (and worse, think) about “manual” and “automated” in commercial aviation.

When it comes to software, there is no manual testing. My trusty Chambers dictionary says manual means “of the hand or hands”; “done, worked or used by the hand(s), as opposed to automatic, computer-operated, etc.”; “working with the hands”. It’s true that sometimes we use our hands to tap keys on a keyboard, but thinking of that as “manual testing” is no more helpful than thinking of using your hands on the steering wheel as “manual driving”. We don’t say that Itzhak Perlman, using his hands, is performing “manual music”, even though his hands play a far more important role in his music than our hands play in our testing. If you’re describing a complex, cognitive, value-laden activity, at least please focus on the brain.

We might choose to automate the exercise of some functions in a program, and then automatically compare the output of that program with some value obtained by another program or process. I’d call that a check. (So did McCracken (1957), and so did Alan Turing.) But whatever you call the automated activity, it’s not really the interesting part. It’s cool to know that the machine can perform a check very precisely, or buhzillions of checks really quickly. But the risk analysis, the design of the check, the programming of the check, the choices about what to observe and how to observe it, the critical interpretation of the result and other aspects of the outcome—those are the parts that actually matter, the parts that are actually testing. And those are exactly the parts are not automated. A machine producing a bit is not doing the testing; the machine, by performing checks, is accelerating and extending our capacity to perform some action that happens as part of the testing that we humans do. The machinery is invaluable, but it’s important not to be dazzled by it. Instead, pay attention to the purpose that it helps us to fulfill, and to developing the skills required to use tools wisely and effectively. Otherwise, the outcome will be like giving a 17-year-old fan of Top Gear the keys to a Ferrari: it will not end well.

People often get excited when they develop or encounter something that helps them to extend their capabilities in some way. Marshall McLuhan used “media” to describe tools and technologies—anything that humans use to effect change—as extensions of man, things that enhance, extend, enable, accelerate or intensify what we do. He also emphasized that media are agnostic about our capabilities and our purposes. Tools not only accelerate what we do, they intensify what we are. The tool also changes the nature of the test that we’re performing, sometimes in ways that may escape our notice. Tools can extend or enhance or enable fabulous testing. But by accelerating some action, tools can enable us to do bad testing faster than ever, far worse than we could possibly do it without using the tool.

As Cem Kaner so perfectly puts it, “When we’re testing, we’re not talking about something you’re doing with your hands, we’re talking about something you do with your head.” Besides, as he and others have also pointed out, it’s exceedingly rare that we don’t use machinery and tools in “manual” testing. Even in review—when we’re not operating the product—we’re often looking at requirement documents or source code on a screen; using a keyboard and mouse as input mechanisms for our notes; using the computer to help us find and sift information; using software and hardware to communicate with other people; comparing ideas in our head with the output from a printer. Is this stuff manual? I prefer to think of it as sapient, something that by definition can’t be performed by a machine (or, using a word coined by our friend Pradeep Soundararajan, “brainual”), and assisted by tools. But if you want to call that stuff “manual” testing, how do you characterize the act of creating automated checks? I imagine you would call that writing, not typing. Through macros, computers can effectively type, and save us time in typing; it takes people to write. See the difference?

Several years ago, I used Excel to construct an automated oracle to model functional aspects in a teller workstation application, and identify which general ledger accounts should be credited or debited. Building the oracle took three weeks of solid analysis and programming work. As I was doing that work, I was learning about the system, performing tests, revealing problems, feeding back learning in the the development of my oracle. During this process, I also found lots of bugs. Sometimes my oracle helped me find a bug; sometimes tests that I performed interactively while developing the oracle helped me find a bug. Was I doing “automated” or “manual” testing?

I worked at another financial institution, where part of the task was to create tables of inputs and actions for functional integration checks, using FITnesse as the tool to help me select the functions, organize the data, and execute the checks. I identified risks. I designed checks to probe those risks. I entered those checks manually, via the keyboard. The tool executed the checks, but I triggered its execution manually. I reviewed the output by eye, but the tool helped by allowing me to see unexpected results easily. I revisited our assumptions frequently, considering what the checks appeared to be telling us and what they might be leaving out. When we realized a lapse in our check coverage, I’d enter new checks manually (although sometimes I laid them out in Excel first). So was I doing automated testing, or not?

Why is this a big deal to me? To me, there are two dominating reasons, and they complement each other.

When we think in terms of “automated testing”, we run the risk that we will narrow our notion of testing to automated checking, and focus our testing effort on the development and execution of automated checks. Checking is not a bad thing; it’s a good thing, and an important thing. But there are more aspects to the quality of a product than those that can be made subject to binary evaluation, which is what checks produce. I’ve had the experience of being fascinated by checking to the point of distraction from a broader and deeper analysis of the product I was working on, which meant that I missed some important problems (thankfully, other testers found those problems before they reached the customer).  Qualitative analysis of a product isn’t about bits; it’s about developing a story. Yet investigating qualitative aspects can be aided by tools, which brings us to the second point.

Talk of “manual testing” often slides into talk of “manual exploratory testing”, as though exploration doesn’t make use of tools, or as though tools could not used in an exploratory way. This slipping limits our ideas about exploration and our ideas about tools. Exploratory testing is not tool-free. When we think in terms of “manual testing”, we run the risk of underestimating the role that tools can play in our interactions with the product and in the testing effort. As a consequence, we may also ignore myriad ways in which automation can assist and amplify and accelerate and enable all kinds of testing tasks: collecting data, setting up the product or the test suite, reconfiguring the product, installing the product, monitoring its behaviour, logging, overwhelming the product, fuzzing, parsing reports, visualizing output, generating data, randomizing input, automating repetitive tasks that are not checks, preparing comparable algorithms, comparing versions—and this list isn’t even really scratching the surface. I’ve had experience of not pausing to think about how tools could help me, of neglecting to use them, and of using them unskillfully, and that compromised the quality of my work too.

To sum up:  “Manual testing” vs. “automated testing” is a false dichotomy, like “manual statistical analysis” vs. “automated statistical analysis”, or like “manual book editing” vs. “automated book editing”. In the sense that most people seem to use them, “manual” and “automated” refer to an input mechanism. People probably started saying “manual test” as a contrast to “automated test”, but just as testing isn’t manual, there’s no such thing as an automated test, either—except of the kind that we call a check, which still requires significant testing skill to design, prepare, interpret, and analyze. The check might be automated—performed by a machine—but the testing that surrounds the check is neither manual nor automated. The person who performs the testing is neither an “automated tester” nor a “manual tester”.  Tools can aid testing powerfully, and in all kinds of ways, but they can also lead to distortion or dysfunction, so practice and develop skill to use them effectively.

If you think that all this is a silly quibble, take a page from my colleague James Bach: you don’t talk about compiling and linking as “automated programming”, do you?—nor do you talk about writing code as “manual programming”, do you?