Blog Posts from April, 2011

“Flawed” Analogies

Wednesday, April 13th, 2011

Note: This post contains plagiarism: I’ve stolen some content from an earlier blog post, and from my comments on another. I beg the forgiveness of faithful and diligent readers.

Recently I’ve had to deal with some complaints from people on Twitter who seem to have misinterpreted certain analogies. Worse than that, sometimes it seems as though they don’t understand why and how we use analogies at all. Here are some examples.

I suggested, “Waiters don’t tell the chef when the food is ready to serve; why should testers tell the managers when the product is ready to ship.”

One fellow objected, claiming that waiters only deliver the food, and that to use that analogy diminishes the role of the tester. He implied that testers would be more like food-tasters who would inform the chef about the quality of the dish and possible problems with it. Fair enough—but even then, the chef would have responsibility for the decision to deliver the food.

Another fellow objected, demanding “Do you mean that testers aren’t first-class members of your dev team?” Well, as a matter of fact they are—but now I have some questions about whether this person might not see waiters as first-class citizens of a restaurant. As Milton said, “They also serve who only stand and wait.” (Yes, that’s a joke.)

Someone else objected when I said that “Testers are responsible for quality in the same way that investigative reporters are responsible for Supreme Court decisions.” The response was, “This surprises me. Don’t we try to get testers involved up front, w/lotsa communication & collaboration, to improve quality?” Yes, we do. The point here is simply that, whatever their activities, responsibilities, or contributions, testers are not responsible for making decisions about quality, but rather for informing decisions about quality. Like investigative reporters might inform the evidence presented in a court case, or in issues of social policy. That’s still a completely valuable role in a society, but it’s not the decision-making role that we would associate with those who make the laws or provide enforceable interpretations of them.

Analogy, metaphor and simile are kinds of associative language. Let’s start with “analogy”. The Oxford Dictionary of English (available as a killer app for the iPhone) calls analogy “a comparison between one thing and another, typically for the purpose of explanation or clarification…; a correspondence or partial similarity (my emphasis there)…; a thing that is comparable to something else in significant respects”.

The word comes from the Greek. “Logos” refers to words, thoughts, and reasoning. The prefix “ana-” denotes several possibilities, including “up, upward, throughout, backward, back, again, anew”. Analogy, therefore, suggests thinking something through, or reasoning back to some similarity, or looking at a thought again, or thinking something up. In other words.

For metaphor and simile, I’ll repeat here a quote from an earlier blog post: In his book The Educated Imagination (based on the Massey Lectures, a set of broadcasts he did for the Canadian Broadcasting Corporation in 1963), Northrop Frye said, “Outside literature, the main motive for writing is to describe this world. But literature itself uses language in a way which associates our minds with it. As soon as you use associative language, you begin using figures of speech. If you say, “this talk is dry and dull”, you’re using figures associating it with bread and breadknives. There are main kinds of association, analogy and identity, two things are like each other and two things that are each other (my emphasis –MB). One produces a figure of speech called the simile. The other produces a figure called metaphor.”

If it helps, think of an instance of associative language as a kind of model. Models are heuristic representations—literally, re-presentations—of something complex in terms of something simpler; of something unfamiliar in terms something more familiar; of something more abstract in terms of something more concrete. “Heuristic” is a key adjective here, suggesting that models help us to learn, and are fallible. As Nassim Nicholas Taleb says in The Black Swan, “Models and constructions, these intellectual maps of reality, are not wrong; they’re wrong only in some specific applications.” Absolutely true—and you can flip it around to say that models are right only in some specific applications too. It is their very imperfection—the fact that they do not represent the thing being modeled in every dimension—that makes models useful and powerful. The relative simplicity of the model is an aid to our understanding of a more complex subject. The purpose of models and associative speech is not to create an exact parallel, but to get people thinking in terms of similarities and differences. Were there a one-to-one correspondence between every aspect of a model and the thing being modeled, there would be no simplification and no point in using the model. It would be like Steven Wright’s full scale map of the United States: “It took me all last summer to fold it.”

Metaphor and simile trigger the capacity of the human mind to forge new links between ideas. Frye again: “The poet, however, uses these two crude, primitive, archaic forms of thought in the most uninhibited way, because his job is not to describe nature but to show you a world completely absorbed and possessed by the human mind… The motive for metaphor, according to Wallace Stevens, is a desire to associate, and finally to identify, the human mind with what goes on outside it, because the only genuine joy you can have is in those rare moments when you feel that although we may know in part, as Paul says, we are also a part of what we know.”

If I were to tell you that my dog were like my car in that they both require a lot of maintenance, would you expect my dog to have a gas tank? Would you expect me to take the dog to my mechanic when it appeared ill? If I said that picking up the guitar again after a couple of years was “riding a bike” for me, would you take me literally and envision a musical instrument with wheels and handlebars? If I were to suggest that my former boss was a pain in the ass, would you expect everyone at work to have trouble sitting down? If I said that a friend is as honest as the day is long, would you think that I measured honesty in hours? If I said that your newer testers were green, would you contradict me because they didn’t match a certain Pantone colour?

Analogies are always flawed by design. They’re intended to make people think about the ways in which something is like something else, while intentionally ignoring the ways in which they’re different. That is, they’re not supposed to be perfect. They’re supposed to draw attention to some factor of something, so that people can consider that factor, and possibly others, in a new or different light. Notice that I can use that expression, “in a new or different light”, and you don’t get confused or think “Hmmm…incandescent or fluourescent?”

Complaining that an analogy is flawed is like complaining that a model airplane (say, one designed for a wind tunnel) is flawed because it’s not big enough to carry real passengers. Confusing an analogy with a fact in this way is a symptom of model blindness, evidence of a failure to distinguish between models and reality. That can bite people in two ways: first, by limiting our imagination to the literal, rendering us capable of less powerful reason, and even making us something less than human (which is what Frye warns against); and second, by allowing us to place too much trust in our models, making us vulnerable to risk when then model doesn’t apply (which is what Taleb warns against).

Some people say that they don’t like analogy, or that they don’t use analogy. To me, that’s a symptom that they’re not paying attention to the way they speak and the way they think. If you’re such a person, I have a suggestion: try paying attention to the number of times per day you describe or envision something in terms of something else. Try monitoring how often you try to represent an idea with a sketch. Note each time you say, “That’s like…” Get a friend to observe each time you use a metaphor. You’ll shortly observe that we’re all using analogy, all the time. Then try quitting for a day, like fasting or giving up cigarettes for Lent. (Did you notice? Those were similes. Did you understand them and absorb them, or did you say, “I don’t smoke”?) Not only will your speech or writing be limited, but I would argue that your capacity to reason will be limited as well.

If you’re in software development, the program that you’re writing or testing is a way of modelling some task that the user wishes to perform. The ways we have of describing those programs are universally framed in terms of models, analogies, metaphor, and simile. We humans are analogy machines (Did you notice? That’s a metaphor.) So if you don’t like “flawed” analogies, it would nonetheless be a good idea to get used to them.

Postscript, 2013/12/10: “A study published in January in PLOS ONE examined how reading different metaphors—’crime is a virus’ and ‘crime is a beast’—affected participants’ reasoning when choosing solutions to a city’s crime problem…. (Researcher Paul) Thibodeau recommends giving more thought to the metaphors you use and hear, especially when the stakes are high. ‘Ask in what ways does this metaphor seem apt and in what ways does this metaphor mislead,’ he says. Our decisions may become sounder as a result.” Excerpted from Salon.

Another postscript, same day: have a look at Surfaces and Essences: Analogy as the Fuel and Fire of Thinking, by Douglas Hofstadter and Emmanuel Sander for a look at how pervasive and fundamental associative speech is.

Posted in Models, Words and Semantics | 9 Comments »

A Few Observations on Structure in Testing

Tuesday, April 12th, 2011

On Twitter, Johan Jonasson reported today that he was about to attend a presentation called “Structured Testing vs Exploratory Testing”. This led to a few observations and comments that I’d like to collect here.

Over the years, it’s been common for people in our community to mention exploratory testing, only to have someone reply, “Oh, so that’s like unstructured testing, right?” That’s a little like someone refer to a cadenza or a musical solo as “unstructured music”; to improv theatre as “unstructured theatre”; to hiking without a map or a precise schedule as “unstructured walking”; to all forms of education outside of a school as “unstructured learning”. When someone says “Exploratory testing is unstructured,” I immediately hear, “I have a very limited view of what ‘structure’ means.”

Typically, by “structured testing”, such people appear to mean “scripted testing”. To me, a script is a set of detailed, ordered set of steps of specific actions and specific observations. Some people call that a procedure, but I think a procedure is characterized by a set of activities that might be but are not necessarily highly specific, and that might be but are not necessarily strictly ordered. So what is structure?

Structure, to me, is any aspect of something (like an object or system) that is required to maintain the thing as it is. You can change some element of a system, add something to it or remove something from it. If the system explodes, or collapses, or changes such that we would consider it a different system, the element or the change was structural. If the system retains its identity, the element or the change wasn’t structural. Someone once described structure to me as “that which remains”.

A system, in James Bach’s useful definition, is a set of things in meaningful relationship. As such, structure is the specific set of things and factors and relationships that make the system a system. Note that the observer is central here: you may see a system where I see only a mess. That is, you see structure, but I don’t. When a system changes, you may call it a different system, where I might see the original system, but modified. In this case, I’m seeing structure that you don’t see or acknowledge as such. We see these things differently because systems, structures, elements, changes, meaningfulness, and relationships are all subject to the Relative Rule: “For any abstract X, X is X to some person.”

One of the principles of general systems thinking is that the extent to which you are able to observe and control a system is limited by the Law of Requisite Variety, coined by W. Ross Ashby: “only variety can destroy variety.” Jurgen Appelo recently produced a marvelous blog post on the subject, which you can read here.

Variety, says Ashby, is the number of distinct states of a system. In order to observe variety, “The observer and his powers of discrimination may have to be specified if the variety is to be well defined.” The observer is limited by the number of states that he (or she or it) can see. This also implies that however many states there are to a system, another system that controls it has to be able to observe that many states and have at least one state in which it can exert control. Wow, that’s pretty heady. What does it mean? The practical upshot is that if you intend to control something, you have to be able to observe it and to change something about it, and that means that you need to have more states available to you than the object has. Or, as Karl Weick put it in Sensemaking in Organizations, “if you want to understand something complicated, you have to complicate yourself.”

This pattern repeats in lots of places. Glenford Myers suggested that testing a program requires more creativity than writing one; more things can go wrong than go right in writing a program. Machines can do a lot of the work of keeping an aircraft aloft, but managing those machines and the work that they do requires more intelligent and more adaptable systems—people. The Law of Requisite Variety also suggests that we can never understand or control ourselves completely, people are insufficient to understand people completely.

So: some people might see exploratory testing as unstructured. I suspect they’re thinking of a particular kind of structure: that scripted procedure that I mentioned above. For me, that suggests an impoverished view of both exploratory testing and of structure. Exploratory testing has tons of structures.

Over the years, many colleagues have been collecting and cataloguing the structures of exploratory testing. James and Jon Bach have led the way, but there have been lots of other contributions from people like Cem Kaner, Mike Kelly, Elisabeth Hendrickson, Michael Hunter, Karen Johnston, Jonathan Kohl, and me (if there’s someone else who should appear in this list, remind me). I’ve collected some of these in this list of structures of exploratory testing. For those who are just joining us, and for the old hands who might need a refresher, you can find my list of exploratory testing structures here.

Programs had structures long before “structured programming” came along. Programming that isn’t structured programing is not “unstructured programming.” The structure in “structured programming” is a particular kind of structure. To many people, there is only one kind of testing structure: scripted procedures. I will guarantee that an incapacity to see alternative kinds of structure will limit your testing. And to paraphrase Weick, if you want to understand testing, you must test yourself.

Posted in Exploratory Testing, Systems Thinking | 6 Comments »

Questioning Test Cases, Part 2: Testers Learn, But Test Cases Don’t Teach

Wednesday, April 6th, 2011

In the last post, my LinkedIn correspondent provided a couple of reasons why she liked writing test cases, and why she thought it necessary to write them down in the early stages of the project. Then she gave a third reason:

When I’m on a project and I am the only one who knows how to test something, then I can’t move on to something new. I’d still be testing Cut/Copy/Paste if I were not able to pass that script on easily to a new recruit.

I often hear this kind of justification for test cases: “We need test cases so that testers can learn quickly how to test the product.” To me, that’s like saying that we need driving cases so that people can learn quickly how to drive in a new city. In a new city, a map or a general set of ideas might be helpful. Driving cases won’t, because to drive through a new city well, you have to know how to drive. In the same way, test cases are unlikely to be anything more than negligbly helpful. In order to learn quickly how to test a new program well, you have to know how to test.

It also seemed to me that she considers writing to be the One True Way to impart information, and reading to be the One True Way to receive it. Now, if I’m wrong about that, it would have to be because I’ve misinterpreted something she’s written. Fancy that!

So for the sake of discussion, let’s say I were to to hire (say) an older domain expert, making a career change, who had no computer experience, and let’s use Cut/Copy/Paste as an example. With the new recruit, I’d do what I’ve done many times in coaching sessions. I’d sit with her down with a copy of Microsoft Word, and open up a document that had some text and some images in it. Then I’d say something like this:

“We’re going to learn how to move text around, cutting and pasting and making copies of it so we can save time on retyping stuff that the machine can do for us. Okay. Put your cursor there, and press Shift-RightArrow there. Good. See what happened?”

(There would be lots of pauses. I’d pause after each question to give the trainee time to answer; frequently to let the trainee experiment; and often to let the trainee ask questions of her own. I won’t do that explicitly here; I’ll let you put most of the pauses in yourself, as you read. For sure, assume that there’s one at least after every question, and usually after every sentence.)

“So try doing that a few more times. What happened? So you’ve highlighted the text; people call that marking or selecting the text. What happens when you add the Ctrl key, and press Ctrl-Alt-RightArrow at the same time? Right: it selects a word at a time. Now, try pressing Ctrl-X.

“Whoa, what happened just there?! Right: pressing Ctrl-X that makes the selected text go away; you can see that. What you might not know is that the text goes onto the Windows Clipboard, a kind of temporary storage area for all kinds of stuff. So you haven’t erased it; it’s more like you’ve cut it out, and the computer still has a copy of it stashed somewhere.

“Now, put your cursor here, and hit Ctrl-V. What did that do? Yeah; it pasted the text you cut before. That is, it retrieved the text from the clipboard, and made a copy right where you pressed Ctrl-V. So, Ctrl-X cuts, and Ctrl-V pastes a copy of the thing you cut.

“Try that again. What happened? Right; another copy. So it’s a little different from regular cutting and pasting, in that you can make many copies of the same thing. When might that come in handy? Good idea! Yep, that’s a good idea too. Try it a few more times, in other places.

“What does Ctrl-V do? What does Ctrl-X do?

“Try selecting something else, and pressing Ctrl-X again. What happens when you paste now? Can you get the old one back? Well, there might be special add-ons that allow you to keep copies of lots of things you pasted, but normally, you can only paste the last thing you put on the Clipboard.

“How are you going to remember that Ctrl-X means “cut”? That’s a great idea! Myself, I think of the X as a pair of open scissors, ready to cut—but your idea is great, and better than that, it’s your own, so you’ll remember it.

“How are you going to remember Ctrl-V as paste? Yeah, it’s tough to come up with a way to remember that, isn’t it? I think of it like the tip of one of those old glue bottles from grade school. On the other hand, what’s in between X and V on the keyboard? Now, go select some text again. Remember how? Good.

“This time, try pressing Ctrl-C, instead of Ctrl-X. Right; nothing seems to happen. And yet… move your cursor here, and try pasting. Yeah; it’s a copy, even though the original stays where it was. So Ctrl-C is Copy–C for copy, which makes some kind of sense, for a change—right there in between cut and paste. And there’s another way to remember; try pressing Alt-E. E stands for Edit, that’s right. What do you see listed on the menu, there?

“Now… there’s some documentation for these cut-and-paste features; it’s in the Windows documentation. Press F1. Great. Now look for ‘Keyboard Shortcuts’.

“There’s a list. How many different ways can you find of marking, cutting, and pasting text? Make me a list, and I’ll be back in ten minutes. Be ready to show me your report, and to demonstrate the ones you’ve discovered. Oh—and one more thing: create a list of at least five circumstances in which you might use this stuff, and we’ll talk about that too.”

It’s taken quite a while to type this example. It would take considerably less time to do it. Moreover, the trainee would learn more quickly and more deeply, because she’d be interacting with the product herself, experiencing the actions in her body, literally incorporating them. The questions, the answers, and the interactions make the learning more sticky. The practice makes it more sticky still.

I try to encourage people to create mnemonics to help remember. While conversing with the trainee, I’d also be observing her. She’ll be making plenty of mistakes; people do that when they’re learning. In fact, people should be encouraged to do that when they’re learning. (Brian Marick’s book Everyday Scripting in Ruby encourages deliberate mistakes, which is one of the reasons I like it so much.) When I see mistakes, I’ll give her a chance to feel the mistake, puzzle out a way around it, and then—if she needs it—help her get out of it. (“Let’s figure out how to fix that. Hit Alt-E again. What’s listed on the Edit menu? So what’s the keyboard shortcut for Undo?” “Oops, you pressed Ctrl-Z one time too many… What does Ctrl-Y do?”).

After a while, I won’t tell her how to do everything right away. I’ll start asking her to figure out some things on her own, and only help out when she’s stuck. That way she’ll be learning it for herself, which is more challenging but which has more effect. I’ll keep trying to raise the bar, giving her more challenging assignments, increasing her independence while also increasing her responsibility to report. When I see her make a mistake, I might correct her right away, or I might let her continue with the mistake for a bit until she encountered some kind of minor consequence. That makes the learning even more sticky.

Finally, I’d cut the conversation short if she told me that she already knew how to cut and paste—but I’d get her to show me that she knew it, and I’d give her some more exercises so that I could observe her.

The trouble that I see with “passing on a script to a new recruit” is that most test scripts I’ve seen are very long on what to do or how to do it, and correspondingly short on why to do it and how to generalize it. That is, they’re short on motivation, and they’re short on risk, and they rarely include a mission to learn. In our teaching, when we’re working with test groups, and when we’re coaching individual testers, James and I put less emphasis on techniques. Instead, we focus on making sure that people have the skills to know what to do when they receive a testing mission, whether that mission is written down in detail or not.

If there are specific things that I want a tester to look for, I’ll provide specific instructions (either written or spoken), but mostly I want to set out on their own to discover things. I want to encourage testers to vary their work and their observations. Variation makes it more likely that they will find problems that would be out of sight if we were to focus on the scripts. As a bonus, giving testers a concise, open-ended task keeps writing to a minimum, which saves on development cost, maintenance cost, and opportunity cost.

We’ll finish up this series in the next post.

Posted in Accountability, Skills, Test Framing | 10 Comments »

Questioning Test Cases, Part 1

Monday, April 4th, 2011

Over the years, LinkedIn seems to have replaced comp.software.testing as the prime repository for wooly thinking and poorly conceived questions about testing.

Recently I was involved in a conversation with someone who, at least, seemed to be more articulate than most of the people on LinkedIn. Alas, I’ve since lost the thread, and after some searching I’ve been unable to find it. No matter: the points of the discussion are, to me, are still worth addressing. She was asking a question about how much time to allocate to writing test cases before starting testing, and I questioned the usefulness of doing that. The first part of her reply went like this:

I would like to point out that I doubt anyone wants to write something that they don’t need to write.

I agree that most people probably don’t want to write things they don’t need to write. But they often feel compelled, or are compelled to write things they don’t need to write.

I find value in writing test cases for a number of reason. One is that I train more junior engineers in testing and it is a good method to have them execute tests that I have written so they learn how a good test plan is put together.

If that were so, wouldn’t your junior engineers learn even more from writing test cases themselves, and getting feedback on their design and their writing? There’s a feedback loop in the design of a test, the execution of a test, the interpretation of a test result, and the learning that happens between them; wouldn’t it be a good idea to keep the feedback loop—and the learning—as rapid as possible? Wouldn’t your junior engineers learn still more from actually testing—under your close supervision, at first, and then with the freedom and responsibility to act more independently as they gain skill? You might want to have a look at this article: http://www.developsense.com/articles/2008-06-KnowWhereYourWheelsAre.pdf.

There’s a common misconception that testing happens in the characters of a written test case. It doesn’t. Testing happens in the mind and actions of the tester. It happens in the design of the test, in the execution of the test, in the observation and interpretation of the outcome of the test. Testing happens in the discovery of problems, in the investigation of those problems, and the learning about those problems and the product. At most, a fraction of this can be written down.

A test is far less something you execute, and far more a line of inquiry that you follow. To me, a good test case is idea-stuff; it’s a question that we want to ask of the program, based on some motivating idea about discovery or risk. In my observation, in writing test cases, people generally write down the least important stuff. They appear to be trying to program the tester’s actions, rather trying than to prime the tester’s thinking and observation.

Moreover, a test plan something quite different from a pile of test cases.

Secondly, [writing test cases] communicates the testing coverage with everyone involved in developing the software. If you are a contractor, this is very important since you want to leave with the client feeling like you did your job and they have the documentation to prove that they have done due diligence if they shop their company around or look for for VC money.

Were you, as a contractor, given the mission to produce test scripts specifically, or is that a mission that you have inferred? Bear in mind, I’ve been witness to many takeovers, as a program manager at a company where our senior managers were ambitiously acquiring technologies, products and companies, and as a consultant to several companies that were taken over by larger companies. In no case did anyone ever ask to see any test case documentation. Those who are investigating the company to be acquired typically don’t go to that level of detail. In my experience, alas, due diligence largely doesn’t happen at all. I’m puzzled, too, by the appeal to the least likely instances in which people might interact with test documention, rather than the everyday.

Meanwhile, there are many ways to communicate test coverage. (For example, see here, here, and here.) There are also many ways to fool yourself (and others) into believing that the more documentation the more coverage, or the more specific the documentation the more coverage—especially when that documentation is prospective, rather than retrospective. In Rapid Testing, we don’t encourage people to eliminate documentation. We encourage people to reduce documentation to what is actually necessary for the mission, and to eliminate wasteful documentation.

We focus on documenting test ideas concisely; on producing coverage outlines and risk lists that can be used to guide, rather than control a tester and her line of inquiry; on producing records of the tester’s thought process, observations, risk ideas, motivations, and so forth (see below). The goal is to capture things far more important than the tester’s mechanical actions. If someone wants a record of that, we recommend video capture software (tools like BB Test Assistant or Camtasia). An automatic log allows the tester to focus on testing the product and recording the ideas, rather than splitting focus between operating the product and writing about operating the product.

You can find examples of the kind of test documentation I’m talking about here, in the appendices to the Rapid Software Testing class, starting at page 47. Note that each example has varying degrees of polish and formal structure. Sometime it’s highly informal, used to aid memory, to frame a conversation, or trigger ideas. Sometimes it’s more formal and polished, when the audience is outside the test group. The over-riding point is to fit the documentation to the mission and to the task at hand.

More to come…

Posted in Accountability, Skills, Test Framing | 7 Comments »

More of What Testers Find, Part II

Friday, April 1st, 2011

As a followup to “More of What Testers Find“, here are some more ideas inspired by James Bach’s blog post, What Testers Find. Today we’ll talk about risk. James noted that…

Testers also find risks. We notice situations that seem likely to produce bugs. We notice behaviors of the product that look likely to go wrong in important ways, even if we haven’t yet seen that happen. Example: A web form is using a deprecated HTML tag, which works fine in current browsers, but may stop working in future browsers. This suggests that we ought to do a validation scan. Maybe there are more things like that on the site.

A long time ago, James developed The Four-Part Risk Story, which we teach in the Rapid Software Testing class that we co-author. The Four-Part Risk Story is a general pattern for describing and considering risk. It goes like this:

  1. Some victim
  2. will suffer loss or harm
  3. due to a vulnerability in the product
  4. triggered by some threat.

A legitimate risk requires all four elements. A problem is only a problem with respect to some person, so if a person isn’t affected, there’s no problem. Even if there’s a flaw in a product, there’s no problem unless some person becomes a victim, suffering loss or harm. If there’s no trigger to make a particular vulnerability manifest, there’s no problem. If there’s no flaw to be triggered, a trigger is irrelevant. Testers find risk stories, and the victims, harm, vulnerabilities, and threats around which they are built.

In this analysis, though, a meta-risk lurks: failure of imagination, something at which humans appear to be expert. People often have a hard time imagining potentional threats, and discount the possibility or severity of threats they have imagined. People fail to notice vulnerabilities in a product, or having noticed them, fail to recognize their potential to become problems for other people. People often have trouble making the connection between inanimate objects (like nuclear reactor vessels), the commons (like the atmosphere or sea water), or intangible things (like trust) on the one hand, and people who are affected by damage to those things on the other. Excellent testers recognize that a ten-cent problem multiplied by a hundred thousand instances is a ten-thousand dollar problem (see Chapter 10 of Jerry Weinberg’s Quality Software Management, Volume 2: First Order Measurement). Testers find connections and extrapolations for risks.

In order to do all that, we have to construct and narrate and edit and justify coherent risk stories. To to that well, we must (as Jerry Weinberg put it in Computer Programming Fundamentals in 1961) develop a suspicious nature and a lively imagination. We must ask the basic questions about our products and how they will be used: who? what? when? where? why? how? and how much? We must anticipate and forestall future Five Whys by asking Five What Ifs. Testers find questions to ask about risks.

When James introduced me to his risk model, I realized that there people held at least three different but intersecting notions of risk.

  1. A Bad Thing might happen. A programmer might make a coding error. A programming team might design a data structure poorly. A business analyst might mischaracterize some required feature. A tester might fail to investigate some part of the product. These are, essentially, technical risks.
  2. A Bad Thing might have consequences. The coding error could result in miscalculation that misrepresents the amount of money that a business should collect. The poorly designed data structure might lead to someone without authorization getting access to privileged information. The mischaracterized feature might lead to weeks of wasted work until the misunderstanding is detected. The failure to investigate might lead to an important problem being released into production. These are, in essence, business risks that follow from technical risks.
  3. A risk might not be a Bad Thing, but an Uncertain Thing on which the business is willing to take a chance. Businesses are always evaluating and acting on this kind of risk. Businesses never know for sure whether the Good Things about the product are sufficiently compelling for the business to produce it or for people to buy it. Correspondingly, the business might consider Bad Things (or the absence of Good Things) and dismiss them as Not Bad Enough to prevent shipment of the product.

So: Testers find not only risks, but links between technical risk and business risk. Establishing and articulating those links are depend on the related skills of test framing and bug advocacy. Test framing is the set of logical connections that structure and inform a test. Bug advocacy is the skill of determining the meaning and significance of a bug, and reporting the bug in terms of potential risks and consequences that other people might have overlooked. Bug advocacy doesn’t mean jumping up and down and screaming until every bug—or even one particular bug—is fixed. It means providing context for your bug report, helping managers to understand and decide why they might to choose to fix a problem, right now, later, or never.

In my travels around the world and around the Web, I observe that some people in our craft have some fuzzy notions about risk. There are at least three serious problems that I see with that.

Tests are focused on (documented) requirements. That is, test strategies are centred around making sure that requirements are checked, or (in Agile contexts) that acceptance tests derived from user stories pass. The result is that tests are focused on showing that a product can meet some requirement, typically in a controlled circumstance in which certain stated conditions assumed necessary have been met. That’s not a bad thing on its own. Risk, however, lives in places where where necessary conditions haven’t been stated, where stated conditions haven’t been met, or where assumptions have been buried, unfulfilled, or inaccurate. Testing is not only about demonstrating that some instance of a requirement has been satisfied. It’s also about identifying things that threaten the successful fulfillment of that requirement. Testers find alternative ideas about risk.

Tests don’t get framed in terms of important risks. Many organizations and many testers focus on functional correctness. That can often lead to noisy testing—lots of problems reported, where those problems might not be the most important problems. Testers find ways to help prioritize risks.

Important risks aren’t addressed by tests. A focus on stated requirements and functional correctness can leave parafunctional aspects of the product in (at best) peripheral vision. To address that problem, instead of starting with the requirements, start with an idea of a Bad Thing happening. Think of a quality criterion (see this post) and test for its presence or its absences, or for problems that might threaten it. Want to go farther? My colleague Fiona Charles likes to mention “story on the front page of the Wall Street Journal” or “question raised in Parliament” as triggers for risk stories. Testers find ways of developing risk ideas.

James’ post will doubtless trigger more ideas about what testers find. Stay tuned!

P.S. I’ll be at the London Testing Gathering, Wednesday, April 6, 2011 starting at around 6:00pm. It’s at The Shooting Star pub (near Liverpool St. Station), 129 Middlesex St., London, UK. All welcome!

Posted in Risk, Systems Thinking, Test Framing, Testing Story | 5 Comments »