Archive for the ‘Test Framing’ Category

« Older Entries

Confusion as an Oracle

Monday, October 17th, 2011

A couple of weeks back, Sylvia Killinen (@skillinen on Twitter) tweeted:

“Seems to me that much of #testing relies on noticing when one is confused rather than accepting it as Something Computer Programs Do.”

That’s a beautiful observation, near and dear to my heart since 2007 at least. The night I read Sylvia’s tweet, I wanted to blog more on the subject, but sometimes blog posts go in a different direction from where I intend them to go. At the time, I went here. And now I’m back.

Sylvia’s tweet reminded me of a story that Jon Bach tells about learning testing with his brother James. Jon had been working in a number of less-than-prestigious jobs. James suggested that Jon become a tester, and offered to train him how to be an excellent tester. Jon agreed to the idea. Everything went fine for the first couple of weeks, but one day Jon slumped into James’ office looking dejected and demoralized. The conversation went something like this.

“What’s the problem?” asked James.

“I dunno,” said Jon. “I don’t think this whole becoming-a-tester thing is going to work out.”

“Not work out? But you’re doing great!” said James.

“Well, it might look that way to you, but…” Jon paused.

“So what’s the problem?”

“Well, you gave me this program to test,” Jon began. “But I’m just so confused.”

James peered over his glasses. “When you’re confused,” he said, “that’s a sign that there’s something confusing going on. I gave you a confusing product to test. Confusion might not be fun, but it’s a natural consequence when you’re dealing with a confusing product.” James was tacitly suggesting that Jon’s confusion cound be used as an oracle—a heuristic principle or mechanism by which we recognize a problem.

This little story suggests and emphasizes a number of serious and important points.

As I mentioned before, here, feelings don’t tell us what they’re about. Confusion doesn’t come with an arrow that points directly back to its source. Jon felt confused, and thought that the confusion was about him. But that confusion wasn’t just about Jon’s internal state; it was also about the state of the product and how Jon felt about it. Feelings—internal, non-specific and ambiguous—don’t tell us what’s going on; they tell us to pay attention to what’s happening around us. When you’re a novice, you might be inclined to believe that your feelings are telling about yourself, but that’s likely not the whole story, since emotions don’t happen in isolation from everything else. It’s more probable that feelings are telling you about the relationship between you and something else, or someone else, or the situation.

Which reminds me of another story. It happened at Jerry Weinberg’s Problem Solving Leadership workshop in 2008. PSL is full of challenging and rich and tricky exercises, and one day, one team had fallen into a couple of traps and had done rather badly. During the debrief, Jerry remarked on it. “You guys handled a much harder problem than this yesterday, you know. What happened this time?”

One of the participants answered, “The problem screwed us up.”

With only the briefest pause, Jerry peered at the fellow and replied in a gently admonishing way, “Your reaction to the complexity of the problem screwed you up.”

Methodologists and process enthusiasts regularly ignore the complex human and emotional aspects of testing, and so don’t take them into account or use them as a resource. Some actively reject feelings as a rich source of information. One colleague reports that she told her boss about a presentation of mine in which I had discussed the role of emotions in software testing.

“There’s no role for emotions in software testing,” he said quietly.

“I’m not sure I agree,” she said. “I think there might be. I think at least it’s worth considering.”

Abruptly he shouted, “THERE’S NO ROLE FOR EMOTIONS IN SOFTWARE TESTING!”

She remarked he had seemed agitated—a strange reaction considered the mismatch between what he was saying and what he appeared to be feeling. What more might we learn by noting his feelings and considering possible interpretations? What inferences might we draw about the differences between his reaction and hers?

As we increasingly emphasize in the Rapid Software Testing course, recognizing and dealing with your feelings is a key self-management skill. Indeed, for testers, feelings are a kind of first-order measurement. It’s okay to be confused. The confusion is valuable and even desirable if it leads you to the right control action, which is to investigate what your emotions might be telling you and why. If we’re willing to acknowledge our feelings, we can use them productively as cues to start looking for oracles and problems in the product that trigger the feelings—before those problems lead our customers to distress.

In my article Testing Without a Map, I discuss some of the oracles that we present in the Rapid Software Testing class and methodology.

Thanks to Sylvia for the inspiration.

I’ll be bringing Rapid Testing to the Netherlands (October 31-November 2), London (November 28-30), and Oslo (December 14-16). See the right-hand panel for registration details. Join us! Spread the word! Thank you!

Posted in Emotions, Heuristics, Learning, Oracles, Test Framing | 9 Comments »

Testing: Difficult or Time-Consuming?

Thursday, September 29th, 2011

In my recent blog post, Testing Problems Are Test Results, I noted a question that we might ask about people’s perceptions of testing itself:

Does someone perceive testing to be difficult or time-consuming? Who? What’s the basis for that perception? What assumptions underlie it?

The answer to that question may provide important clues to the way people think about testing, which in turn influences the cost and value of testing.

As an example, an pseudonymous person (“PM Hut”) who is evidently associated with project management in some sense (s/he provides the URL http://www.pmhut.com) answered my questions above.

Just to answer your question “Does someone perceive testing to be difficult or time-consuming?” Yes, everyone, I can’t think of a single team member I have managed who doesn’t think that testing is time consuming, and they’d rather do something else.

This, alas, isn’t an unusual response. To someone like me who offers help in increasing the value and reducing the cost of testing, it triggers some questions that might prompt reframes or further questions.

  • What do the team members think testing is? Do they think that it’s something ancillary to the project, rather than an essential and integrated aspect of software development? To me, testing is about gathering information and raising awareness that’s essential for identifying product risks and steering the project. That’s incredibly important and valuable.

    So when the team members are driving a car, do they perceive looking out the windshield to be difficult or time-consuming? Do they perceive looking at the dashboard to be difficult or time-consuming? If so, why? What are the differences between the way they obtain awareness when they’re driving a car, versus the way they obtain awareness when they’re contributing to the development of a product or service?

  • Do the team members think testing is the mindless repetition of actions and observation of specific outputs, as prescribed by someone else? If so, I’d agree with them that testing is an unpalatable activity—except I don’t call that testing. I call it checking, and I’d rather let a machine do it. I’d also ask if checking is being done automatically by the programmers at lower levels where it tends to be fast, cheap, easy, useful and timely—or manually at higher levels, where it tends to be slower, more expensive, more difficult, less useful, and less timely—and tedious?

  • Is testing focused mostly on confirmation of things that we already know or hope to be true? Is it mostly focused on the functional aspects of the program (which are amenable to checking)? People tend to find this dull and tedious, and rightly so. Or is testing an active search for new information, problems, and risks? Does it include focus on parafunctional aspects of the product—the things that provide important perceptions of real value to real people? Are the testers given the freedom and responsibility to manage a good deal of their own investigation? Testers tend to find this kind of approach a lot more engaging and a lot more interesting, and the results are typically more wide-ranging, informative, and valuable to programmers and managers.

  • Is testing overburdened by meaningless and valueless paperwork, bureaucracy, and administrivia? How did that come to pass? Are team members aware that there are simple, lightweight, rapid, and highly effective ways of planning, recording, and reporting testing work and project status?

  • Are there political issues? Are testers (or people acting temporarily in a testing role) routinely blown off (as in this example)? Are the nuggets of information revealed by testing habitually dismissed? Is that because testing is revealing trivial information? If so, is there a problem with specific testing skills like modeling the test space, determining coverage, determining oracles, recording, or reporting?

  • Have people been trained on the basis of testing as a skilled, sophisticated thinking art? Or is testing something for which capability can be assessed by a trivial, 40-question multiple choice exam?

  • If testing is being done well (which given people’s attitudes expressed above would be a surprise), are programmers or managers afraid of having to deal with the information that testing reveals? Does that lead to recrimination and conflict?

  • If there’s a perception that testing is by its nature dull and slow, are the testers aware of the quick testing approaches in our Rapid Software Testing class (PDF, page 97-99) , in the Black Box Software Testing course offered by the Association for Software Testing, or in James Whittaker’s How to Break Software? Has anyone read and absorbed Lessons Learned in Software Testing?

  • If there’s a perception that technical reviews are slow, have the testers, programmers, or managers read Perfect Software and Other Illusions About Testing? Do they recognize the ways in which careful observation provides us with “instant reviews” (see Perfect Software, page 143)? Has anyone on the team read any other of Jerry Weinberg’s books on software management and measurement?

  • Have the testers, programmers, and managers recognized the extent to which exploratory testing is going on all the time? Do they recognize that issues revealed by testing might be even more important than bugs? Do they understand that every test result and every testing problem points to meta-information that can be extremely valuable in managing the project?

On PM Hut’s own Web site, there’s an article entitled “Why Project Managers Fail“. The author, Jim Benson, lists five common problems, each of which could be quickly revealed by looking at testing as a source of information, rather than by simply going through the motions. Take it from the former program manager of a product that, in its day, was the best-selling piece of commercial software in the world: testers, testing, and the information they reveal are a project manager’s best friends and most valuable assets—when you have the awareness to recognize them.

Testing need not be difficult, tedious or time-consuming. A perception that it is so, or that it must be so, suggests a problem with testing as practised or testing as perceived. Astute managers and teams will investigate that important and largely mistaken perception.

Posted in Accountability, Context, Heuristics, Learning, Management, Models, Risk, Systems Thinking, Test Framing, Tester Skill, Testing Story, Time, Words and Semantics | 5 Comments »

Testing Problems Are Test Results

Tuesday, September 6th, 2011

I often do an exercise in the Rapid Software Testing class in which I ask people to catalog things that, for them, make testing harder or slower. Their lists fit a pattern I hear over and over from testers (you can see an example of the pattern in this recent question on Stack Exchange). Typical points include:

  • I’m a tester working alone with several programmers (or one of a handful of testers working with many programmers).
  • I’m under enormous time pressure. Builds are coming in continuously, and we’re organized on one- or two-week development cycles.
  • The product(s) I’m testing is (are) very complex.
  • There are many interdependencies between modules within the product, or between products.
  • I’m seeing a consistent pattern of failures specifically related to those interdependencies; the tiniest change here can have devastating impact there—or anywhere.
  • I believe that I have to run a complete regression test on every build to try to detect those failures.
  • I’m trying to cope by using automated checks, but the complexity makes the automation difficult, the program’s testing hooks are minimal at best, and frequent product changes make the whole relationship brittle.
  • The maintenance effort for the test automation is significant, at a cost to other testing I’d like to do.
  • I’m feeling overwhelmed by all this, but I’m trying to cope.

On top of that,

  • The organization in which I’m working calls itself Agile.
  • Other than the two-week iterations, we’re actually using at most two other practices associated with Agile development, (typically) daily scrums or Kanban boards.

Oh, and for extra points,

  • The builds that I’m getting are very unstable. The system falls over under the most basic of smoke tests. I have to do a lot of waiting or reconfiguring or both before I can even get started on the other stuff.

How might we consider these observations?

We could choose to interpret them as problems for testing, but we could think of them differently: as test results.

Test results don’t tell us whether something is good or bad, but they may inform a decision or an evaluation or more questions. People observe test results and decide whether there are problems and what the problems are, what further questions are warranted, and what decisions should be made. Doing that requires human judgement and wisdom, consideration of lots of factors, and a number of possible interpretations.

Just as for automated checks and other test results, it’s important to consider a variety of explanations and interpretations for testing meta-results—observations about testing—lest we miss an important problem. As Jerry Weinberg points out in Perfect Software and Other Illusions About Testing, whatever else something might be, it’s information. If testing is, as Jerry says, gathering information with the intention of informing a decision, it seems a mistake to leave potentially valuable observations lying around on the floor. Indeed, rather than thinking of them as problems for testing, we could choose to think of them as symptoms of product or project problems—problems that testing can help to solve.

For example, when a tester feels outnumbered by programmers, or when a tester feels under time pressure, that’s a test result. The feeling often comes from the programmers generating more work and more complexity than the tester can handle. Yet complexity, like quality, is a relationship between some person and something else. Complexity on its own isn’t necessarily a problem; it’s how people deal with it and its attendant risks that’s a problem. When we observe the ways in which people react to a perception of complexity, we might learn a lot.

  • Are people conscious of the risks—especially the Black Swans—that typically accompany complexity?
  • If people are conscious of risk, are they paying attention to it? Are they panicking over it? Or are they ignoring it and whistling past the graveyard? Or…
  • Are people reacting calmly and pragmatically? Are they acknowledging and dealing with the complexity of the product? If they can’t make the product or the process that it models less complex, are they at least taking steps to make understanding of the product more tractable?
  • Might the programmers be generating or modifying code so quickly that they’re not taking the time to understand what’s really going on with it?
  • If someone feels that more testers are needed, what’s behind that feeling? (I took a stab at an answer to that question a few years back.)

How might we figure that out answers to those questions? One way might be to look at more of the test results and test meta-results.

  • Does someone perceive testing to be difficult or time-consuming? Who? What’s the basis for that perception? What assumptions underlie it?
  • Does the need to investigate and report bugs overwhelm the testers’ capacity to obtain good test coverage? (I wrote about that problem here.)
  • Does testing consistently reveal consistent patterns of failure?
  • Are programmers consistently surprised by such failures and patterns?
  • Do small changes in the code cause problems that are disproportionately large or hard to find?
  • Do the programmers understand the interdependencies clearly? Are those interdependencies necessary, or could they be eliminated?
  • Are programmers taking steps to anticipate or prevent problems related to interfaces and interactions?
  • If automated checks are difficult to develop and maintain, does that say something about the skill of the tester, the quality of the automation interfaces, or the scope of checks? Or about something else?
  • Are unstable builds a problem that get in the way of deeper testing? Or could we interpret them as a sign that the product has problems so numerous and serious that even shallow testing reveals them?
  • When a “stable” build appears after a long series of unstable builds, how stable is it really?

Perhaps, with the answers to those questions, we could raise even more questions.

  • What risks do those problems present for the success of the product, whether in the short term or the longer term?
  • When testing consistently reveals patterns of failures and attendant risk, what does the product team do with that information?
  • Are the programmers mandated to deliver code? Or are the programmers mandated to deliver code with a warrant that the code does what it should (and doesn’t do what it shouldn’t), to the best of their knowledge? Do the programmers adamantly prefer the latter mandate?
  • Is someone pressuring the programmers to make schedule or scope commitments that they can’t really fulfill?
  • Are the programmers and the testers empowered to push back on scope or schedule pressue when it adds to product or project risk?
  • Do the business people listen to the development team’s concerns? Are they aware of the risks that testers and programmers bring to their attention? When the development team points out risks, do managers and business people deal with them congruently?
  • Is the team working at a sustainable pace, or might we expect the product and the project to become overwhelmed by complexity, interdependencies, fragility, and problems that lurk just beyond the reach of our development and testing effort?
  • Is the development team really Agile, in the sense of the precepts of the Agile Manifesto? Or is “agility” being used in a cargo-cult way, using practices or artifacts to mask over an incoherent project?

Testers often feel that their role is to find, investigate, and report on bugs in the product. That’s usually true, but it’s also a pretty limited view of the kinds of information that testing reveals. When seen one way, the problems I’ve listed above sound like serious problems for testing. What if we also remembered Jerry’s definition of testing as “gathering information with the intention of informing a decision”? If that’s the case, then everything that we notice or discover during testing is a test result.

(See also this discussion for an example of looking beyond the test result for possible product and project risks.)

Posted in Acceptance Tests, Exploratory Testing, Management, Models, Risk, Systems Thinking, Test Framing, Tester Skill, Testing Story, Testing vs. Checking, Time, Words and Semantics | 6 Comments »

Common Languages Ain’t So Common

Tuesday, June 28th, 2011

A friend told me about a payment system he worked on once. In the system models (and in the source code), the person sending notification of a pending payment was the payer. The person who got that notice was called the payee. That person could designate somone else—the recipient—to pick up the money. The transfer agent would credit the account of the recipient, and debit the account of the person who sent notification—the payer, who at that point in the model suddenly became known as the sender. So, to make that clear: the payee sends email to the payer, who receives it. The sender pays money to the recipient (who accepts the payment.) Got that clear? It turns out there was a logical, historical reason for all this. Everything seemed okay at the beginning of the project; there was one entity named “payer” and another named “payee”. Payer A and Payee B exchanged both email and money, until someone realized that B might give someone else, C, the right to pick up the money. Needing another word for C, the development group settled on “recipient”, and then added “sender” to the model for symmetry, even though there was no real way for A to split into two roles as B had. Uh, so far.

There’s a pro-certification argument that keeps coming back to the discussion like raccoons to a garage: the claim that, whatever its flaws, “at least certification training provides us with a common language for testing.” It’s bizarre enough that some people tout this rationalization; it’s even weirder that people accept it without argument. Fortunately, there’s an appropriate and accurate response: No, it doesn’t. The “common language” argument is riddled with problems, several of which on their own would be showstoppers.

  • Which certification training, specifically, gives us a common language for testing? Aren’t there several different certification tribes? Do they all speak the same language? Do they agree, or disagree on the “common language”? What if we believe certification tribes present (at best) a shallow understanding and a shallow description of the ideas that they’re describing?
  • Who is the “us” referred to in the claim? Some might argue that “us” refers to the testing “industry”, but there isn’t one. Testing is practiced in dozens of industries, each with its own contexts, problems, and jargon.
  • Maybe “us” refers to our organization, or our development shop. Yet within our own organization, which testers have attended the training? Of those, has everyone bought into the common language? Have people learned the material for practical purposes, or have they learned it simply to pass the certification exam? Who remembers it after the exam? For how long? Even if they remember it, do they always and everafter use the language that has been taught in the class?
  • While we’re at it, have the programmers attended the classes? The managers? The product owners? Have they bought in too?
  • With that last question still hanging, who within the organization decides how we’ll label things? How does the idea of a universal language for testing fit with the notion of the self-organizing team? Shouldn’t choices about domain-specific terms in domain-specific teams be up to those teams, and specific to those domains?
  • What’s the difference between naming something and knowing something? It’s easy enough to remember a label, but what’s the underlying idea? Terms of art are labels for constructs—categories, concepts, ideas, thought-stuff. What’s in and what’s out with respect to a given category or label? Does a “common language” give us a deep understanding of such things? Please, please have a look at Richard Feynman’s take on differences between naming and knowing, http://www.youtube.com/watch?v=05WS0WN7zMQ.
  • The certification scheme has representatives from over 25 different countries, and must be translated into a roughly equivalent number of languages. Who translates? How good are the translations?
  • What happens when our understanding evolves? Exploratory testing, in some literature, is equated with “ad hoc” testing, or (worse) “error guessing”. In the 1990s, James Bach and Cem Kaner described exploratory testing as “simultaneous test design, test execution, and learning”. In 2006, participants in the Workshop on Heuristic and Exploratory Techniques discussed and elaborated their ideas on exploratory testing. Each contributed a piece to a definition synthesized by Cem Kaner: “Exploratory software testing is a style of software testing that emphasizes the personal freedom and responsibility of the individual tester to continually optimize the value of her work by treating test-related learning, test design, test execution, and test result interpretation as mutually supportive activities that run in parallel throughout the project.” That doesn’t roll off the tonque quite so quickly, but it’s a much more thorough treatment of the idea, identifying exploratory testing as an approach, a way that you do something, rather than something that you do. Exploratory work is going on all the time in any kind of complex cognitive activity, and our understanding of the work and of exploration itself evolves (as we’ve pointed out here, and here, and here, and here, and here.). Just as everyday, general-purpose languages adopt new words and ideas, so do the languages that we use in our crafts, in our communities, and with our clients.

In software development, we’re alway solving new problems. Those new problems may involve people to work with entirely new technological or business domains, or to bridge existing domains with new interactions and new relationships. What happens when people don’t have a common language for testing, or for anything else in that kind of development process? Answer: they work it out. As Peter Galison notes in his work on trading zones, “Cultures in interaction frequently establish contact languages, systems of discourse that can vary from the most function-specific jargons, through semispecific pidgins, to full-fledged creoles rich enough to support activities as complex as poetry and metalinguistic reflection.”  Each person in a development group brings elements of his or her culture along for the ride; each project community develops its own culture and its own language.

Yes, we do need common language for testing. Anthropology shows us that meaningful language develops organically when people gather for a common purpose in a particular context. Just as we need tests that are specific to a given context, we need terms that are that way too. So instead of focusing training on memorizing glossary entries, let’s teach testers more about the relationships between words and ideas. Let’s challenge each other to ask better questions about the language we’re using, and how it might be fooling us.

Posted in Certification, Context, Exploratory Testing, Learning, Test Framing, Tester Skill, Training, Words and Semantics | 1 Comment »

Framing Test Framing

Tuesday, May 17th, 2011

Rikard Edgren is a testing philosopher in Sweden. He independently develops his own ideas on testing (an example here), collaborates with his colleagues on The Test Eye, and critiques and builds on the work of other people in the community. He offered a comment to my recent post on test framing, and the comment deserves a post on its own. He said,

I don’t think explicit framing is needed very often in reality, often it is apparent that a test is needed. But I think it is an essential skill, that when being trained, will bring along other testing skills as well. And sometimes you definitely need it, e.g. when presenting a problem that might not be adressed otherwise.

Yes, and that’s an important point. We will rarely be asked to frame a test, and we will rarely do it explicitly, or in writing. However, I argue that we should always and at any time be able to frame a test; that is, to tie the elements of the testing story together with logical connections.

However, framing would fit my testing world better with a few changes:

1) add “description” (or similar) so framing is about how you describe and present a test or a problem.
This also aligns better with how “framing” is used in other fields, e.g. you can frame something in different ways, so it suits the audience, and the most important aspects.

Well, maybe. To me, the description dominates the framing; the framing, the logical structure, is a subset of and a basis for the description. When you frame something in different ways—that is, when you reframe it—you’re presenting a new perspective, for sure. But to me the key element is that the new description is based on different premises, and a different set of propositions and logical connections.

2) Use “a” instead of “the”. With current definition it is easy to think that there is one “true” frame, but in reality there can be many.

Doh! You’ve caught me in violation of the “a vs. the” heuristic. The “the” in this case implies that you’ve got a particular frame in mind, and that it’s the one you’re going use to deliver your interpretation. Nonetheless, you’re right: there may be multiple ways of framing a given test. Lucky for both of us that the “a vs. the” heuristic is a heuristic, not a rule.

3) Remove “logical”, so framing can take advantage of people’s ability to connect things in other ways than logic, so it isn’t necessary to give all the details for all framings. The logic part is one important bit, but there should be focus on other rationale/feelings as well.

There is every opportunity for other rationales and feelings to appear in the framing of a test. Framing doesn’t deny emotions; on the contrary. I think you might be conflating logical oracles and logical argumentation. There’s a difference.

There are certain aspects of a program that are informed by logic, such that a logical principle or mechanism points to a problem. That is, the oracle is based on a logical decision rule. To use the trivial example of an adding program,

IF the program is intended to add numbers, AND
IF we are using the rules of everyday mathematics, AND
IF we add two numbers
AND the addition of those two numbers produces a result that is not equal to their sum,
THEN the result is mathematically false AND
IF we see a result that’s mathematically false
THEN we assume that there’s a problem with the program.

In this case, logical and mathematical truth or falsehood act as our oracle—while we’re also framing the test by establishing logical connections between proposition.

Yet we can use logical connections to frame tests based other oracles as well.

IF the program is intended to provide us with the service of adding numbers conveniently AND
IF we perform a test in which we tab from one control to the next AND
IF the controls are laid out such that tabbing from the first input field value to the second involves a tab stop on another control AND
IF that wastes the customer’s time or distracts her needlessly
THEN we can justify the claim that there’s a bug here.

Note that the basis for the bug is qualitative, but the line of reasoning used to frame the test is logical, based on propositions and logical connectors.

This gives us something like: Test framing is a description of connections that structure, inform, and motivate a test.

I’d see the “description” you have here as implicit in the way James and I define framing: the set of logical connections that stucture and inform a test.

And here’s a made-up problem report example:

Main content of web site should be loaded first
When the web site is launched it starts to display sidebars and various frames before the main content of the page.
This is normally not a problem, but with slow connection, e.g. a mobile phone outside town, it gives a not so strong impression (see video available at …)

First impressions are important to our site, and my testing colleagues agree with me that a better display sequence would be:
company logo -> main content -> the rest

Here we have holes, subjective opinions, but still something that is easy to understand and make a decision from.
OK as framing or not?

I’m glad you’ve given me an example, because it clarifies the misunderstanding. In answer to your question, I would suggest No, not okay as framing. There are propositions—statements—here, but they’re not linked by logical connectives. Let me propose a counter-example:

IF we colleagues are in agreement on principles that govern the order in which elements are displayed AND
IF we believe that our company’s logo should be the most prominent element in the Web site AND
IF we believe that the main content is the most important element of a web page to the users, AND
THEREFORE we further believe that the logo and the main content should be displayed before the other elements
UNLESS the connection is so fast that users cannot perceive a difference; WHEREAS
IF, when the web site is launched, it starts to display sidebars and various frames before the main content of the page, AND
IF there the order of loading is perceptible AND
IF the that order is not only perceptible but prominent on a slow connection AND
IF slow connections are common on mobile phones outside of town AND
IF we presume that a significant proportion of our users will be using such connections AND
IF we believe that those users will suffer annoyance AND
IF that annoyance is also the first impression that the user receives
THEREFORE we believe that this delay represents a threat to our image AND
IF we view threats to our image as a bug
THEREFORE we have reason to characterize our observation of the loading order as a bug or at least an issue.

Now, this is a pretty elaborate construction, but it can happen pretty much at the speed of thought. The point is that we can trace from premise to conclusion through logically connected statements and propositions. I think if I asked you to do that with your example above, you could do it easily, using informal yet rigorous logic. Yet, as my earlier post suggested, testers aren’t used to doing that. I think the craft could use practice in the art of logical argumentation, even when it’s about qualitative issues that aren’t themselves governed by logic.

Posted in Test Framing | 2 Comments »

Exploratory Testing is All Around You

Monday, May 16th, 2011

I regularly converse with people who say they want to introduce exploratory testing in their organization. They say that up until now, they’ve only used a scripted approach.

I reply that exploratory testing is already going on all the time at your organization.  It’s just that no one notices, perhaps because they call it

  • “review”, or
  • “designing scripts”, or
  • “getting ready to test”, or
  • “investigating a bug”, or
  • “working around a problem in the script”, or
  • “retesting around the bug fix”, or
  • “going off the script, just for a moment”, or
  • “realizing the significance of what a programmer said in the hallway, and trying it out on the system”, or
  • “pausing for a second to look something up”, or
  • “test-driven development”, or
  • “Hey, watch this!”, or
  • “I’m learning how to use the product”, or
  • “I’m shaking out it a bit”, or
  • “Wait, let’s do this test first instead of that test”, or
  • “Hey, I wonder what would happen if…”, or
  • “Is that really the right phone number?”, or
  • “Bag it, let’s just play around for a while”, or
  • “How come what the script says and what the programmer says and what the spec says are all different from each other?”, or
  • “Geez, this feature is too broken to make further testing worthwhile; I’m going to go to talk to the programmer”, or
  • “I’m training that new tester in how to use this product”, or
  • “You know, we could automate that; let’s try to write a quickie Perl script right now”, or
  • “Sure, I can test that…just gimme a sec”, or
  • “Wow… that looks like it could be a problem; I think I’ll write a quick note about that to remind me to talk to my test lead”, or
  • “Jimmy, I’m confused… could you help me interpret what’s going on on this screen?”, or
  • “Why are we always using ‘tester’ as the login account? Let’s try ‘tester2′ today”, or
  • “Hey, I could cancel this dialog and bring it up again and cancel it again and bring it up again”, or
  • “Cool! The return value for each call in this library is the round-trip transaction time—and look at these four transactions that took thirty times longer than average!”, or
  • “Holy frijoles! It blew up! I wonder if I can make it blow up even worse!”, or
  • “Let’s install this and see how it works”, or
  • “Weird… that’s not what the Help file says”, or
  • “That could be a cool tool; I’m going to try it when I get home”, or
  • “I’m sitting with a new tester, helping her to learn the product”.

Now it’s possible that none of that stuff ever happens in your organization. Or maybe people aren’t paying attention or don’t know how to observe testing. Or both.

Then, just before I posted this blog post, James Bach offered me two more sure-fire clues that people are doing exploratory testing. If they say, “I am in no way doing exploratory testing”, or “We’re doing only highly rigorous formal testing”. In both cases, the emphatic nature of the claim guarantees that the claimant is not sufficiently observant about testing to realize that exploratory testing is happening all around them.

Posted in Exploratory Testing, Heuristics, Learning, Management, Oracles, Test Framing, Tester Skill, Testing Story, Testing vs. Checking, Words and Semantics | 8 Comments »

I’ve Been Framed!

Monday, May 16th, 2011

Last week in the Netherlands, I presented a workshop on test framing, a skill that James Bach and I like to talk about. Test framing is the set of logical connections that structure, inform, and motivate a test.

Testing is a process of composing, editing, narrating, and justifying a story with three parallel threads. There is a product element—the part of the story that describes product, how it works, how it fails, and how it might fail in ways that matter to our clients. That story is backed by the activity element—the part of the story that describes what we did and how we did it. Here we describe what we did, how we designed our tests, and how we configured, operated, observed and evaluated the product. The narrative of our activities gives warrant to the product story, making it credible and accountable.

In turn, the testing story needs its own warrant, so there’s a justification element to supply that. The activity element is about what we did, how we did, when we did it, who was involved. The justification element is about why. In the justification part of the story, we talk about the quality of our work, and the obstacles and issues that might threaten it. We explain why we’ve done what we’ve done, why we haven’t done other things yet, or why we won’t do certain things at all unless things change. We relate the activities to the testing mission and the product story, and explain why the testing work we’ve done is good enough, and why it’s reasonable to have postponed or written off the testing work we haven’t done.

Framing is the network of logical statements and connectives that link the three story elements and their details.  To frame a test means to create an unbroken line of logic from the mission—the information objective within some context—through the product story and our activities, down to the observed outcome of the test and our interpretation of it. (I’ve written about that here, too.) The logic consists of propositions—philospher-talk for “regular statements”—linked by logical connectives.  In some kinds of formal logic, only specific operators are permitted, like “if”, “and”, “not”, “or”, “if and only if”. Test framing uses a somewhat less formal and more permissive form of logic, but the intention is to provide a means of reasoning through a test idea, its execution, and its interpretation.  There are a couple of worked-out examples in this document, which was first presented at EuroSTAR in November, 2010, and which has been undergoing revision from time to time.

James and I have observed that many of the people in our classes and those who approach us for coaching really struggle with framing a test. Some of these testers are relatively inexperienced, of course. Yet we also observe that very experienced and skilled testers often struggle almost as hard. It seems that people aren’t used to the practice. It’s not that their testing is unmotivated, exactly; more that they’re not used to articulating and justifying their testing. That creates a feedback loop, which I’ll describe with a few gross generalizations:

a) Since testers don’t tell the story of their testing, clients don’t understand what testers are up to.

b) In order to evaluate the quality of testing, clients use a bunch of surrogate measures such as test case counts and defect escape ratios.

c) Testers get out of the habit of telling the testing story, and learn to emphasize the numbers (to please the client).

d) Go to (a).

So if you can work as a tester for years without framing your tests, and people ask you to frame tests rarely or not at all, is an inability to frame tests a problem? We think so.

  • Test framing is an important element in bug advocacy—the skill of presenting a problem report to a testing client in a way that accurately conveys the meaning, significance, and risk associated with our actions, observations and evaluations. Without the capacity to trace the logical reasoning behind test activity, our bug advocacy will be considerably weaker.
  • Decisions about quality are inherently subjective. As Jerry Weinberg points out, they’re political and emotional, yet the people who make them want to appear rational. Good test framing can help to provide some rationally based grounds for justifying quality decisions—and for providing reasonable alternatives.
  • Test framing provides a way for more experienced testers to describe important parts of testing to novices or trainees.
  • Usually people talk about traceability to requirements, when really they mean traceability to explicit requirements in requirements documents. Good test framing can provide traceability to implicit or as-yet-unstated requirements. That is, test framing can help to reveal and develop a richer and more representative requirements story.
  • Test framing logically ties the elements of testing story together—the product element, the testing element, and the justification element. Test framing is like the sinew that joins the bones of the testing story.
  • Practicing test framing can help us to frame other aspects of the project, including requirements, specifications, bugs, and management decisions. Is someone saying something that doesn’t make sense to you? Try framing it. Framing an observation, inference, or decision can help you to understand the elements that have gone into it, and the logic that links them—or the logical gaps that make it unreasonable, unjustified, or risky.
  • Finally, and perhaps most importantly, framing is an important aspect of self-management and exploratory approaches. A tester that can connect the mission of testing to the test result and everything in between can also connect the logical and emotional aspects of testing. Moreover, a tester that can frame his or her work can show a rational basis for it, which is key to developing trust. A tester that can’t frame testing work into a logical structure is more likely to be directed and controlled by managers and clients.  Whether true or not, managers will percieve that the testers don’t understand what they are doing.  Even if the testers do understand, they won’t get credit for it.

After the aforementioned workshop, I posted a couple of documents on the Web and notified the people who attended. Shortly thereafter, I got this reply that I can quote with Francien’s kind permisison:

From: Francien Ramakers [mailto:xxxxxxxx@xxx.com]
Sent: 12 May 2011 05:36
To: Michael Bolton
Subject: RE: Test Framing at TestNet

Hi Michael,

The first link does not work :( .

When I click on it I get a page with “You know, I could have sworn that page was here a minute ago.” from your blog.

Framing of this test:

- If the purpose of the link was to provide the readers of your email with a document and if the idea was or this to be possible simply by clicking on the link within the email and if clicking on it does not result in display of a document, and if the other links do work fine (indicating that that the issue is unlikely to be caused by another problem, such as being offline) then something is wrong with the first link.

- If the purpose of the first link in your email was to test how many people download the material after you mail them and if all people that try to download any of it click on the first link and if all people that click on the link have got the same result as me and if they all mail you when they do then this is not a bug but a feature.

In both cases, I would still really like to receive the document it refers to. Could you please send it to me?

Hmmm… and now I am not sure whether I framed the test or the bug…

Kind regards,

Francien.

Note that Francien has supplied not only a logical basis for the theory that she’s witnessing a bug—but also that she’s providing another, equally logical basis for an alternative interpretation of her observations. That’s important, because a tester is someone who knows that things can be different. Locking on to a single interpretation is a risky business for a tester, since a single proposition somewhere in the frame can flip the interpretation of the test from bug to feature, and vice-versa.

So I replied.

From: Michael Bolton [mailto:michael@developsense.com]
Sent: donderdag 12 mei 2011 12:56
To: Francien Ramakers
Subject: RE: Test Framing at TestNet

Hi, Francien…

Well done!

I would like to suggest that you framed the test and the bug. I can frame the reason for the bug,

if you like.

IF I give a presentation AND

IF a part of my mission in delivering the presentation is to supply people potentially with useful notes and information

THEN I should prepare an email for them which has the correct links to that information in it

AND IF someone complains that one of those links doesn’t work

THEN I suspect that I’ve made an error in the links

THEREFORE I test the link

AND IF I can reproduce the problem

THEN I suspect the link is incorrect

UNLESS I didn’t upload the file in the first place (which I now realize I didn’t)

THEREFORE I really shouldn’t do all this stuff too late at night.

:)

—Michael B.

Francien replied,

Hi Michael,

Thanks! :)

I think it’s cool that you framed the answer!

Mailing about it made me realize framing can be applied to other things then just tests.

Cheers,

Francien.

Francien, of course, is right. Framing can be about many other things: not only testing, but also about bugs, requirements, specifications, and so forth.

For extra points: in my framing, can you identify the proposition in my framing that is not logically connected to the others?

Posted in Test Framing | 3 Comments »

Questioning Test Cases, Part 2: Testers Learn, But Test Cases Don’t Teach

Wednesday, April 6th, 2011

In the last post, my LinkedIn correspondent provided a couple of reasons why she liked writing test cases, and why she thought it necessary to write them down in the early stages of the project. Then she gave a third reason:

When I’m on a project and I am the only one who knows how to test something, then I can’t move on to something new. I’d still be testing Cut/Copy/Paste if I were not able to pass that script on easily to a new recruit.

I often hear this kind of justification for test cases: “We need test cases so that testers can learn quickly how to test the product.” To me, that’s like saying that we need driving cases so that people can learn quickly how to drive in a new city. In a new city, a map or a general set of ideas might be helpful. Driving cases won’t, because to drive through a new city well, you have to know how to drive. In the same way, test cases are unlikely to be anything more than negligbly helpful. In order to learn quickly how to test a new program well, you have to know how to test.

It also seemed to me that she considers that writing to be the One True Way to impart information, and that reading to be the One True Way to receive it. If I’m wrong about that, it’s because I’ve misinterpreted something she’s written. Fancy that!

So for the sake of discussion, let’s say I were to to hire (say) an older domain expert, making a career change, who had no computer experience, and let’s use Cut/Copy/Paste as an example. With the new recruit, I’d do what I’ve done many times in coaching sessions. I’d sit with her down with a copy of Microsoft Word, and open up a document that had some text and some images in it. Then I’d say something like this:

“We’re going to learn how to move text around, cutting and pasting and making copies of it so we can save time on retyping stuff that the machine can do for us. Okay. Put your cursor there, and press Shift-RightArrow there. Good. See what happened?”

(There would be lots of pauses. I’d pause after each question to give the trainee time to answer; frequently to let the trainee experiment; and often to let the trainee ask questions of her own. I won’t do that explicitly here; I’ll let you put most of the pauses in yourself, as you read. For sure, assume that there’s one at least after every question, and usually after every sentence.)

“So try doing that a few more times. What happened? So you’ve highlighted the text; people call that marking or selecting the text. What happens when you add the Ctrl key, and press Ctrl-Alt-RightArrow at the same time? Right: it selects a word at a time. Now, try pressing Ctrl-X.

“Whoa, what happened just there?! Right: pressing Ctrl-X that makes the selected text go away; you can see that. What you might not know is that the text goes onto the Windows Clipboard, a kind of temporary storage area for all kinds of stuff. So you haven’t erased it; it’s more like you’ve cut it out, and the computer still has a copy of it stashed somewhere. Now, put your cursor here, and hit Ctrl-V. What did that do? Yeah; it pasted the text you cut before. That is, it retrieved the text from the clipboard, and made a copy right where you pressed Ctrl-V. So, Ctrl-X cuts, and Ctrl-V pastes a copy of the thing you cut. Try that again. What happened? Right; another copy. So it’s a little different from regular cutting and pasting, in that you can make many copies of the same thing. When might that come in handy? Good idea! Yep, that’s a good idea too. Try it a few more times, in other places. What does Ctrl-V do? What does Ctrl-X do?

“Try selecting something else, and pressing Ctrl-X again. What happens when you paste now? Can you get the old one back? Well, there might be special add-ons that allow you to keep copies of lots of things you pasted, but normally, you can only paste the last thing you put on the Clipboard.

“How are you going to remember that Ctrl-X means “cut”? That’s a great idea! Myself, I think of the X as a pair of open scissors, ready to cut—but your idea is great, and better than that, it’s your own, so you’ll remember it. How are you going to remember Ctrl-V as paste? Yeah, it’s tough to come up with a way to remember that, isn’t it? I think of it like the tip of one of those old glue bottles from grade school. On the other hand, what’s in between X and V on the keyboard? Now, go select some text again. Remember how? Good. This time, try pressing Ctrl-C, instead of Ctrl-X. Right; nothing seems to happen. And yet… move your cursor here, and try pasting. Yeah; it’s a copy, even though the original stays where it was. So Ctrl-C is Copy–C for copy, which makes some kind of sense, for a change—right there in between cut and paste. And there’s another way to remember; try pressing Alt-E. E stands for Edit, that’s right. What do you see listed on the menu, there?

“Now… there’s some documentation for these cut-and-paste features; it’s in the Windows documentation. Press F1. Great. Now look for “Keyboard Shortcuts”. There’s a list. How many different ways can you find of marking, cutting, and pasting text? Make me a list, and I’ll be back in ten minutes. Be ready to show me your report, and to demonstrate the ones you’ve discovered. Oh—and one more thing: create a list of at least five circumstances in which you might use this stuff, and we’ll talk about that too.”

It’s taken quite a while to type this example. It would take considerably less time to do it. Moreover, the trainee would learn more quickly and more deeply, because she’d be interacting with the product herself, experiencing the actions in her body, literally incorporating them. The questions, the answers, and the interactions make the learning more sticky. The practice makes it more sticky still. I try to encourage people to create mnemonics to help remember. While conversing with the trainee, I’d also be observing her. She’ll be making plenty of mistakes; people do that when they’re learning. In fact, people should be encouraged to do that when they’re learning. (Brian Marick’s book Everyday Scripting in Ruby encourages deliberate mistakes, which is one of the reasons I like it so much.) When I see mistakes, I’ll give her a chance to feel the mistake, puzzle out a way around it, and then—if she needs it—help her get out of it. (“Let’s figure out how to fix that. Hit Alt-E again. What’s listed on the Edit menu? So what’s the keyboard shortcut for Undo?” “Oops, you pressed Ctrl-Z one time too many… What does Ctrl-Y do?”). After a while, I won’t tell her how to do everything right away. I’ll start asking her to figure out some things on her own, and only help out when she’s stuck. That way she’ll be learning it for herself, which is more challenging but more affecting. I’ll keep trying to raise the bar, giving her more challenging assignments, increasing her independence while also increasing her responsibility to report. When I see her make a mistake, I might correct her right away, or I might let her continue with the mistake for a bit until she encountered some kind of minor consequence. That makes the learning even more sticky. Finally, I’d cut the conversation short if she told me that she already knew how to cut and paste—but I’d get her to show me that she knew it, and I’d give her some more exercises so that I could observe her.

The trouble that I see with “passing on a script to a new recruit” is that most test scripts I’ve seen are very long on what to do or how to do it, and correspondingly short on why to do it and how to generalize it. That is, they’re short on motivation, and they’re short on risk, and they rarely include a mission to learn. In our teaching, when we’re working with test groups, and when we’re coaching individual testers, James and I put less emphasis on techniques. Instead, we focus on making sure that people have the skills to know what to do when they receive a testing mission, whether that mission is written down in detail or not.

If there are specific things that I want a tester to look for, I’ll provide specific instructions (either written or spoken), but mostly I want to set out on their own to discover things. I want to encourage testers to vary their work and their observations. Variation makes it more that they will find problems that would go off hiding, were we to focus on the scripts. As a bonus, giving testers a concise, open-ended task keeps writing to a minimum, which saves on development cost, maintenance cost, and opportunity cost.

We’ll finish up this series in the next post.

Posted in Accountability, Test Framing, Tester Skill, Training | 9 Comments »

Questioning Test Cases, Part 1

Monday, April 4th, 2011

Over the years, LinkedIn seems to have replaced comp.software.testing as the prime repository for wooly thinking and poorly conceived questions about testing.

Recently I was involved in a conversation with someone who, at least, seemed to be more articulate than most of the people on LinkedIn. Alas, I’ve since lost the thread, and after some searching I’ve been unable to find it. No matter: the points of the discussion are, to me, are still worth addressing. She was asking a question about how much time to allocate to writing test cases before starting testing, and I questioned the usefulness of doing that. The first part of her reply went like this:

I would like to point out that I doubt anyone wants to write something that they don’t need to write.

I agree that most people probably don’t want to write things they don’t need to write. But they often feel compelled, or are compelled to write things they don’t need to write.

I find value in writing test cases for a number of reason. One is that I train more junior engineers in testing and it is a good method to have them execute tests that I have written so they learn how a good test plan is put together.

If that were so, wouldn’t your junior engineers learn even more from writing test cases themselves, and getting feedback on their design and their writing? There’s a feedback loop in the design of a test, the execution of a test, the interpretation of a test result, and the learning that happens between them; wouldn’t it be a good idea to keep the feedback loop—and the learning—as rapid as possible? Wouldn’t your junior engineers learn still more from actually testing—under your close supervision, at first, and then with the freedom and responsibility to act more independently as they gain skill? You might want to have a look at this article: http://www.developsense.com/articles/2008-06-KnowWhereYourWheelsAre.pdf.

There’s a common misconception that testing happens in the characters of a written test case. It doesn’t. Testing happens in the mind and actions of the tester. It happens in the design of the test, in the execution of the test, in the observation and interpretation of the outcome of the test. Testing happens in the discovery of problems, in the investigation of those problems, and the learning about those problems and the product. At most, a fraction of this can be written down.

A test is far less something you execute, and far more a line of inquiry that you follow. To me, a good test case is idea-stuff; it’s a question that we want to ask of the program, based on some motivating idea about discovery or risk. In my observation, in writing test cases, people generally write down the least important stuff. They appear to be trying to program the tester’s actions, rather trying than to prime the tester’s thinking and observation.

Moreover, a test plan something quite different from a pile of test cases.

Secondly, [writing test cases] communicates the testing coverage with everyone involved in developing the software. If you are a contractor, this is very important since you want to leave with the client feeling like you did your job and they have the documentation to prove that they have done due diligence if they shop their company around or look for for VC money.

Were you, as a contractor, given the mission to produce test scripts specifically, or is that a mission that you have inferred? Bear in mind, I’ve been witness to many takeovers, as a program manager at a company where our senior managers were ambitiously acquiring technologies, products and companies, and as a consultant to several companies that were taken over by larger companies. In no case did anyone ever ask to see any test case documentation. Those who are investigating the company to be acquired typically don’t go to that level of detail. In my experience, alas, due diligence largely doesn’t happen at all. I’m puzzled, too, by the appeal to the least likely instances in which people might interact with test documention, rather than the everyday.

Meanwhile, there are many ways to communicate test coverage. (For example, see here, here, and here.) There are also many ways to fool yourself (and others) into believing that the more documentation the more coverage, or the more specific the documentation the more coverage—especially when that documentation is prospective, rather than retrospective. In Rapid Testing, we don’t encourage people to eliminate documentation. We encourage people to reduce documentation to what is actually necessary for the mission, and to eliminate wasteful documentation.

We focus on documenting test ideas concisely; on producing coverage outlines and risk lists that can be used to guide, rather than control a tester and her line of inquiry; on producing records of the tester’s thought process, observations, risk ideas, motivations, and so forth (see below). The goal is to capture things far more important than the tester’s mechanical actions. If someone wants a record of that, we recommend video capture software (tools like BB Test Assistant or Camtasia). An automatic log allows the tester to focus on testing the product and recording the ideas, rather than splitting focus between operating the product and writing about operating the product.

You can find examples of the kind of test documentation I’m talking about here, in the appendices to the Rapid Software Testing class, starting at page 47. Note that each example has varying degrees of polish and formal structure. Sometime it’s highly informal, used to aid memory, to frame a conversation, or trigger ideas. Sometimes it’s more formal and polished, when the audience is outside the test group. The over-riding point is to fit the documentation to the mission and to the task at hand.

More to come…

Posted in Accountability, Test Framing, Tester Skill, Training | 6 Comments »

More of What Testers Find, Part II

Friday, April 1st, 2011

As a followup to “More of What Testers Find“, here are some more ideas inspired by James Bach’s blog post, What Testers Find. Today we’ll talk about risk. James noted that…

Testers also find risks. We notice situations that seem likely to produce bugs. We notice behaviors of the product that look likely to go wrong in important ways, even if we haven’t yet seen that happen. Example: A web form is using a deprecated HTML tag, which works fine in current browsers, but may stop working in future browsers. This suggests that we ought to do a validation scan. Maybe there are more things like that on the site.

A long time ago, James developed The Four-Part Risk Story, which we teach in the Rapid Software Testing class that we co-author. The Four-Part Risk Story is a general pattern for describing and considering risk. It goes like this:

  1. Some victim
  2. will suffer loss or harm
  3. due to a vulnerability in the product
  4. triggered by some threat.

A legitimate risk requires all four elements. A problem is only a problem with respect to some person, so if a person isn’t affected, there’s no problem. Even if there’s a flaw in a product, there’s no problem unless some person becomes a victim, suffering loss or harm. If there’s no trigger to make a particular vulnerability manifest, there’s no problem. If there’s no flaw to be triggered, a trigger is irrelevant. Testers find risk stories, and the victims, harm, vulnerabilities, and threats around which they are built.

In this analysis, though, a meta-risk lurks: failure of imagination, something at which humans appear to be expert. People often have a hard time imagining potentional threats, and discount the possibility or severity of threats they have imagined. People fail to notice vulnerabilities in a product, or having noticed them, fail to recognize their potential to become problems for other people. People often have trouble making the connection between inanimate objects (like nuclear reactor vessels), the commons (like the atmosphere or sea water), or intangible things (like trust) on the one hand, and people who are affected by damage to those things on the other. Excellent testers recognize that a ten-cent problem multiplied by a hundred thousand instances is a ten-thousand dollar problem (see Chapter 10 of Jerry Weinberg’s Quality Software Management, Volume 2: First Order Measurement). Testers find connections and extrapolations for risks.

In order to do all that, we have to construct and narrate and edit and justify coherent risk stories. To to that well, we must (as Jerry Weinberg put it in Computer Programming Fundamentals in 1961) develop a suspicious nature and a lively imagination. We must ask the basic questions about our products and how they will be used: who? what? when? where? why? how? and how much? We must anticipate and forestall future Five Whys by asking Five What Ifs. Testers find questions to ask about risks.

When James introduced me to his risk model, I realized that there people held at least three different but intersecting notions of risk.

  1. A Bad Thing might happen. A programmer might make a coding error. A programming team might design a data structure poorly. A business analyst might mischaracterize some required feature. A tester might fail to investigate some part of the product. These are, essentially, technical risks.
  2. A Bad Thing might have consequences. The coding error could result in miscalculation that misrepresents the amount of money that a business should collect. The poorly designed data structure might lead to someone without authorization getting access to privileged information. The mischaracterized feature might lead to weeks of wasted work until the misunderstanding is detected. The failure to investigate might lead to an important problem being released into production. These are, in essence, business risks that follow from technical risks.
  3. A risk might not be a Bad Thing, but an Uncertain Thing on which the business is willing to take a chance. Businesses are always evaluating and acting on this kind of risk. Businesses never know for sure whether the Good Things about the product are sufficiently compelling for the business to produce it or for people to buy it. Correspondingly, the business might consider Bad Things (or the absence of Good Things) and dismiss them as Not Bad Enough to prevent shipment of the product.

So: Testers find not only risks, but links between technical risk and business risk. Establishing and articulating those links are depend on the related skills of test framing and bug advocacy. Test framing is the set of logical connections that structure and inform a test. Bug advocacy is the skill of determining the meaning and significance of a bug, and reporting the bug in terms of potential risks and consequences that other people might have overlooked. Bug advocacy doesn’t mean jumping up and down and screaming until every bug—or even one particular bug—is fixed. It means providing context for your bug report, helping managers to understand and decide why they might to choose to fix a problem, right now, later, or never.

In my travels around the world and around the Web, I observe that some people in our craft have some fuzzy notions about risk. There are at least three serious problems that I see with that.

Tests are focused on (documented) requirements. That is, test strategies are centred around making sure that requirements are checked, or (in Agile contexts) that acceptance tests derived from user stories pass. The result is that tests are focused on showing that a product can meet some requirement, typically in a controlled circumstance in which certain stated conditions assumed necessary have been met. That’s not a bad thing on its own. Risk, however, lives in places where where necessary conditions haven’t been stated, where stated conditions haven’t been met, or where assumptions have been buried, unfulfilled, or inaccurate. Testing is not only about demonstrating that some instance of a requirement has been satisfied. It’s also about identifying things that threaten the successful fulfillment of that requirement. Testers find alternative ideas about risk.

Tests don’t get framed in terms of important risks. Many organizations and many testers focus on functional correctness. That can often lead to noisy testing—lots of problems reported, where those problems might not be the most important problems. Testers find ways to help prioritize risks.

Important risks aren’t addressed by tests. A focus on stated requirements and functional correctness can leave parafunctional aspects of the product in (at best) peripheral vision. To address that problem, instead of starting with the requirements, start with an idea of a Bad Thing happening. Think of a quality criterion (see this post) and test for its presence or its absences, or for problems that might threaten it. Want to go farther? My colleague Fiona Charles likes to mention “story on the front page of the Wall Street Journal” or “question raised in Parliament” as triggers for risk stories. Testers find ways of developing risk ideas.

James’ post will doubtless trigger more ideas about what testers find. Stay tuned!

P.S. I’ll be at the London Testing Gathering, Wednesday, April 6, 2011 starting at around 6:00pm. It’s at The Shooting Star pub (near Liverpool St. Station), 129 Middlesex St., London, UK. All welcome!

Posted in Risk, Systems Thinking, Test Framing, Testing Story | 3 Comments »

« Older Entries