Blog Posts from February, 2018

Finding the Happy Path

Wednesday, February 28th, 2018

In response to yesterday’s post on The Happy Path colleague and friend Albert Gareev raises an important issue:

Until we sufficiently learned about the users, the product, and the environment, we have no idea what usage pattern is a “happy path” and what would be the “edge cases”.

I agree with Albert. (See more of what he has to say here.) This points to a kind of paradox in testing and development: some say that we can’t test the product unless we know what the requirements are—yet we don’t know what many of the requirements are until we’ve tested! Testing helps to reduce ambiguity, uncertainty, and confusion about the product and about its requirements—and yet we don’t know how to test until we’ve tried to test!

Here’s how I might address Albert’s point:

To test a whole product or system means more than demonstrating that it can work, based on the most common or optimistic patterns of its use. We might start testing the whole system there, but if we wanted to develop a comprehensive understand of it, we wouldn’t stop at that.

On the other hand, the whole system consists of lots of sub-systems, elements, components, and interactions with other things. Each of those can be seen as a system in itself, and studying those systems contributes to our understanding of the larger system.

We build systems, and we build ideas on how to test them. At each step, considering only the most capable, attentive, well-trained users; preparing only the most common or favourable environments; imagining only the most optimistic scenarios; performing only happy-path testing on each part of the product as we build it; all of these present the risk of misunderstanding not only the product but also the happy paths and edge cases for the greater system. If we want to do excellent testing, all of these things—and our understanding of them—must not only be demonstrated, but must be tested as well. This means we must do more than creating a bunch of high-level, automated, confirmatory checks at the beginning of the sprint, and then declaring victory when they all “pass”.

Quality above depends on quality below; excellent testing above depends on excellent testing below. It’s testing all the way down—and all the way up, too.

Very Short Blog Posts (32): The Happy Path

Monday, February 26th, 2018

“Happy path testing” isn’t really testing at all. Following the “happy path” is a demonstration.

Here’s the role demonstration plays in testing: it’s nice to know that your product can achieve the happy path before you start to test it. To the degree a demonstration is a test, it’s a very shallow test.

If you’re building something new and non-trivial that matters to people, or that could harm people, there’s a risk that you might not entirely understand it or what it affects. To develop your understanding, you’ll probably want to test it; to learn about it; to investigate it; to interact with it directly; to probe it with tools; to stress it out. You’ll probably want to explore it and experiment with it; to evaluate it. That’s testing.

If you can’t even achieve the happy path, you’re not ready for testing.

Related posts:
Finding the Happy Path
Testing and Checking Refined
Why Checking is Not Enough
Acceptance Tests: Let’s Change the Title, Too
More of What Testers Find
Why We Do Scenario Testing

How is the testing going?

Thursday, February 8th, 2018

Last week on Twitter, I posted this:

“The testing is going well.” Does this mean the product is in good shape, or that we’re obtaining good coverage, or finding lots of bugs? “The testing is going badly.” The product is in good shape? Testing is blocked? We’re noting lots of bugs erroneously?

People replied offering their interpretations. That wasn’t too surprising. Their interpretations differed; that wasn’t too surprising either. I was more surprised at how many people seemed to believe that there was a single basis on which we could say “the testing is going well” or “the testing is going badly”—along with the implicit assumption that people would automatically understand the answer.

To test is—among many other things—to construct, edit, narrate, and justify a story. Like any really good story, a testing story involves more than a single subject. A excellent, expert testing story has at least three significant elements, three plot lines that weave around each other like a braid. Miss one of those elements, and the chance of misinterpretation skyrockets. I’ve talked about this before, but it seems it’s time for a reminder.

In Rapid Software Testing, we emphasize the importance of a testing story with three strands, each of which is its own story.

We must tell a story about the product and its status. As we have tested, we have learned things about the product: what it is, what it does, how it works, how it doesn’t work, and how it might not work in ways that matter to our various clients. The overarching reason that most clients hire testers is to learn about problems that threaten the value of the product, so bugs—actual, manifest problems—tend to lead in the product story.

Risks—unobserved but potential problems—figure prominently in the product story too. From a social perspective, good news about the product is easier to deliver, and it does figure in a well-rounded report about the product’s state. But it’s the bad news—and the potential for more of it—that requires management attention.

We must tell a story about the testing. If we want management to trust us, our product story needs a warrant. Our product story becomes justified and is more trustworthy when we can describe how we configured, operated, observed, and evaluated the product. Part of this second strand of the testing story involves describing the ways in which we recognized problems; our oracles. Another part of this strand involves where we looked for problems; our coverage.

It’s important to talk about what we’ve covered with our testing. It may be far more important to talk about what we haven’t covered yet, or won’t cover at all unless something changes. Uncovered areas of the test space may conceal bugs and risks worse than any we’ve encountered so far.

Since we have limited time and resources, we must make choices about what to test. It’s our responsibility to make sure that our clients are aware of those choices, how we’re making them, and why we’re making them. We must highlight potentially important testing that hasn’t been done. When we do that, our clients can make informed decisions about the risks of leaving areas of the product untested—or provide the direction and resources to make sure that they do get tested.

We must tell a story about how good the testing is. If the second strand of the testing story supports the first, this third strand supports the second. Here it’s our job to describe why our testing is the most fabulous testing we could possibly do—or to the degree that it isn’t, why it isn’t, and what we need or recommend to make it better.

In particular, we must describe the issues that present obstacles to the fastest, least expensive, most powerful testing we can do. In the Rapid Software Testing namespace, a bug is a problem that threatens the value of the product; an issue is a problem that threatens the value of the testing. (Some people say “issue” for what we mean by “bug”, and “concern” for what we mean by “issue”. The labels don’t matter much, as long people recognize that there may be problems that get in the way of testing, and bring them to management’s attention.)

A key element in this third strand of the testing story is testability. Anything that makes testing harder, slower, or weaker gives bugs more time and more opportunity to survive undetected. Managers need to know about problems that impede testing, and must make management decisions to address them. As testers, we’re obliged to help managers make informed decisions.

On an unhappy day, some time in the future, when a manager asks “Why didn’t you find that bug?”, I want to be able to provide a reasonable response. For one thing, it’s not only that I didn’t notice the bug; no one on the team noticed the bug. For another, I want to be able to remind the manager that, during development, we all did our best and that we collaboratively decided where to direct our attention in testing and testability. Without talking about testing-related issues during development, those decisions will be poorly informed. And if we missed bugs, I want to make sure that we learn from whatever mistakes we’ve made. Allowing issues to remain hidden might be one of those mistakes.

In my experience, testers tend to recognize the importance of the first strand—reporting on the status of the product. It’s not often that I see testers who are good at the second strand—modeling and describing their coverage. Worse, I almost never encounter test reports in which testers describe what hasn’t been covered yet or will not be covered at all; important testing not done. As for the third strand, it seems to me that testers are pretty good at reporting problems that threaten the value of the testing work to each other. They’re not so good, alas, at reporting those problems to managers. Testers also aren’t necessarily so good at connecting problems with the testing to the risk that we’ll miss important problems in the product.

Managers: when you want a report from a tester and don’t want to be misled, ask about all three parts of the story. “How’s the product doing?” “How do you know? What have you covered, and what important testing hasn’t been done yet?” “Why should we be happy with the testing work? Why should we be concerned? What’s getting in the way of your doing the best testing you possibly could? How can we make the testing go faster, more easily, more comprehensively?”

Testers: when people ask “How is the testing going?”, they may be asking about any of the three strands in the testing story. When we don’t specify what we’re talking about, and reply with vague answers like “the testing is going well”, “the testing is going badly”, the person asking may apply the answer to the status of the product, the test coverage, or the quality of the testing work. The report that they hear may not be the report that we intended to deliver. To be safe, even when you answer briefly, make sure to provide a reply that touches on all three strands of the testing story.

Testing and Management Efficiency

Monday, February 5th, 2018

To a naïve manager who doesn’t understand testing very well, the visible manifestation of testing is a tester, sitting in front of a computer, banging on keys to operate a product, comparing output with a predicted result, and marking “pass” or “fail” on a spreadsheet.

Therefore, thinks the manager: one way to make testing dramatically more efficient and effective is to automate the testing. Substitute the human tester for a program on the same computer. Have the program bang on virtual keys to manipulate products and tools. Then algorithmically compare program output to a predicted result; and then mark “pass” or “fail” on an algorithmically generated report. (It doesn’t really help that many testers, managed by such managers, may believe in the same idea.)

To a naïve tester who doesn’t understand management very well, the visible manifestation of management is a manager, sitting in front of a computer, receiving messages from employees, banging on keys to scroll through documents and write a memo, checking the memo for spelling and grammar, attaching it to a message in Microsoft Outlook, and pressing “Send”.

Therefore, thinks the tester: one way to make management dramatically more efficient and effective is to automate the management. Install optical character recognition and voice recognition software onto the manager’s computer. Have a program feed problems, requests, and supporting documents into a spreadsheet full of business rules. Next, apply those rules, and feed the output into a memo-generating template that includes the words “approved” or “rejected”. Then have the program run the memo through a spelling and grammar checker, attach the memo to an Outlook message, and press Send. (In reality, even the most naive tester doesn’t believe that would lead to good management.)

Moral: some physical, visible, mechanical behaviours are included in each job, but is not the important part of either one. Both jobs require studying a situation, wrestling with uncertainty, comprehending business value (and threats to it), observing processes, analyzing risk, making decisions, performing actions, and interpreting outcomes.

Tools can help enormously with all of those activities, but for testing and management alike, fixation on visible behaviours misses the point. Obsession with mechanizing the visible things risks displacing the real work.