Blog Posts for the ‘Requirements’ Category

Four (and More) Questions for Testers to Ask

Sunday, March 11th, 2018

Testers investigate problems and risk. Other people manage the project, design the product, and write the code. As testers, we participate in that process, but in a special way and from a special perspective: it’s our primary job to anticipate, seek, and discover problems.

We testers don’t prevent problems; we don’t design or build or fix the product. We may help to prevent existing problems from going any farther, by discovering bugs, misunderstandings, issues, and risks and bringing them to light. With our help, the people who build and manage the project can address the problems we have revealed, and prevent worse problems down the line.

Over the last while, I’ve been working with clients that are “shifting left”, “going Agile”, “doing DevOps”, or “getting testers involved early”. Typically this takes the form of having a tester present for design discussions, planning meetings, grooming sessions, and the like.

This is usually a pretty good idea; if there is no one in the testing role, people tend not to think very deeply about testing—or about problems or risk. That’s why, even if you don’t have someone called “tester” on the team, it’s an awfully good idea to have someone in the testing role and the testing mindset. Here, I’ll call that person “tester”.

Alas, I’ve sometimes observed that, once invited to the meetings, testers are sometimes uncertain about what they’re doing there.

A while back, I proposed at least four things for testers to do in planning meetings: learning; advocating for testability; challenging what we’re hearing; and establishing our roles as testers. These activities help to enable sensemaking and critical thinking about the product and the project. How can testers do these things successfully? Here’s a set of targeted questions.

What are we building? Part of our role as testers is to come to a clear understanding of the system, product, feature, function, component, or service that we’re being asked to test. (I’ll say “product” from here on, but remember I could be referring to anything in the list.) We could be talking about the product itself or a representation of it. We could be looking at a diagram of it;reviewing a document or description of it; evaluating a workflow; playing with a prototype. Asking for any these can help if we don’t have them already. A beneficial side effect is helping to refine everyone’s understanding of the product—and how we’d achieve successful completion of the project or task.

So we might also ask: What will be there when we’ve built it? What are the bits and pieces? (Can we see a diagram?) What are the functions that the product offers; what should the product do? What gets input, processed, and output? (Do we have a data dictionary?) What does the product depend upon? What depends on the product? (Has someone prepared a list of dependencies? A list of what’s supported and what isn’t?)

For whom are we building it? If we’re building a product, we’re ultimately building it for people to use. Sometimes we make the mistake of over-focusing on a particular kind of user: the person who is immediately encountering the product, with eyes on screen and fingers on keyboard, mouse, or glass. Often, however, that person is an agent for someone else—for a bank teller’s application, think of the bank teller, but also think of the customer on the other side of the counter; the bank’s foreign exchange traders; the bank teller’s manager. Beyond using the product, there are other stakeholders: those who support it, connect to its APIs, test it, document it, profit from it, or defend it in court.

So we might also ask: Who else is affected by this product? Who do they work for, or with? What matters to them? (These questions are targeted towards operations value-related testability.) Who will support the product? Maintain it? Test it? Document it?

What could go wrong? The most important questions for testers to raise are questions about problems and risks. Developers, designers, business people, or others might discuss features or functions, but people who are focused on building a product are not always focused on how things could go badly. Switching from a builder’s mindset to a tester’s mindset is difficult for builders. For testers, it’s our job.

So we might also ask: What Bad Things could happen? What Good Things could fail to happen? Under what conditions might they happen or not happen? What might be missing? What might be there when it shouldn’t be there? And for whom are we not building this product—like hackers or thieves?

When something goes wrong, how would we know? Once again, this is a question about testability, and also a question about oracles. As James Bach has said, “software testing is the infinite art of comparing the invisible to the ambiguous to prevent the unthinkable from happening to the anonymous”. For any non-trivial program, there’s a huge test space to cover, and bugs and failures don’t always announce themselves. Part of our job is to think of the unthinkable and to help those invisible things to become visible so that we can find problems—ideally in the lab before we ship. Some problems might escape the lab (or our continuous deployment checks, if we’re doing that).

So we might also ask: How might we miss something going wrong? What do we need for intrinsic testability—at the very least, log files, scriptable interfaces, and code that has been reviewed, tested, and fixed as it’s being built. And what about subjective testability? Do we have the domain knowledge to recognize problems? What help might we need to obtain that? Do we have the specialist skills—in (for example) security, performance, or tooling—on the team? Do we need help there? If we’re working in a DevOps context,
doing live site testing or testing in production, how would we detect problems rapidly?

In sprint planning meetings, or design discussions, or feature grooming sessions, questions like these are important. Questions focused on problems don’t come naturally to many people, but asking such questions should be routine for testers. While everyone else is envisioning success, it’s our job to make sure that we’re anticipating failure. When everyone else is focused on how to build the product, it’s important for us to keep an eye on how the entire team can study and test it. When everyone else is creatively optimistic, it’s important for us to be pragmatically pessimistic.

None of the activities in planning and review replace testing of the product that is being built. But when we participate in raising problems and risks early on, we can help the team to prevent those problems—including problems that make testing harder or slower, allowing more bugs to survive undetected. Critical thinking now helps to enable faster and easier testing and development later.

Now a word from our sponsor: I help testers, developers, managers, and teams through consulting and training in Rapid Software Testing (RST). RST is a skill set and a mindset of testing focused on sharpening critical thinking, eliminating waste, and identifying problems that threaten the value of the product or the project, and the principles can be adapted to any development approach. If you need help with testing, please feel free to get in touch.

Very Short Blog Posts (31): Ambiguous or Incomplete Requirements

Monday, December 19th, 2016

This question came up the other day in a public forum, as it does every now and again: “Should you test against ambiguous/incomplete requirements?”

My answer is Yes, you should. In fact, you must, because all requirements documents are to some degree ambiguous or incomplete. And in fact, all requirements are to some degree ambiguous and incomplete.

And that is an important reason why we test: to help discover how the product is inconsistent with people’s current requirements, even though it might be consistent with requirements that they may have specified—ambiguously or incompletely—at some point in the past.

In other words: we test not only to compare the product to documented requirements, but to discover and help refine requirements that may otherwise be ambiguous, unclear, inconsistent, out of date, unrecognized, or emergent.

Very Short Blog Posts (9): “Insufficient Requirements”

Wednesday, November 27th, 2013

Some people say they “don’t have enough requirements to start testing,” or that the requirements are unclear or incomplete or contradictory or out of date.

First, those people usually mean requirements documents; don’t confuse that with requirements, which may be explicit or tacit. There are plenty of sources of requirements-related information, and it’s a tester’s job to discover them and make inferences about them.

Second, insufficient clarity about requirements is both a test result and a project risk, and awareness of them may be crucially important project information. (If the testers aren’t clear on the requirements, are the programmers? And if they’re not, how is it that they’re building the product?)

Finally, if there is uncertainty about requirements, one great way around that problem is to start testing and reporting on what you find. “Insufficient requirements” may be a problem for testing—but it’s also precisely a problem that testing can help to solve.

Very Short Blog Posts (5): Understanding the Requirements

Monday, October 28th, 2013

People often suggest that “understanding the requirements” is an essential step before you can begin testing. This may be true for checking or formal testing—examining a product in a specific way, or to check specific facts. But understanding the requirements is not a necessary precursor to testing, which is learning about a product through experimentation (a larger activity which might include checking) and creating the conditions to make that activity possible. Indeed, you may need to test in order to develop an understanding of the requirements, which in turn triggers more and better testing, yielding even better understanding of the requirements—and so on.

More generally, when thinking about testing, think more about loops, and less about lines.