Blog: Testing Doesn’t Improve the Product

(This post is adapted from my recent article on LinkedIn.)

Out there in the world, there is a persistent notion that “preventing problems early in the software development process will lead to higher-quality products than testing later will”. That isn’t true.

It’s untrue, but not for the reason that might first occur to most people. The issue is not that addressing problems early on is a bad idea. That’s usually a really good idea.

The issue is the statement is incoherent. Testing on its own, whether done early or late, will not lead to higher quality products at all.

Problem prevention, product improvements, and testing are different pursuits within development work. These activities are related, but testing can neither prevent problems nor improve the product. Something else, beyond testing, must happen.

Coming from me—a teacher and an advocate for skilled testing—that might seem crazy, but it’s true: testing doesn’t improve the product.

Investigative journalism is an information-gathering activity. Investigative journalists reveal problems in companies and governments and groups, problems that affect society.

Awareness of those problems may lead to public concern or outcry. The news reports on their own, though, don’t change anything. Change happens when boards, regulators, politicians, legal systems, leaders, workers, or social groups take action.

Testing, too, is an information-gathering activity. That information can be used to recognize problems in the product (“bugs”), or to identify aspects of the product that do not represent errors but that nonetheless could be improved (“enhancement requests”). Gathering information plays a role in making things better, but it doesn’t make things better intrinsically and automatically.

Consider: weighing yourself doesn’t cause you to lose weight. Blood tests don’t make you healthier. Standardized tests in schools don’t make kids smarter, and certainly don’t improve the quality of education.

What testing can do is to improve our understanding and awareness of whatever might be in front of us. Testing—the process of evaluating a product by learning about it through experiencing, exploring, and experimenting—helps our teams and our clients to become aware of problems. On becoming aware of them, our teams and clients might decide to address them.

To put it another way: testing is questioning a product in order to evaluate it. Neither the questions nor the answers make the product better. People acting on the answers can make the product better.

Similarly, in daily life, a particular reading on a bathroom scale might prompt us to eat more carefully, or to get more exercise, whereupon we might become more fit. A blood test might prompt a doctor to prescribe anti-malarial drugs, and if we take them as prescribed, we’re likely to control the malaria. Those standardized school tests might suggest changes to the curriculum, or to funding for education, or to teacher training. But until someone takes action, the test only improves awareness of the situation, not the situation itself.

In software development, improvement doesn’t happen unlesss someone addresses the problems that testing helps us to discover. Of course, if the problems aren’t discovered, improvement is much less likely to happen—and that’s why testing is so important. Testing helps us to understand the product we’ve got, so we can decide whether it’s the product we want. Where improvement is necessary, testing can reveal the need for improvement.

Some people believe that testing requires us to operate a product, thinking in terms of the product as a built piece of software. That’s a very important kind of testing, but it’s only one kind of testing activity, referring to one kind of product.

It can be helpful to consider a more expansive notion of a product as something that someone has produced. This means that testing can be applied to units or components or mockups or prototypes of an application.

And although we might typically call it review, we can a kind of testing to things people have written, or sketched, or said about a software product that does not yet exist. In these cases, the product is the artifact or the ideas that is represents. In this kind of test, experimentation consists of thought experiments; exploration applies to the product and to the space, or context, in which it is situated; experiencing the product applies to the process of analysis, and to experiences that we could imagine.

The outcome of the test-as-thought-experiment is the evaluation and learning that happens through these activities. That learning can be applied to correcting errors in the design and the development of the product—but once again, it’s the work that happens in response to testing, not the testing itself, that improves the product.

Just as testing doesn’t improve products, testing doesn’t prevent problems either. As testers, we have an abiding faith that there are already problems in anything that we’ve been asked to test. That is, the problems in the product are there before we encounter them. We must believe that problems have not been prevented. Indeed, our belief that problems have not been successfully prevented is a key motivating idea for testing work.

So what good is testing if it can’t prevent problems? Testing can help us to become aware of real problems that are really there. That’s good. That might even be great, because with that awareness, people can make changes to prevent those unprevented problems from going any further, and that’s good too.

It’s a great idea for people who design and build products to try to prevent problems early in development. To the degree that the attempt can be successful, the builders are more likely to develop a high-quality product. Nonetheless, problems can elude even a highly disciplined development process. There are at least two ways to find out if that has happened.

One way is to test the product all the way through its development, from intention to realization. Test the understanding of what the customer really wants, by engaging with a range of customers and learning about what they do. Test the initially fuzzy and ever-sharper vision of the designer, through review and discussion and what-if questions.

Test the code at its smallest units and at every stage of integration, through more review, pairing, static analysis tools, and automated output checking. Check the outputs of the build process for bad or missing components and incorrect settings. These forms of testing are usually not terribly deep. That’s a a good thing, because deep testing may take time, effort, and preparation that can be disruptive to developers. Without deep testing, though, bugs can elude the developers.

So, in parallel to the developers’ testing, assign some people to focus on and to perform deep testing. Deep testing is targeted towards rare, hidden, subtle, intermittent, emergent bugs that can get past the speedy, shallow, non-disruptive testing that developers—quite reasonably—prefer most of the time.

If your problem-prevention and problem-mitigation strategies have been successful, if you’ve been testing all along, and if you’ve built testability into the product, you’ll have a better understanding of it. You’ll also be less likely to encounter shallow problems late in the game. If you don’t have to investigate and report those problems, deep testing can be relatively quicker and easier.

If your problem-prevention and problem-mitigation strategies have been unsuccessful, deep testing is one way to find out. The problems that you discover can be addressed; the builders can make improvements to the product; and problems for the business and for the customer can be prevented before the product ships.

The other way to find out if a problem has eluded your problem prevention processes is to release the product to your otherwise unsuspecting customers, and take the chance that the problems will be both infrequent and insignificant enough that your customers won’t suffer much.

Here are some potential objections:

If software testing does not reveal the need for improvement then the improvement will not happen.

That’s not true. Although testing shines light on things that can be improved, improvement can happen without testing.

Testing can happen without improvement, too. For instance…

  • I perform a test. I find a bug in a feature. The program manager says, “I disagree that that’s a bug. We’re not doing anything in response to that report.”
  • I perform a test. I find a bug in a feature. The program manager says “I agree that that’s a bug. However, we don’t have time to fix it before we ship. We’ll fix it in the next cycle.”
  • I test. I find a bug. The program manager agrees that it’s a bug. The developer tries to fix it, but makes a mistake and the fix is ineffective.
  • I test. I find a bug. The program manager agrees, the developer fixes that bug, but along the way introduces new bugs, each of which is worse than the first.

In each case above, 1) Has the product been tested? (Yes.) 2) Has the product been improved? (No.)

Saying that testing doesn’t improve the product diminishes the perceived value of testing.

Saying that testing does improve the product isn’t true, and miscalibrates the role of the tester relative to the people who design, build, and manage the product.

Let’s be straight about this: we play a role in product improvement, and that’s fine and valuable and honourable. Being truthful and appropriately humble about the extents and limits of what testing can actually do diminishes none of its value. We don’t design or develop or improve the product, but we give insight to the people who do.

The value argument in favour of testing is easy to make. As I pointed out above, investigative journalists don’t run governments and don’t set public policy. Would you want them to? Probably not; making policy is appropriately the role of policy-makers. On the other hand, would you want to live in a society without investigative journalism? Now: would you want to live a world of products that had been released without sufficiently deep testing?

When there’s the risk of loss, harm, bad feelings, or diminished value for people, it’s a good idea to be aware of problems before it’s too late, and that’s where testing helps. Testing on its own neither prevents problems nor improves the product. But testing does make it possible to anticipate problems that need to be prevented, and testing shines light on the places where the product might need to be improved.

Want to know more? Learn about upcoming Rapid Software Testing classes here.

2 responses to “Testing Doesn’t Improve the Product”

  1. That’s simply untrue.

    Michael replies: No, it isn’t untrue. Testing does not improve the product. I don’t know why this is so difficult for people to accept.

    Changing the code in a happy way improves the product. Adding a feature that people like improves the product (for those people, but maybe not for others). Fixing a bug in the code improves the product. Revising ideas about the product improves those ideas, whereupon we might improve the product.

    Testing provides an understanding of the status and behaviour of the product, but doesn’t improve it.

    Weighing yourself does not improve your health. Reviewing the movie doesn’t make the movie better.
    Diagnosing a problem on the printer does not make the printer better. Spotting a forest fire from a tower doesn’t put the fire out and save the forest. Noticing a problem in a Formula One car doesn’t make the car better or win the race.

    And, I might add, when I critically evaluate and challenge people’s idea that testing improves the product, they don’t automatically change their minds. They have to decide to consider my arguments and choose to change their minds. If they don’t change their minds, that’s evidence that testing an idea doesn’t improve it. Something else has to happen.

    No, it isn’t.

    Testing, especially in the early stages (requirement gathering) improves the product.

    No it does not, as you immediately point out in the very next sentence.

    Sure, people might ignore bug reports, might accept some risks and might disagree with the tester, but most of the times (even in those cases), the added knowledge will improve their work since they now treat implicit requirements as explicit – Should we validate incoming input? What should happen if both features here are active at the same time? etc.

    If people ignore bug reports, the product does not improve. If people pay attention to bug reports, preferring to decline action and accept some risks, the product does not improve. If people disagree with the tester and do not change the product, the product does not improve. The product doesn’t improve unless someone chooses to do something about the information that testing reveals. Your protesting that testing improves the product does not change this fact.

    While it’s reasonable to say that a single instance of testing had no significant impact on the product (e.g.: I did a shallow confirmatory check and found that the software behaves as expected),

    In this case you tested, and there was no impact on the product at all. You tested, and you didn’t improve the product.

    a good testing process will improve the software since it will provoke responses from the intended audience.

    Yes; and unless those responses involve improving the product, the product doesn’t improve.

    Suppose that the response is “Thanks, but I don’t care to address that report,” and then the product doesn’t improve. Did testing improve the product? No; the product did not improve, and testing did not improve the product.

    Suppose that after testing, someone attempted to change the product, and the change made no difference. Did testing improve the product? No; the product did not improve, and testing did not improve the product.

    Suppose that after testing, someone attempted to change the product, and muffed up the change so badly that the product got worse. Did testing improve the product? No; the product in fact got worse, and testing did not improve the product.

    Reporting and advocating are an integral part of testing (If I interact with a product and learn about it, but don’t report my findings, were I testing? I’d say not, at best I was practicing some testing-related skills), and those two are measured by the reactions they manage to provoke.

    In the case that you test and don’t report your findings, you’re still testing. Testing without reporting happens all the time even when you do report your findings, since you are not reporting every thought, every action, every observation.

    Besides, the reason companies do testing is in order to improve their product, decision making and processes – so if testing does not lead to any improvement, we’re doing it wrong.

    Imagine that we’re about to release the product. The developers have done an excellent job at building the product and fixing it. Our last burst of testing reveals that there are no problems in the product worth fixing. The product is not changed, and not improved. Was that last round of testing done wrong?

    If you want to improve the product, testing helps a lot. Testing helps to turn assumptions and inferences and theories about the product—and gaps in all of those—into observations and facts. Testing helps to improve our understanding of the product, and that improved understanding helps the process of improving the product, but testing on its own does not improve the product itself.

  2. […] Testing Doesn’t Improve the Product Written by: Michael Bolton […]

Leave a Reply

Your email address will not be published. Required fields are marked *