Blog Posts for the ‘Uncategorized’ Category

Test Cases and Coverage

Thursday, June 28th, 2018

A tester recently asked about creating an in-house testing process, wanting to know how to start writing test cases so that testing effort could be estimated. My reply—which could just as easily apply to a developer or business analysts in a testing role—went something like this:

Test cases are not testing!  While that’s true, just saying so probably won’t help you very much, so let me offer an alternative to thinking about your assignment in terms of test cases.

At the beginning of the project, you don’t yet know how to test it. To develop a relevant test strategy, you must learn the product. There may be many ways for you to learn about the product—by attending meetings, talking to people, reviewing project plans, studying designs or sketches. You may even have an opportunity to work with the product directly; with a previous version; with a mockup or prototype; or with a part of the product.

Without a good deal of that learning, you cannot know what test cases should describe or contain—so instead of the typical fine granularity of test cases, consider the broader notion of coverage areas.

As you learn about a product or feature, create a coverage outline based on product factors. (A product factor is something that could be examined in a test, or something that could affect the outcome of a test.)  Your outline may take one or more of many forms—a hierarchical list, a table, an annotated diagram; these days we like mind maps.  The outline identifies areas of the product and its relationship to its context that can be covered with testing activity. In general, these areas will tend to fall into categories like structure, functions, data, interfaces, platforms, operations, and time.

You might want to note any specific factors or conditions that you believe must be checked—but since you’re at the beginning of the project, also note that no one can be aware of all of them yet.

While you’re at it, begin advocating for testabilty; things that can make testing faster and easier. A product tends to be faster and easier to test when developers provide controllability through scriptable interfaces (APIs); visibility through log files; and code that they have already tested for basic problems and coding errors. When programmers do more of their own testing (including automated checks), testers won’t have to investigate and report on shallow bugs—which leaves more time for deep testing.

Your coverage outline goes hand in hand with a risk list—in which you identify technological and business risks that will influence your strategies for covering the test space. As development progresses, we learn about potential problems that threaten the value of the product. It’s a special responsibility of someone in the testing role to explore ideas about problems and where they might be lurking; to perform experiments to identify problems; and to determine the nature of those problems so that developers and managers can make informed decisions on what to do about them.

Consider the kinds of activities that testers perform: making contact with the team; learning about the product; analyzing how to test it; preparing equipment and tools; developing ideas for deep coverage; doing deep testing; and winding down the testing and helping prepare the product for deployment.  Then, instead of estimating by test cases, or by hours or days, consider estimating by sessions.

Sessions are bursts of nominally uninterrupted testing work, guided by a mission statement or charter. The time span of a session is intentionally imprecise, and we say “nominally uninterrupted” because, in the real world, it’s unusual for work not to get interrupted.  When planning a session, we base its charter on the idea that it will take around 90 minutes.  In reality, a session may last from 45 minutes to two and a half hours.  That’s because we can identify what we’d like to happen in a session, but our knowledge of what will happen is uncertain.

For instance, the bugs that you encounter during a session will affect how much you’re able to cover during the session. When bugs require investigation and reporting, coverage suffers. So, after the session, the charter may change to reflect what actually happened.  The difference between what you hope to do and what you’ll be able to do can’t be known very well in advance—so guess how many sessions you’ll need to cover the stuff that you know about.  Then double your guess.

What will happen over the course of a collection of sessions is uncertain too. You’ll need to allocate plenty of time to account for setting up for each session; dealing with obstacles; debriefing; debugging with developers; planning meetings; breaks and lunch; and so forth. As with the specifics of what you’re going to cover, that stuff is uncertain too.  However, it’s reasonable to assume, given the non-session work, a tester will have time for two or three sessions at most in a day.

In the early stages, you’re going to have to guess about a lot of this. That’s normal; testing is about exploration, discovery, investigation, and learning.  Those things aren’t calculable except in the most general way, and work tends to expand or shrink to fit the time available for it. Recognize that little of your early planning will play out as anticipated.

As you go through loops of learning and testing, your coverage outline and risk list will evolve. Commit to providing daily reports on what is being learned; and on the gap between what has been covered and what has not yet been covered—product coverage and risk coverage. Daily reports will keep managers alert to problems that need their attention—problems that might threaten the value of the product or the success of the project.  Managers need to know about those, not about test case counts.

As a key element in your reporting, provide notes on testability and its absence—anything that makes the product easier or harder to test. Reduced testability means testing takes longer or reduces coverage.  Reduced coverage means bugs will survive undetected for longer—maybe into production.

Every day, as you learn about the product and update your models of it, you will be informing your clients on what testing has been done, and what testing could be done.  In the end, your clients will make the decision about when they have sufficient information about coverage and risk to ship the product.  It’s your job to keep them up to date and well-informed; it’s their job to decide whether the product they’ve got is the product they want.

All that said, you might not find it necessary to estimate the amount of testing that the project needs. Since both you and the client already have an idea of what the schedule and budget will be for this project anyway (based on past experience and other factors), your estimate may already be done for you. If not, and you’re asked to provide and estimate, don’t provide one; provide three: a reasonable estimate; a safer estimate; and a riskier estimate.

I anticipate someone will respond “all this sounds harder than preparing a list of test cases.” I don’t believe it is—I’d say it’s easier, in fact. A coverage outline is certainly easier to maintain than a library of test cases.  Working from an outline acknowledges the fact that we’ll learn and refine our plans as we go, where the test case approach turns us into ostriches when it comes to anticipating change.

Some people will respond “But managers want test cases!” Yes, some do; and kids want candy instead of real food, and addicts want illegal drugs, too. We are not obliged to do harmful things, and fixation on test cases puts more emphasis on documentation and procedures than on actual testing. I believe most managers want test cases because they’re used to them, and because they’re not aware of alternatives. Testers provide test cases because managers keep asking about them. I’d offer real testing work, continuous collaboration, refinement of the tasks at hand—and solid information about coverage, problems, and risks.

Related reading:

No Test Cases Required (PowerPoint presentation)
Questioning Test Cases
Testers Learn, but Test Cases Don’t Teach

Rapid Software Testing is designed for managers, developers, testers, and teams who want help with making testing faster, easier, cheaper, and above all, effective. Read about RST classes here, and consider the upcoming public class taught by both James Bach and Michael Bolton.

Very Short Blog Posts (34): Checking Inside Exploration

Monday, April 23rd, 2018

Some might believe that checking and exploratory work are antithetical. Not so.

In our definition, checking is “the algorithmic process of operating and observing a product, applying decision rules to those observations, and reporting the outcome of those decision rules”.

We might want to use some routine checks, but not all checks have to be rote. We can harness algorithms and tools to induce variation that can help us find bugs. Both during development of a feature and when we’re studying and evaluating it, we can run checks that use variable input; in randomized sequences; at varying pace; all while attempting to stress out or overwhelm the product.

For instance, consider inducing variation and invalid data into checks embedded in a performance test while turning up the stress. When we do that, we can discover when the product falls to its knees under how much load—how the product fails, and what happens next. That in turn affords the opportunity to find out whether the product deals with overhead associated with error handling—which may result in feedback loops and cascades of stress and performance problems.

We can use checks as benchmarks, too. If a function takes significantly and surprisingly more or less time to do its work after a change, we have reason to suspect a problem.

We could run checks in confirmatory ways, which, alas, is the only way most people seem to use them. But we can also design and run checks taking a disconfirmatory and exploratory approach, affording discovery of bugs and problems. Checking is always embedded in testing, which is fundamentally exploratory to some degree. If we want to find problems, it would be a good idea to explore the test space, not just tread the same path over and over.

Interview on Rapid Software Testing

Tuesday, March 27th, 2018

Not too long ago, I had a conversation with Joe Colantonio. Joe asks questions that prompt answers about the nature of testing, accountability, and telling the testing story.

Enjoy!

Signing Off

Thursday, March 1st, 2018

Testers ask: “I’m often given a product to test, but not enough time to test it. How am I supposed to sign off on the release when I haven’t tested enough?”

My reply goes like this:

If you’re a tester, it seems profoundly odd to me that you are responsible for signing off on a release. The decision to release a product is a business decision, not a technical one. The decision is, of course, informed by technical considerations, so it’s entirely reasonable for you to provide a report on those considerations. It would be foolish for for someone in a business role to ignore your report. But it would be just as foolish for a someone in a business role to abdicate the business decision to technical people. We serve the business; we don’t run it, and technical people often aren’t privy to things in the business’ domain.

The idea that testers can either promote or prevent a release can be tested easily. Try refusing to sign off until you’re entirely happy with what you know about the product, and with the extent of what you know. You’ll get a fairly quick result.

Perhaps managers will go along with you. You’ll test the product until you are satisfied that you have covered everything important, that you have discovered all the significant problems, that those problems have been fixed, and that the discovery of any more problems that matter would be a big surprise to you. Your client (that is, the person to whom you report) will give you all the time and all the resources you need until the product meets your standards. Yes, that seems unlikely to me, too.

More likely, at some point you will be overruled. Management will decide that your concerns are not important enough to block the release. Then you will be able to say, “So, the decision to approve or prevent the release is not really my decision? Cool.” Then, perhaps silently, “I’m glad that’s your job after all. I’m happy not being a fake product owner.”

Without a product owner’s authority (and title, and salary), I’m not willing to sign—or even make—a go/no-go decision. That decision is not my responsibility, but the responsibility of people with the authority to make it. If they ask for my recommendation, I’ll provide a list of problems that I know about, and any reasons to believe that there might be more problems lurking. Then I will politely recommend that they weigh these against the needs and objectives of everyone else involved—development, operations, support, sales, marketing, finance, and so forth.

So what if someone asks you to sign something? I am willing to sign an account of what I know about the status of the product, and what I’ve done to learn about it. That account can be fairly concise, but in expanded form, it will look like this:

I typically start with “I’ve tested the product, and I’ve found these problems in it.” I then provide a list of a few problems that I believe to be most important to inform management’s decisions. For the rest, I’ll provide a summary, including patterns or classes of general problems, or pointers to problem reports. My special focus is on the problems; as the newspaper reporters will tell you, “if it bleeds it leads”. I’ll welcome requests for more information. If there’s any doubt about the matter, I emphasize that the decision to ship or not rests with the person responsible for making the release decision.

(Some people claim “the whole team decides when to release and when not to”. That’s true when the whole team agrees, or when disagreements are tractable. When they’re not, in every business situation I’ve been in, there is a single person who is ultimately responsible for the release decision.)

If I haven’t found any problems—which is rare—I won’t sign anything claiming that there are no problems in the product. I’m willing to assert that I’m not aware of any problems. I cannot responsibly say that there are no problems, or that I’m capable of finding all problems. To say that there are no problems is only an inference; to say that I’m not aware of any problems is a fact.

Whether I’ve found problems or not, I’m willing to sign a statement like this: “I have covered these conditions to this degree, and I have not covered these other conditions.” The conditions include specific product factors and quality criteria, like those found in the Heuristic Test Strategy model, or others that are specific to the project’s context. This gives warrant to my statement that there are problems (or that I’m not aware of them), and identifies why management should be appropriately trusting and appropriately skeptical about my evaluation. For an informed release decision, management needs to know about things I haven’t covered, and my perception of the risk associated with not covering them.

Happy news about the product might be worth mentioning, but it takes second place to reporting the problems and risks. I want to make sure that any concerns I have are prominent and not buried in my report.

I’m also willing to sign a statement saying “Here are some of the things that helped me, and here are some of the things that didn’t help; things that slowed my testing down, made it more difficult, reduced the breadth and depth of coverage I was able to obtain.” Whether I sign such a statement or not, I want to make sure I’ve been heard. I also want to offer some ideas that address the obstacles, and note that with management help, maybe we can reduce or remove some of them so that I can provide more timely, more valuable coverage of the product. When I can do that, I can find deeper, or more subtle, or more intermittent, and possibly more dangerous bugs.

Of course, I don’t run the project. There may be business considerations that prevent management from helping me to address the obstacles. If I’ve been heard, I’ll play the hand I’ve been dealt; I’ll do my best to address any problems I’ve got, using any resources I can bring to bear. It’s my job to make management aware of any risks associated with not dealing with the obstacles—on paper or in a quick email, if I’m worried about accountability. After that, decisions on how to manage the project belong with management.

In other words: I’m prepared to sign off on a three-part testing story. As a tester, I’m prepared to accept responsibility for my story about the quality of the product, but the product does not have to adhere to my quality standards. I’ll sign off on a report, but not on a decision. The business doesn’t need my permission to ship.

Testing and Management Efficiency

Monday, February 5th, 2018

To a naïve manager who doesn’t understand testing very well, the visible manifestation of testing is a tester, sitting in front of a computer, banging on keys to operate a product, comparing output with a predicted result, and marking “pass” or “fail” on a spreadsheet.

Therefore, thinks the manager: one way to make testing dramatically more efficient and effective is to automate the testing. Substitute the human tester for a program on the same computer. Have the program bang on virtual keys to manipulate products and tools. Then algorithmically compare program output to a predicted result; and then mark “pass” or “fail” on an algorithmically generated report. (It doesn’t really help that many testers, managed by such managers, may believe in the same idea.)

To a naïve tester who doesn’t understand management very well, the visible manifestation of management is a manager, sitting in front of a computer, receiving messages from employees, banging on keys to scroll through documents and write a memo, checking the memo for spelling and grammar, attaching it to a message in Microsoft Outlook, and pressing “Send”.

Therefore, thinks the tester: one way to make management dramatically more efficient and effective is to automate the management. Install optical character recognition and voice recognition software onto the manager’s computer. Have a program feed problems, requests, and supporting documents into a spreadsheet full of business rules. Next, apply those rules, and feed the output into a memo-generating template that includes the words “approved” or “rejected”. Then have the program run the memo through a spelling and grammar checker, attach the memo to an Outlook message, and press Send. (In reality, even the most naive tester doesn’t believe that would lead to good management.)

Moral: some physical, visible, mechanical behaviours are included in each job, but is not the important part of either one. Both jobs require studying a situation, wrestling with uncertainty, comprehending business value (and threats to it), observing processes, analyzing risk, making decisions, performing actions, and interpreting outcomes.

Tools can help enormously with all of those activities, but for testing and management alike, fixation on visible behaviours misses the point. Obsession with mechanizing the visible things risks displacing the real work.

The End of Manual Testing

Wednesday, November 8th, 2017

Testers: when we speak of “manual testing”, we help to damage the craft.

That’s a strong statement, but it comes from years of experience in observing people thinking and speaking carelessly about testing. Damage arises when some people who don’t specialize in testing (and even some who do) become confused by the idea that some testing is “manual” and some testing is “automated”. They don’t realize that software development and the testing within it are design studio work, not factory work. Those people are dazzled by the speed and reliability of automation in manufacturing. Very quickly, they begin to fixate on the idea that testing can be automated. Manual bad; automated good.

Soon thereafter, testers who have strong critical thinking skills and who are good at finding problems that matter have a hard time finding jobs. Testers with only modest programming skills and questionable analytical skills get hired instead, and spend months writing programs that get the machine to press its own buttons. The goal becomes making the automated checks run smoothly, rather than finding problems that matter to people. Difficulties in getting the machine to operate the product take time away from interaction with and observation of the product. As a result, we get products that may or may not be thoroughly checked, but that have problems that diminish or even destroy value.

(Don’t believe me? Here’s an account of testing from the LinkedIn engineering blog, titled “How We Make Our UI Tests Stable“. It’s wonderful that LinkedIn’s UI tests (checks, really) are stable. Has anyone inside LinkedIn noticed that LinkedIn’s user interface is a hot, confusing, frustrating, unusable mess? That LinkedIn Groups have lately become well-nigh impossible to find? That LinkedIn rudely pops up a distracting screen after each time you’ve accepted a new invitation to connect, interrupting your flow, rather than waiting until you’ve finished accepting or rejecting invitations? That these problems dramatically reduce the desire of people to engage with LinkedIn and see the ads on it?)

Listen: there is no “manual testing”; there is testing. There are no “manual testers”; there are testers. Checking—an element of testing, a tactic of testing—can be automated, just as spell checking can be automated. A good editor uses the spelling checker, while carefully monitoring and systematically distrusting it. We do not call spell checking “automated editing”, nor do we speak of “manual editors” and “automated editors”. Editors, just “editors”, use tools.

All doctors use tools. Some specialists use or design very sophisticated tools. No one refers to those who don’t as “manual doctors”. No one speaks of “manual researchers”, “manual newspaper reporters”, “manual designers”, “manual programmers”, “manual managers”. They all do brain- and human-centred work, and they all use tools.

Here are seven kinds of testers. The developer tests as part of coding the product, and the good ones build testability into the product, too. The technical tester builds tools for herself or for others, uses tools, and in general thinks of her testing in terms of code and technology. The adminstrative tester focuses on tasks, agreements, communication, and getting the job done. The analytical tester develops models, considers statistics, creates diagrams, uses math, and applies these approaches to guide her exploration of the product. The social tester enlists the aid of other people (including developers) and helps organize them to cover the product with testing. The empathic tester immerses himself in the world of the product and the way people use it. The user expert comes at testing from the outside, typically as a supporting tester aiding responsible testers.

Every tester interacts with the product by various means, perhaps directly and indirectly, maybe at high levels or low levels, possibly naturalistically or artificially. Some testers are, justifiably, very enthusiastic about using tools. Some testers who specialize in applying and developing specialized tools could afford to develop more critical thinking and analytical skill. Correspondingly, some testers who focus on analysis or user experience or domain knowledge seem to be intimidated by technology. It might help everyone if they could become more familiar and more comfortable with tools.

Nonetheless, referring to any of the testing skill sets, mindsets, and approaches as “manual” spectacularly misses the mark, and suggests that we’re confused about the key body part for testing: it’s the brain, rather than the hands. Yet testers commonly refer to “manual testing” without so much as a funny look from anyone. Would a truly professional community play along, or would it do something to stop that?

On top of all this, the “manual tester” trope leads to banal, trivial, clickbait articles about whether “manual testing” has a future. I can tell you: “manual testing” has no future. It doesn’t have a past or a present, either. That’s because there is no manual testing. There is testing.

Instead of focusing on the skills that excellent testing requires, those silly articles provide shallow advice like “learn to program” or “learn Selenium”. (I wonder: are these articles being written by manual writers or automated writers?) Learning to program is a good thing, generally. Learning Selenium might be a good thing too, in context. Thank you for the suggestions. Let’s move on. How about we study how to model and analyze risk? More focus on systems thinking? How about more talk about describing more kinds of coverage than code coverage? What about other clever uses for tools, besides for automated checks?

(Some might reply “Well, wait a second. I use the term ‘manual testing’ in my context, and everybody in my group knows what I mean. I don’t have a problem with saying ‘manual testing’.” If it’s not a problem for you, I’m glad. I’m not addressing you, or your context. Note, however, that your reply is equivalent to saying “it works on my machine.”)

Our most important goal as testers, typically, is to learn about problems that threaten value in our products, so that our clients can deal with those problems effectively. Neither our testing clients nor people who use software divide the problems they experience into “manual bugs” and “automated bugs”. So let’s recognize and admire technical testers, testing toolsmiths and the other specialists in our craft. Let us not dismiss them as “manual testers”. Let’s put an end to “manual testing”.

RST Slack Channel

Sunday, October 8th, 2017

Over the last few months, we’ve been inviting people from the Rapid Software Testing class to a Slack channel. We’re now opening it up to RST alumni.

If you’ve taken RST in the past, you’re welcome to join. Click here (or email me at slack@developsense.com), let me know where and when you took the class, and with which instructor. I’ll reply with an invitation.

Dev*Ops

Wednesday, October 4th, 2017

A while ago, someone pointed out that Development and Operations should work together in order to fulfill the needs and goals of the business, and lo, the DevOps movement was born. On the face of it, that sounds pretty good… except when I wonder: how screwed up must things have got for that to sound like a radical, novel, innovative idea?

Once or twice, I’ve noticed people referring to DevTestOps, which seemed to be a somewhat desperate rearguard attempt to make sure that Testing doesn’t get forgotten in the quest to fulfill the needs and goals of the business. And today—not for the first time— I noticed a reference to DevSecOps, apparently suggesting that Security is another discipline that should also be working with other groups in order to fulfill the needs and goals of the business.

Wow! This is great! Soon everyone who is employed by a business will be working together to fulfill the needs and goals of the business! Excelsior!

So, in an attempt to advance this ground-breaking, forward-thinking, transformative concept, I hereby announce the founding of a new movement:

DevSecDesTestDocManSupSerFinSalMarHumCEOOps

Expect a number of conferences and LinkedIn groups about it real soon now, along with much discussion about how to shift it left and automate it.

(How about we all decide from the get-go that we’re all working together, collaboratively, using tools appropriately, supporting each other to fulfill the needs and goals of the business? How about we make that our default assumption?)

Deeper Testing (3): Testability

Friday, September 29th, 2017

Some testers tell me that they are overwhelmed at the end of “Agile” sprints. That’s a test result—a symptom of unsustainable pace. I’ve written about that in a post called “Testing Problems are Test Results“.

In Rapid Software Testing, we say that testing is evaluating a product by learning about it through exploration and experimentation, which includes to some degree: questioning, study, modeling, observation, inference, and plenty of other stuff—perhaps including the design and programming of automated checks, and analysis of the outcome after they’ve been run. Note that the running of these checks—the stuff that the machine does—does not constitute testing, just as the compiling of the product—also the stuff that the machine does—does not constitute programming.

If you agree with this definition of testing, when someone says “We don’t have enough time for testing,” that literally means “we don’t have enough time to evaluate our product.” In turn, that literally means “we don’t have time to learn about this thing that (presumably) we intend to release.” That sounds risky.

If you believe, as I do, that evaluating a product before you release it is usually a pretty good idea, then it would probably also be a good idea to make testing as fast and as easy as possible. That is the concept of testability.

Most people think of a testability quite reasonably in terms of visibility and controllability in the product. Typically visibility refers to log files, monitoring, and other ways of seeing what the product is up to; controllability usually refers to interfaces that allow for easy manipulation of the product, most often via scriptable application programming interfaces (APIs).

It’s a good thing to have logging and scriptable interfaces, testability isn’t entirely a property of the product. Testability is a set of relationships between the product, the team, the tester, and the context in which the product is being developed and maintained. Changes in any one of these can make a product more or less testable. In Rapid Software Testing, we refer to the set of these relationships as practical testability. This breaks down into five other subcategories that overlap and interact to some degree.

Epistemic testability. (Yes, it’s a ten-dollar word. Epistemology is the study of how we know what we know. Excellent testing requires us to study epistemology if we want to avoid being fooled about what we know.) As we’re building a product, there’s a risk gap, the difference between what we know and what we need to know. A key purpose of testing is to explore the risk gap, shining light on the things that we don’t know, and identifying places that are beyond the current extent of our knowledge. A product that we don’t know well in a test space that we don’t know much about tends to make testing harder or slower.

Value-related testability. It’s easier to test a product when we know something about how people might use it and how they might intend get value from it. That means understanding people’s goals and purposes, and how the product is designed to fulfill and support them. It means considering who matters&mash;not just end users or customers, but also anyone who might have a stake in the success of the product. It means learning about dimensions of quality that might be more important or not so important to them.

Intrinsic testability. It’s easier to test a product when it is designed to help us understand its behaviour and its state. When the parts of the product are built cleanly and simply, testing as we go, testing the assembled parts will be easier. When we have logging and visibility into the workings of the product, and when we have interfaces that allow us to control it using tools, we can induce more variation that helps to shake the bugs out.

Project-related testability. It’s easier to test when the project is organized to support evaluation, exploration, experimentation, and learning. Testing is faster and easier when testers have access to other team members, to information, to tools, and to help.

Subjective testability. The tester is at the centre of the relationships between the product, the project, and the testing mission. Testing will be faster, easier, and better when the tester’s skills—and testing skill on the team—are sufficient to deal with the situation at hand.

Each one of these dimensions of testability fans out into specific ideas for making a product faster and easier to test. You can find a set of ideas and guidewords in a paper called Heuristics of Software Testability.

On an Agile team, a key responsibility for the tester is to ask and advocate for testability, and to highlight things that make testing harder or slower. Testability doesn’t come automatically. Teams and their managers are often unaware of obstacles. Programmers may have created unit checks for the product, which may help to reduce certain kinds of coding and design errors. Still, those checks will tend to be focused on testing functions deep in the code. Testability for other quality criteria—usability, compatibility, performance, installabilty, or compatibility, to name only a few—may not get much attention without testers speaking up for them.

A product almost always gets bigger and more complex with every build. Testability helps us to keep the pace of that growth sustainable. A less testable product contributes to an unsustainable pace. Unsustainable pace ratchets up the risk of problems that threaten the value of the product, the project, and the business.

So here’s a message for the tester to keep in front of the team during that sprint planning meeting, during the sprint, and throughout the project:

Let’s remember testability. When testing is harder or slower, bugs have more time and more opportunity to stay hidden. The hidden bugs are harder to find than any bugs we’ve found so far—otherwise we would have found them already. Those bugs—deeper, less obvious, more subtle, more intermittent—may be far worse than any bugs we’ve found so far. Right now, testability is not as good as it could be. Is everyone okay with that?

Deeper Testing (1): Verify and Challenge

Thursday, March 16th, 2017

What does it mean to do deeper testing? In Rapid Software Testing, James Bach and I say:

Testing is deep to the degree that it has a probability of finding rare, subtle, or hidden problems that matter.

Deep testing requires substantial skill, effort, preparation, time, or tooling, and reliably and comprehensively fulfills its mission.

By contrast, shallow testing does not require much skill, effort, preparation, time, or tooling, and cannot reliably and comprehensively fulfill its mission.

Expressing ourselves precisely is a skill. Choosing and using words more carefully can sharpen the ways we think about things. In the next few posts, I’m going to offer some alternative ways of expressing the ideas we have, or interpreting the assignments we’ve been given. My goal is to provide some quick ways to achieve deeper, more powerful testing.

Many testers tell me that their role is to verify that the application does something specific. When we’re asked to that, it can be easy to fall asleep. We set things up, we walk through a procedure, we look for a specific output, and we see what we anticipated. Huzzah! The product works!

Yet that’s not exactly testing the product. It can easily slip into something little more than a demonstration—the kinds of things that you see in a product pitch or a TV commercial. The demonstration shows that the product can work, once, in some kind of controlled circumstance. To the degree that it’s testing, it’s pretty shallow testing. The product seems to work; that is, it appears to meet some requirement to some degree.

If you want bugs to survive, don’t look too hard for them! Show that the product can work. Don’t push it! Verify that you can get a correct result from a prescribed procedure. Don’t try to make the product expose its problems.

But if you want to discover the bugs, present a challenge to the product. Give it data at the extremes of what it should be able to handle, just a little beyond, and well beyond. Stress the product out; overfeed it, or starve it of something that it needs. See what happens when you give the product data that it should reject. Make it do things more complex than the “verification” instructions suggest. Configure the product (or misconfigure it) in a variety of ways to learn how it responds. Violate an explicitly stated requirement. Rename or delete a necessary file, and observe whether the system notices. Leave data out of mandatory fields. Repeat things that should only happen once. Start a process and then interrupt it. Imagine how someone might accidentally or maliciously misuse the product, and then act on that idea. While you’re at it, challenge your own models and ideas about the product and about how to test it.

We can never prove by experiment—by testing—that we’ve got a good product; when the product stands up to the challenge, we can only say that it’s not known to be bad. To test a product in any kind of serious way is to probe the extents and limits of what it can do; to expose it to variation; to perform experiments to show that the product can’t do something—or will do something that we didn’t want it to do. When the product doesn’t meet our challenges, we reveal problems, which is the first step towards getting them fixed.

So whenever you see, or hear, or write, or think “verify”, try replacing it with “challenge“.