DevelopsenseLogo

Test Cases and Coverage

A tester recently asked about creating an in-house testing process, wanting to know how to start writing test cases so that testing effort could be estimated. My reply—which could just as easily apply to a developer or business analysts in a testing role—went something like this:

Test cases are not testing!  While that’s true, just saying so probably won’t help you very much, so let me offer an alternative to thinking about your assignment in terms of test cases.

At the beginning of the project, you don’t yet know how to test it. To develop a relevant test strategy, you must learn the product. There may be many ways for you to learn about the product—by attending meetings, talking to people, reviewing project plans, studying designs or sketches. You may even have an opportunity to work with the product directly; with a previous version; with a mockup or prototype; or with a part of the product.

Without a good deal of that learning, you cannot know what test cases should describe or contain—so instead of the typical fine granularity of test cases, consider the broader notion of coverage areas.

As you learn about a product or feature, create a coverage outline based on product factors. (A product factor is something that could be examined in a test, or something that could affect the outcome of a test.)  Your outline may take one or more of many forms—a hierarchical list, a table, an annotated diagram; these days we like mind maps.  The outline identifies areas of the product and its relationship to its context that can be covered with testing activity. In general, these areas will tend to fall into categories like structure, functions, data, interfaces, platforms, operations, and time.

You might want to note any specific factors or conditions that you believe must be checked—but since you’re at the beginning of the project, also note that no one can be aware of all of them yet.

While you’re at it, begin advocating for testabilty; things that can make testing faster and easier. A product tends to be faster and easier to test when developers provide controllability through scriptable interfaces (APIs); visibility through log files; and code that they have already tested for basic problems and coding errors. When programmers do more of their own testing (including automated checks), testers won’t have to investigate and report on shallow bugs—which leaves more time for deep testing.

Your coverage outline goes hand in hand with a risk list—in which you identify technological and business risks that will influence your strategies for covering the test space. As development progresses, we learn about potential problems that threaten the value of the product. It’s a special responsibility of someone in the testing role to explore ideas about problems and where they might be lurking; to perform experiments to identify problems; and to determine the nature of those problems so that developers and managers can make informed decisions on what to do about them.

Consider the kinds of activities that testers perform: making contact with the team; learning about the product; analyzing how to test it; preparing equipment and tools; developing ideas for deep coverage; doing deep testing; and winding down the testing and helping prepare the product for deployment.  Then, instead of estimating by test cases, or by hours or days, consider estimating by sessions.

Sessions are bursts of nominally uninterrupted testing work, guided by a mission statement or charter. The time span of a session is intentionally imprecise, and we say “nominally uninterrupted” because, in the real world, it’s unusual for work not to get interrupted.  When planning a session, we base its charter on the idea that it will take around 90 minutes.  In reality, a session may last from 45 minutes to two and a half hours.  That’s because we can identify what we’d like to happen in a session, but our knowledge of what will happen is uncertain.

For instance, the bugs that you encounter during a session will affect how much you’re able to cover during the session. When bugs require investigation and reporting, coverage suffers. So, after the session, the charter may change to reflect what actually happened.  The difference between what you hope to do and what you’ll be able to do can’t be known very well in advance—so guess how many sessions you’ll need to cover the stuff that you know about.  Then double your guess.

What will happen over the course of a collection of sessions is uncertain too. You’ll need to allocate plenty of time to account for setting up for each session; dealing with obstacles; debriefing; debugging with developers; planning meetings; breaks and lunch; and so forth. As with the specifics of what you’re going to cover, that stuff is uncertain too.  However, it’s reasonable to assume, given the non-session work, a tester will have time for two or three sessions at most in a day.

In the early stages, you’re going to have to guess about a lot of this. That’s normal; testing is about exploration, discovery, investigation, and learning.  Those things aren’t calculable except in the most general way, and work tends to expand or shrink to fit the time available for it. Recognize that little of your early planning will play out as anticipated.

As you go through loops of learning and testing, your coverage outline and risk list will evolve. Commit to providing daily reports on what is being learned; and on the gap between what has been covered and what has not yet been covered—product coverage and risk coverage. Daily reports will keep managers alert to problems that need their attention—problems that might threaten the value of the product or the success of the project.  Managers need to know about those, not about test case counts.

As a key element in your reporting, provide notes on testability and its absence—anything that makes the product easier or harder to test. Reduced testability means testing takes longer or reduces coverage.  Reduced coverage means bugs will survive undetected for longer—maybe into production.

Every day, as you learn about the product and update your models of it, you will be informing your clients on what testing has been done, and what testing could be done.  In the end, your clients will make the decision about when they have sufficient information about coverage and risk to ship the product.  It’s your job to keep them up to date and well-informed; it’s their job to decide whether the product they’ve got is the product they want.

All that said, you might not find it necessary to estimate the amount of testing that the project needs. Since both you and the client already have an idea of what the schedule and budget will be for this project anyway (based on past experience and other factors), your estimate may already be done for you. If not, and you’re asked to provide and estimate, don’t provide one; provide three: a reasonable estimate; a safer estimate; and a riskier estimate.

I anticipate someone will respond “all this sounds harder than preparing a list of test cases.” I don’t believe it is—I’d say it’s easier, in fact. A coverage outline is certainly easier to maintain than a library of test cases.  Working from an outline acknowledges the fact that we’ll learn and refine our plans as we go, where the test case approach turns us into ostriches when it comes to anticipating change.

Some people will respond “But managers want test cases!” Yes, some do; and kids want candy instead of real food, and addicts want illegal drugs, too. We are not obliged to do harmful things, and fixation on test cases puts more emphasis on documentation and procedures than on actual testing. I believe most managers want test cases because they’re used to them, and because they’re not aware of alternatives. Testers provide test cases because managers keep asking about them. I’d offer real testing work, continuous collaboration, refinement of the tasks at hand—and solid information about coverage, problems, and risks.

Related reading:

No Test Cases Required (PowerPoint presentation)
Questioning Test Cases
Testers Learn, but Test Cases Don’t Teach

2 replies to “Test Cases and Coverage”

  1. “Some people will respond “But managers want test cases!” Yes, some do; and kids want candy instead of real food, and addicts want illegal drugs, too.”
    Very funny! Love this post, thanks for your interesting article.

    Reply

Leave a Comment