Blog Posts from June, 2018

Test Cases and Coverage

Thursday, June 28th, 2018

A tester recently asked about creating an in-house testing process, wanting to know how to start writing test cases so that testing effort could be estimated. My reply—which could just as easily apply to a developer or business analysts in a testing role—went something like this:

Test cases are not testing!  While that’s true, just saying so probably won’t help you very much, so let me offer an alternative to thinking about your assignment in terms of test cases.

At the beginning of the project, you don’t yet know how to test it. To develop a relevant test strategy, you must learn the product. There may be many ways for you to learn about the product—by attending meetings, talking to people, reviewing project plans, studying designs or sketches. You may even have an opportunity to work with the product directly; with a previous version; with a mockup or prototype; or with a part of the product.

Without a good deal of that learning, you cannot know what test cases should describe or contain—so instead of the typical fine granularity of test cases, consider the broader notion of coverage areas.

As you learn about a product or feature, create a coverage outline based on product factors. (A product factor is something that could be examined in a test, or something that could affect the outcome of a test.)  Your outline may take one or more of many forms—a hierarchical list, a table, an annotated diagram; these days we like mind maps.  The outline identifies areas of the product and its relationship to its context that can be covered with testing activity. In general, these areas will tend to fall into categories like structure, functions, data, interfaces, platforms, operations, and time.

You might want to note any specific factors or conditions that you believe must be checked—but since you’re at the beginning of the project, also note that no one can be aware of all of them yet.

While you’re at it, begin advocating for testabilty; things that can make testing faster and easier. A product tends to be faster and easier to test when developers provide controllability through scriptable interfaces (APIs); visibility through log files; and code that they have already tested for basic problems and coding errors. When programmers do more of their own testing (including automated checks), testers won’t have to investigate and report on shallow bugs—which leaves more time for deep testing.

Your coverage outline goes hand in hand with a risk list—in which you identify technological and business risks that will influence your strategies for covering the test space. As development progresses, we learn about potential problems that threaten the value of the product. It’s a special responsibility of someone in the testing role to explore ideas about problems and where they might be lurking; to perform experiments to identify problems; and to determine the nature of those problems so that developers and managers can make informed decisions on what to do about them.

Consider the kinds of activities that testers perform: making contact with the team; learning about the product; analyzing how to test it; preparing equipment and tools; developing ideas for deep coverage; doing deep testing; and winding down the testing and helping prepare the product for deployment.  Then, instead of estimating by test cases, or by hours or days, consider estimating by sessions.

Sessions are bursts of nominally uninterrupted testing work, guided by a mission statement or charter. The time span of a session is intentionally imprecise, and we say “nominally uninterrupted” because, in the real world, it’s unusual for work not to get interrupted.  When planning a session, we base its charter on the idea that it will take around 90 minutes.  In reality, a session may last from 45 minutes to two and a half hours.  That’s because we can identify what we’d like to happen in a session, but our knowledge of what will happen is uncertain.

For instance, the bugs that you encounter during a session will affect how much you’re able to cover during the session. When bugs require investigation and reporting, coverage suffers. So, after the session, the charter may change to reflect what actually happened.  The difference between what you hope to do and what you’ll be able to do can’t be known very well in advance—so guess how many sessions you’ll need to cover the stuff that you know about.  Then double your guess.

What will happen over the course of a collection of sessions is uncertain too. You’ll need to allocate plenty of time to account for setting up for each session; dealing with obstacles; debriefing; debugging with developers; planning meetings; breaks and lunch; and so forth. As with the specifics of what you’re going to cover, that stuff is uncertain too.  However, it’s reasonable to assume, given the non-session work, a tester will have time for two or three sessions at most in a day.

In the early stages, you’re going to have to guess about a lot of this. That’s normal; testing is about exploration, discovery, investigation, and learning.  Those things aren’t calculable except in the most general way, and work tends to expand or shrink to fit the time available for it. Recognize that little of your early planning will play out as anticipated.

As you go through loops of learning and testing, your coverage outline and risk list will evolve. Commit to providing daily reports on what is being learned; and on the gap between what has been covered and what has not yet been covered—product coverage and risk coverage. Daily reports will keep managers alert to problems that need their attention—problems that might threaten the value of the product or the success of the project.  Managers need to know about those, not about test case counts.

As a key element in your reporting, provide notes on testability and its absence—anything that makes the product easier or harder to test. Reduced testability means testing takes longer or reduces coverage.  Reduced coverage means bugs will survive undetected for longer—maybe into production.

Every day, as you learn about the product and update your models of it, you will be informing your clients on what testing has been done, and what testing could be done.  In the end, your clients will make the decision about when they have sufficient information about coverage and risk to ship the product.  It’s your job to keep them up to date and well-informed; it’s their job to decide whether the product they’ve got is the product they want.

All that said, you might not find it necessary to estimate the amount of testing that the project needs. Since both you and the client already have an idea of what the schedule and budget will be for this project anyway (based on past experience and other factors), your estimate may already be done for you. If not, and you’re asked to provide and estimate, don’t provide one; provide three: a reasonable estimate; a safer estimate; and a riskier estimate.

I anticipate someone will respond “all this sounds harder than preparing a list of test cases.” I don’t believe it is—I’d say it’s easier, in fact. A coverage outline is certainly easier to maintain than a library of test cases.  Working from an outline acknowledges the fact that we’ll learn and refine our plans as we go, where the test case approach turns us into ostriches when it comes to anticipating change.

Some people will respond “But managers want test cases!” Yes, some do; and kids want candy instead of real food, and addicts want illegal drugs, too. We are not obliged to do harmful things, and fixation on test cases puts more emphasis on documentation and procedures than on actual testing. I believe most managers want test cases because they’re used to them, and because they’re not aware of alternatives. Testers provide test cases because managers keep asking about them. I’d offer real testing work, continuous collaboration, refinement of the tasks at hand—and solid information about coverage, problems, and risks.

Related reading:

No Test Cases Required (PowerPoint presentation)
Questioning Test Cases
Testers Learn, but Test Cases Don’t Teach

Rapid Software Testing is designed for managers, developers, testers, and teams who want help with making testing faster, easier, cheaper, and above all, effective. Read about RST classes here, and consider the upcoming public class taught by both James Bach and Michael Bolton.

Which Test Cases Should I Automate?

Friday, June 15th, 2018

When someone asks “I have a big suite of manual tests; which tests (or worse, which test cases) should I automate?”, I often worry about several things.

The first thing is that focusing on test cases is often a pretty lousy way to think about testing.  That’s because test cases are often cast in terms of following an explicit procedure in order to observe a specific result.  At best, this confirms that the product can work if someone follows that procedure, and it also assumes that any result unobserved in the course of that procedure and after it is unimportant.

The trouble is that there are potentially infinite variations on the procedure, and many factors might make a difference in a test or in its outcome. Will people use the product in only one way? Will this specific data expose a problem?  Might other data expose a problem that this data does not?  Will a bug appear every time we follow this procedure? The test case often actively suppresses discoveryTest cases are not testing , and bugs don’t follow the test cases.

Second: testing is neither manual nor automated. A test cannot be automated.  Elements of the procedure within the test (in particular, checks for specific facts) can be automated.  But your test is not just the machine performing virtual key presses or comparing an output to some reference.

Your test is a process of activity and reasoning:  analyzing risk; designing an experiment; performing the experiment; observing what happens before, during, and after the experiment; interpreting the results; and preparing and a relevant report.  This depends on your human intentions, your mindset, and your skill set.  Tools can help every activity along the way, but the test is something you do, not something the machine does.

Third:  lots of existing test cases are shallow, pointless, out of date, ponderous, inefficient, cryptic, and unmotivated by risk.  Often they are focused on the user interface, a level of the product that is often quite unfriendly to tools. Because the test cases exist, they are often pointlessly repeated, long after they have lost any power to find a bug.  Why execute pointless test cases more quickly?

It might be a much better idea to create new automated checks that are focused on specific factors of the product, especially at low levels, with the goal of providing quick feedback to the developer.  It might be a good idea to prepare those checks as a collaboration between the developer and the tester (or between two developers, one of whom is in the builder’s role, with the other taking on a testing role). It might be a good idea to develop those checks as part of the process of developing some code. And it might be a really good idea to think about tools in a way that goes far beyond faster execution of a test script.

So I encourage people to reframe the question.  Instead of thinking what (existing) test cases should I automate? try thinking:

  • What reason do we have for preserving these test cases at all? If we’re going to use tools effectively, why not design checks with tools in mind from the outset?  What tool-assisted experiments could we design to help us learn about the product and discover problems in it?
  • What parts of a given experiment could tools accelerate, extend, enhance, enable, or intensify?
  • What do we want to cover? What product factors could we check? (A product factor is something that can be examined during a test, or that might influence the outcome of a test.)
  • To what product factors (like data, or sequences, or platforms,…) could we apply tools to help us to induce variation into our testing?
  • How could tools help us to generate representative, valid data and exceptional or pathological data?
  • In experiments that we might perform on the product, how might tools help us make observations and recognize facts that might otherwise escape our notice?  How can tools make invisible things visible?
  • How can tools help us to generate lots of outcomes that we can analyze to find interesting patterns in the data?  How can tools help us to visualize those outcomes?
  • How could the tool help us to stress out the product; overwhelm it; perturb it; deprive it of things that it needs?
  • How can developers make various parts of the system more amenable to being operated, observed, and evaluated with the help of tools?
  • How might tools induce blindness to problems that we might be able to see without them?
  • What experiments could we perform on the product that we could not perform at all without help from tools?

Remember:  if your approach to testing is responsible, clever, efficient, and focused on discovering problems that matter, then tools will help you to find problems that matter in an efficient, clever, and responsible way.  If you try to “automate” bad testing, you’ll find yourself doing bad testing faster and worse than you’ve ever done it before.

More reading:
A Context-Driven Approach to Automation in Testing
Manual and Automated Testing
The End of Manual Testing

Learn about Rapid Software Testing

Sign up for the RST class, taught by both James Bach and Michael Bolton, in Seattle September 26-28, 2018.  It’s not just for testers!  Anyone who wants to get better at testing is welcome and encouraged to attend—developers, managers, designers, ops people, tech support folk…