Blog Posts from February, 2013

“Manual” and “Automated” Testing

Sunday, February 24th, 2013

The categories “manual testing” and “automated testing” (and their even less helpful byproducts, “manual tester” and “automated tester”) were arguably never meaningful, but they’ve definitely outlived their sell-by date. Can we please put them in the compost bin now? Thank you.

Here’s a long and excellent rant by pilot Patrick Smith, who for years has been trying to address a similar problem in the way people talk (and worse, think) about “manual” and “automated” in commercial aviation.

When it comes to software, there is no manual testing. My trusty Chambers dictionary says manual means “of the hand or hands”; “done, worked or used by the hand(s), as opposed to automatic, computer-operated, etc.”; “working with the hands”. It’s true that sometimes we use our hands to tap keys on a keyboard, but thinking of that as “manual testing” is no more helpful than thinking of using your hands on the steering wheel as “manual driving”. We don’t say that Itzhak Perlman, using his hands, is performing “manual music”, even though his hands play a far more important role in his music than our hands play in our testing. If you’re describing a complex, cognitive, value-laden activity, at least please focus on the brain.

We might choose to automate the exercise of some functions in a program, and then automatically compare the output of that program with some value obtained by another program or process. I’d call that a check. (So did McCracken (1957), and so did Alan Turing.) But whatever you call the automated activity, it’s not really the interesting part. It’s cool to know that the machine can perform a check very precisely, or buhzillions of checks really quickly. But the risk analysis, the design of the check, the programming of the check, the choices about what to observe and how to observe it, the critical interpretation of the result and other aspects of the outcome—those are the parts that actually matter, the parts that are actually testing. And those are exactly the parts that are not automated. A machine producing a bit is not doing the testing; the machine, by performing checks, is accelerating and extending our capacity to perform some action that happens as part of the testing that we humans do. The machinery is invaluable, but it’s important not to be dazzled by it. Instead, pay attention to the purpose that it helps us to fulfill, and to developing the skills required to use tools wisely and effectively. Otherwise, the outcome will be like giving a 17-year-old fan of Top Gear the keys to the Ferrari: it will not end well.

People often get excited when they develop or encounter something that helps them to extend their capabilities in some way. Marshall McLuhan used “media” to describe tools and technologies—anything that humans use to effect change—as extensions of man, things that enhance, extend, enable, accelerate or intensify what we do. He also emphasized that media are agnostic about our capabilities and our purposes. Tools not only accelerate what we do, they intensify what we are. The tool also changes the nature of the test that we’re performing, sometimes in ways that may escape our notice. Tools can extend or enhance or enable fabulous testing. But by accelerating some action, tools can enable us to do bad testing faster than ever, far worse than we could possibly do it without using the tool.

As Cem Kaner so perfectly puts it, “When we’re testing, we’re not talking about something you’re doing with your hands, we’re talking about something you do with your head.” Besides, as he and others have also pointed out, it’s exceedingly rare that we don’t use machinery and tools in “manual” testing. Even in review—when we’re not operating the product—we’re often looking at requirement documents or source code on a screen; using a keyboard and mouse as input mechanisms for our notes; using the computer to help us find and sift information; using software and hardware to communicate with other people; comparing ideas in our head with the output from a printer. Is this stuff manual? I prefer to think of it as sapient, something that by definition can’t be performed by a machine (or, using a word coined by our friend Pradeep Soundararajan, “brainual”), and assisted by tools. But if you want to call that stuff “manual” testing, how do you characterize the act of creating automated checks? I imagine you would call that writing, not typing. Through macros, computers can effectively type, and save us time in typing; it takes people to write. See the difference?

Several years ago, I used Excel to construct an automated oracle to model functional aspects in a teller workstation application, and identify which general ledger accounts should be credited or debited. Building the oracle took three weeks of solid analysis and programming work. As I was doing that work, I was learning about the system, performing tests, revealing problems, feeding back learning in the the development of my oracle. During this process, I also found lots of bugs. Sometimes my oracle helped me find a bug; sometimes tests that I performed interactively while developing the oracle helped me find a bug. Was I doing “automated” or “manual” testing?

I worked at another financial institution, where part of the task was to create tables of inputs and actions for functional integration checks, using FITnesse as the tool to help me select the functions, organize the data, and execute the checks. I identified risks. I designed checks to probe those risks. I entered those checks manually, via the keyboard. The tool executed the checks, but I triggered its execution “manually”, by clicking on a button. I reviewed the output by eye, but the tool helped by allowing me to see unexpected results easily. I revisited our assumptions frequently, considering what the checks appeared to be telling us and what they might be leaving out. When we realized a lapse in our check coverage, I’d enter new checks manually, by typing them in (although sometimes I laid them out in Excel first, and copied and pasted them in). So was I doing automated testing, or not?

Why is this a big deal to me? To me, there are two dominating reasons, and they complement each other.

When we think in terms of “automated testing”, we run the risk that we will narrow our notion of testing to automated checking, and focus our testing effort on the development and execution of automated checks. Checking is not a bad thing; it’s a good thing, and an important thing. But there are more aspects to the quality of a product than those that can be made subject to binary evaluation, which is what checks produce. I’ve had the experience of being fascinated by checking to the point of distraction from a broader and deeper analysis of the product I was working on, which meant that I missed some important problems (thankfully, other testers found those problems before they reached the customer).  Qualitative analysis of a product isn’t about bits; it’s about developing a story. Yet investigating qualitative aspects can be aided by tools, which brings us to the second point.

Talk of “manual testing” often slides into talk of “manual exploratory testing”, as though exploration doesn’t make use of tools, or as though tools could not used in an exploratory way. This slipping limits our ideas about exploration and our ideas about tools. Exploratory testing is not tool-free. When we think in terms of “manual testing”, we run the risk of underestimating the role that tools can play in our interactions with the product and in the testing effort. As a consequence, we may also ignore myriad ways in which automation can assist and amplify and accelerate and enable all kinds of testing tasks: collecting data, setting up the product or the test suite, reconfiguring the product, installing the product, monitoring its behaviour, logging, overwhelming the product, fuzzing, parsing reports, visualizing output, generating data, randomizing input, automating repetitive tasks that are not checks, preparing comparable algorithms, comparing versions—and this list isn’t even really scratching the surface. I’ve had experience of not pausing to think about how tools could help me, of neglecting to use them, and of using them unskillfully, and that compromised the quality of my work too.

To sum up:  “Manual testing” vs. “automated testing” is a false dichotomy, like “manual statistical analysis” vs. “automated statistical analysis”, or like “manual book editing” vs. “automated book editing”. In the sense that most people seem to use them, “manual” and “automated” refer to an input mechanism. People probably started saying “manual test” as a contrast to “automated test”, but just as testing isn’t manual, there’s no such thing as an automated test, either—except of the kind that we call a check, which still requires significant testing skill to design, prepare, interpret, and analyze. The check might be automated—performed by a machine—but the testing that surrounds the check is neither manual nor automated. The person who performs the testing is neither an “automated tester” nor a “manual tester”.  Tools can aid testing powerfully, and in all kinds of ways, but they can also lead to distortion or dysfunction, so practice and develop skill to use them effectively.

If you think that all this is a silly quibble, take a page from my colleague James Bach: you don’t talk about compiling and linking as “automated programming”, do you?—nor do you talk about writing code as “manual programming”, do you?