Blog Posts from December, 2013

Your User Is Not You

Tuesday, December 31st, 2013

The title of this post, “Your User Is Not You”, is cheerfully cribbed from a fellow named David Platt, who wrote a pointed and very funny book called Why Software Sucks…and What You Can Do About It. He also gave a terrific talk at SD West 2007, excerpts of which you can see here and here. Attending that talk was one of the highlights of that year for me.

Today some of us on Twitter were discussing Twitter’s “Favorite” feature. I realized that some people use Twitter favorites to indicate approval of tweets, and that others use it to save the tweet for later reading or review. Colleague Ben Simo (@QualityFrog) said “For me, Twitter favorites are like IE favorites: bookmarks to help me find things later.” George Dinwiddie (@gdinwiddie) said, “‘Favorite’ can also mean a positive acknowledgement of a tweet not felt worth retweeting.” Tracy Harms startled some of us by saying “I know it’s used sometimes to signal recognition, sometimes approval, sometimes dislike (my emphasis).” Later, Tracy added “Marketing often involves calling something by an inspiring name rather than a dryly accurate label.”

That’s true. And people respond both to inspiring names and to dryly accurate labels in their own ways. People not only observe or use relationships between things; they develop those relationships (that’s an important theme of the book Surfaces and Essences: Analogy as the Fuel and Fire of Thinking, by Douglas Hofstadter and Emmanuel Sander.)

All that is why David Platt’s talk and book were so significant. To me, the two most important points of his talk for testers were 1) “Know thy user, for he is not thee”; and 2) “People don’t want to use your software; they want to have used your software.” Some people may want to do something; others may simply want to get something done, which is subtly different, and which has important implications for testing. In order to get things done, people will use products in ways that their designers and developers did not intend or anticipate. Sometimes what we infer from observable behaviour is not what someone else intended to imply, or to do. This doesn’t apply only to end users, either. Even within a development shop, the function or object or library or API developed by one person or group for some purpose may get used by other people in a surprisingly different way, or for a strikingly different purpose.

That’s why, when you’re testing, it’s so important to learn about your users; about their purposes; about competitive products and their features; and about the importance of parallels, analogies, and metaphors. That’s why it’s important to create and maintain good relationships with customers and with vigilant, helpful technical support people who talk with your customers. That’s why it’s important to develop, as Jerry Weinberg says, “a suspicious nature and a lively imagination”: our big challenge is to test not only for reliability but also for adaptability; not only for what happens, but what could happen; not only for what people are expected to do, but what they might do. Learning about those things—and reporting on them—can help our clients to decide whether the product they’ve got is the product they want.


A related post: Why Would a User Do THAT?

Counting the Wagons

Monday, December 30th, 2013

A member of Linked In asks if “a test case can have multiple scenarios”. The question and the comments reinforce, for me, just how unhelpful the notion of the “test case” is.

Since I was a tiny kid, I’ve watched trains go by—waiting at level crossings, dashing to the window of my Grade Three classroom, or being dragged by my mother’s grandchildren to the balcony of her apartment, perched above a major train line that goes right through the centre of Toronto. I’ve always counted the cars (or wagons, to save us some confusion later on). As a kid, it was fun to see how long the train was (were more than a hundred wagons?!). As a parent, it was a way to get the kids to practice counting while waiting for the train to pass and the crossing gates to lift.

train

Often the wagons are flatbeds, loaded with shipping containers or the trailers from trucks. Others are enclosed, but when I look through the screening, they seem to be carrying other vehicles—automobiles or pickup trucks. Some of the wagons are traditional boxcars. Other wagons are designed to carry liquids or gases, or grain, or gravel. Sometimes I imagine that I could learn something about the economy or the transportation business if I knew what the trains were actually carrying. But in reality, after I’ve counted them, I don’t know anything significant about the contents or their value. I know a number, but I don’t know the story. That’s important when a single car could have explosive implications, as in another memory from my youth.

A test case is like a railway wagon. It’s a container for other things, some of which have important implications and some of which don’t, some of which may be valuable, and some of which may be other containers. Like railway wagons, the contents—the cargo, and not the containers—are the really interesting and important parts. And like railway wagons, you can’t tell much about the contents without more information. Indeed, most of the time, you can’t tell from the outside whether you’re looking at something full, empty, or in between; something valuable or nothing at all; something ordinary and mundane, or something complex, expensive, or explosive. You can surely count the wagons—a kid can do that—but what do you know about the train and what it’s carrying?

To me, a test case is “a question that someone would like to ask (and presumably answer) about a program”. There’s nothing wrong with using “test case” as shorthand for the expression in quotes. We risk trouble, though, when we start to forget some important things.

  • Apparently simple questions may contain or infer multiple, complex, context-dependent questions.
  • Questions may have more outcomes than binary, yes-or-no, pass-or-fail, green-or-red answers. Simple questions can lead to complex answers with complex implications—not just a bit, but a story.
  • Both questions and answers can have multiple interpretations.
  • Different people will value different questions and answers in different ways.
  • For any given question, there may be many different ways to obtain an answer.
  • Answers can have multiple nuances and explanations.
  • Given a set of possible answers, many people will choose to provide a pleasant answer over an unpleasant one, especially when someone is under pressure.
  • The number of questions (or answers) we have tells us nothing about their relevance or value.
  • Most importantly: excellent testing of a product means asking questions that prompt discovery, rather than answering questions that confirm what we believe or hope.

Testing is an investigation in which we learn about the product we’ve got, so that our clients can make decisions about whether it’s the product they want. Other investigative disciplines don’t model things in terms of “cases”. Newspaper reporters don’t frame their questions in terms of “story cases”. Historians don’t write “history cases”. Even the most reductionist scientists talk about experiments, not “experiment cases”.

Why the fascination with modeling testing in terms of test cases? I suspect it’s because people have a hard time describing testing work qualitatively, as the complex cognitive activity that it is. These are often people whose minds are blown when we try to establish a distinction between testing and checking. Treating testing in terms of test cases, piecework, units of production, simplifies things for those who are disinclined to confront the complexity, and who prefer to think of testing as checking at the end of an assembly line, rather than as an ongoing, adaptive investigation. Test cases are easy to count, which in turn makes it easy to express testing work in a quantitative way. But as with trains, fixating on the containers doesn’t tell you anything about what’s in them, or about anything else that might be going on.


As an alternative to thinking in terms of test cases, try thinking in terms of coverage. Here are links to some further reading:

  • Got You Covered: Excellent testing starts by questioning the mission. So, the first step when we are seeking to evaluate or enhance the quality of our test coverage is to determine for whom we’re determining coverage, and why.
  • Cover or Discover: Excellent testing isn’t just about covering the “map”—it’s also about exploring the territory, which is the process by which we discover things that the map doesn’t cover.
  • A Map By Any Other Name: A mapping illustrates a relationship between two things. In testing, a map might look like a road map, but it might also look like a list, a chart, a table, or a pile of stories. We can use any of these to help us think about test coverage.
  • What Counts“, an article that I wrote for Better Software magazine, on problems with counting things.
  • Braiding the Stories” and “Delivering the News“, two blog posts on describing testing qualitatively.
  • My colleague James Bach has a presentation on the case against test cases.
  • Apropos of the reference to “scenarios” in the original thread, Cem Kaner has at least two valuable discussions of scenario testing, as tutorial notes and as an article.