DevelopSense Blog

Return to Ellis Island

michael.a.bolton@gmail.com (Michael Bolton http://www.developsense.com) — Tue, 23 Feb 2010 15:44:00 +0000

Dave Nicollette responds to my post on the Ellis Island bug. I appreciate his continuing the conversation that started in the comments to my post.

Dave says, "In describing a 'new' category of software defect he calls Ellis Island bugs...".

I want to make it clear: there is nothing new about Ellis Island bugs, except the name. They've been with us forever, since before there were computers, even.

He goes on to say "Using the typical behavior-driven approach that is popular today, one of the very first things I would think to write (thinking as a developer, not as a tester) is an example that expresses the desired behavior of the code when the input values are illogical. Protection against Ellis Island bugs is baked in to contemporary software development technique."

I'm glad Dave does that. I'm glad his team does that. I'm glad that it's baked in to contemporary software development technique. That's a good thing.

First, there's no evidence to suggest that excellent coding practices are universal, and plenty of evidence to suggest that they aren't. Second, the Ellis Island problem is not a problem that you introduce in your own code. It's a class of problem that you have to discover. As Dave rightly points out,

"...only way to catch this type of defect is by exploring the behavior of the code after the fact. Typical boundary-condition testing will miss some Ellis Island situations because developers will not understand what the boundaries are supposed to be."

The issue is not that "developers" will not understand what the boundaries are supposed to be. (I think Dave means "programmers" here, but that's okay, because other developers, including testers won't understand what the boundaries are supposed to be either.) People in general will not understand what the boundaries are supposed to be without testing and interacting with the built product. And even then, people will understand only to the extent that they have the time and resources to test.

Dave seems to have locked onto the triangle program as an example of a "badly developed program". Sure it's a badly developed program. I could do better than that, and so could Dave. Part of the point of our exercise is that if the testers looked at the source code (which we supply, quietly, along with the program), they'd be more likely to find that kind of bug. Indeed, when programmers are in the class and have the initiative to look at the source, they often spot that problem, and that provides an important lesson for the testers: it might be a really good idea to learn to read code.

Yet testing isn't just about questioning and evaluating the code that we write, because the code that we write is Well Tested and Good and Pure. We don't write badly developed programs. That's a thing of the past. Modern development methods make sure that problem never happens. The trouble is that APIs and libraries and operating systems and hardware ROMs weren't written by our ideal team. They were written by other teams, whose minds and development practices and testing processes we do not, cannot, know. How do we know that the code that we're calling isn't badly developed code? We don't know, and so we have to test.

I think we'd agree that Ruby, in general, is much better developed software than the triangle program, so let's look at that instead.

The Pickaxe says of the String::to_i() method: "If there is not a valid number at the start of str, 0 is returned. The method never raises an exception." That's cool. Except that I see two things that are suprising.

The first is that to_i returns zero, instead of an exception. That is, it returns a value (quite probably the wrong value) in exactly the same data type as the calling function would expect. That leaves the door wide open for misinterpretation by someone who hasn't tested the function seeking that kind of problem. We thought we had done that, and we were mistaken. Our tests were revealing accurately that invalid data of a certain kind was being rejected appropriately, but we weren't yet sensitized to a problem that was revealed only by later tests.

The second surprising thing is that the documentation is flatly wrong: to_i absolutely does throw exceptions when you hand it a parameter outside the range 2 through 36. We discovered that through testing too. That's interesting. I'd far rather it threw an exception on a number that it can't parse properly, so that I could more easily detect that situation and handle it more in the way that I'd like.

Well, after a bunch of testing by students and experts alike, we finally surprised ourselves with some data and a condition that revealed the problem. We thought that we had tested really well, and we found out that we hadn't caught everything. So now I have to write some code that checks the string and the return value more carefully than Ruby itself does. That's okay. No problem. Now... that's one method in one class of all of Ruby. What other surprises lurk?

(Here's one. When I copied the passage in bold above from my PDF copy of the Pickaxe, I got more than I bargained for: in addition to the text that I copied, I got this: "Report erratum Prepared exclusively for Michael Bolton". Should I have been surprised by that or not?)

Whatever problem we anticipate, we can insert code to check for that problem. Good. Whatever problem we discover, we can insert code to check for that problem too. That's great. In fact, we check for all the problems that our code could possibly run into. Or rather we think we do, and we don't know when we're not doing it. To address that problem, we've got a team around us who provides us with lots of test ideas, and pairs and reviews and exercises the code that we write, and we all do that stuff really well.

The problem comes with the fact that when we're writing software, we're dealing with far more than just the software we write. That other software is typically a black box to us. It often comes to us documented poorly and tested worse. It does things that we don't know about, that we can't know about. It may do things that its developers considered reasonable but that we would consider surprising. Having been surprised, we might also consider it reasonable... but we'd consider it surprising first.

Let me give you two more Ellis Island examples. Many years ago, I was involved with supporting (and later program managing and maintaining) a product called DESQview. Once we had a fascinating problem that we heard about from customers. On a particular brand of video card (from a company called "Ahead"), typing DV wouldn't start DESQview and give you all that multitasking goodness. Instead, it would cause the letters VD to appear in the upper left corner of the display, and then hang the system. We called the manufacturer of that card—headquartered in Germany—, and got one in. We tested it, and couldn't reproduce the problem. Yet customers kept calling in with the problem. At one point, I got a call from a customer who happened to be a systems integrator, and he had a card to spare. He shipped it to us.

The first Ellis Island surprise was that this card, also called "Ahead" was from a Taiwanese company, not a German one. The second surprise was that, at the beginning of a particular INT 10h call, the card saved the contents of the CPU registers, and restored them at the end of that call. The Ellis Island issue here was that the BX register was not returned in its original state, but set to 0 instead. After the fact, after the discovery, the programmer developed a terminate-and-stay-resident program to save and restore the registers, and later folded that code into DESQview itself to special-case that card.

Now: our programmers were fantastic. They did a lot of the Agile stuff before Agile was named; they paired, they tested, they reviewed, they investigated. This problem had nothing to do with the quality of the code that they had written. It had everything to do with the fact that you'd expect someone using the processor not to muck with what was already there, combined with the fact that in our test lab we didn't have every video card on the planet.

The oddest thing about Dave's post is that he interprets my description of the Ellis Island problem as an argument "to support status quo role segregation." Whaa...? This has nothing to do with role segregation. Nothing. At one point, I say "the programmer's knowledge is, at best, is a different set compared to what empirical testing can reveal." That's true in any situation, be it a solo shop, a traditional shop, or an Agile shop. It's true of anyone's understanding of any situation. There's always more to know than we think there is, and there's always another interpretation that one could take, rightly or wrongly. Let me give you an example of that:

When I say "the programmer's knowledge is, at best, is a different set compared to what empirical testing can reveal," there is nothing in that sentence, nor in the rest of the post, to suggest that the programmers shouldn't explore, or that testers should be the only ones to explore. Dave simply made that part up. My post says one thing, mostly on epistemology, that we don't know what we don't know. From my post, Dave takes another interpretation about organizational dynamics that is completely orthogonal to my point. Which, in fact, is an Ellis Island kind of problem on its own.

The Ellis Island Bug

michael.a.bolton@gmail.com (Michael Bolton http://www.developsense.com) — Wed, 10 Feb 2010 06:46:00 +0000

A couple of years ago, I developed a version of a well-known reasoning exercise. It's a simple exercise, and I implemented it as a really simple computer program. I described it to James Bach, and suggested that we put it in our Rapid Software Testing class.

James was skeptical. He didn't figure from my description that the exercise would be interesting enough. I put in a couple of little traps, and tried it a few times with colleagues and other unsuspecting victims, sometimes in person, sometimes over the Web. Then I tried the actual exercise on James, using the program. He helpfully stepped into one of the traps. Thus emboldened, I started using the exercise in classes. Eventually James found an occasion to start using it too. He watched students dealing with it, had some epiphanies, tried some experiments. At one point, he sat down with his brother Jon and they tested the program aggressively, and revealed a ton of new information about it—many of which I hadn't known myself. And I wrote the thing.

Experiential exercises are like peeling an onion; beneath everything we see on the surface, there's another layer that we can learn about. Today we made a discovery; we found a bug as we transpected on the exercise, and James put a name on it.

We call it an Ellis Island bug. Ellis Island bugs are data conversion bugs, in which a program silently converts an input value into a different value. They're named for the tendency of customs officials at Ellis Island, a little way back in history, to rename immigrants unilaterally with names that were relatively easy to spell. With an Ellis Island bug, you could reasonably expect an error on a certain input. Instead you get the program's best guess at what you "really meant".

There are lots of examples of this. We have an implementation of the famous triangle program, written many years ago in Delphi. The program takes three integers as input, with each number representing the length of a side of a triangle. Then the program reports on whether the triangle is scalene, isoceles, or equilateral. Here's the line that takes the input:

function checksides (a, b, c : shortint) : string

Here, no matter what numeric value you submit, the Delphi libraries will return that number as a signed integer between -128 and 127. This leads to all kinds of amusing results: a side of length greater than 127 will invisibly be converted to a negative number, causing the program to report "not a triangle" until the number is 256 or greater; and entries like 300, 300, 44 will be interpreted as an equilateral triangle.

Ah, you say, but no one uses Delphi any more. So how about C? We've been advised forever not to trust input formatting strings, and to parse them ourselves. How about Ruby?

Ruby's String object supplies a to_i method, which converts a string to its integer representation. Here's what the Pickaxe says about that:

to_i str.to_i( base=10 ) → int

Returns the result of interpreting leading characters in str as an integer base base (2 to 36). Given a base of zero, to_i looks for leading 0, 0b, 0o, 0d, or 0x and sets the base accordingly. Leading spaces are ignored, and leading plus or minus signs are honored. Extraneous characters past the end of a valid number are ignored. If there is not a valid number at the start of str, 0 is returned. The method never raises an exception.

We discovered a bunch of things today as we experimented with our program. The most significant thing was the last two sentences: an invalid number is silently converted to zero, and no exception is raised!

We found the problem because we thought we were seeing a different one. Our program parses a string for three numbers. Depending upon the test that we ran, it appeared as though multiple signs were being accepted (+--+++--), but that only the first sign was being honoured. Or that only certain terms in the string tolerated multiple signs. Or that you could use multiple signs once in a string—no, twice. What the hell? All our confusion vanished when we put in some debug statements and saw invalid numbers being converted to 0, a kind of guess as to what Ruby thought you meant.

This is by design in Ruby, so some would say it's not a bug. Yet it leaves Ruby programs spectacularly vulnerable to bugs wherein the programmer isn't aware of the behaviour of the language. I knew about to_i's ability to accept a parameter for a number base (someone showed it to me ages ago), but I didn't know about the conversion-to-zero error handling. I would have expected an exception, but it doesn't do that. It just acts like an old-fashioned customs agent: "S-C-H-U-M-A-C... What did you say? Schumacher? You mean Shoemaker, right? Let's just make that Shoemaker. Youse'll like that better here, trust me."

We also discovered that the method is incorrectly documented: to_i does raise an exception if you pass it an invalid number base—37, for example.

There are many more stories to tell about this program—in particular, how the programmer's knowledge is, at best, is a different set compared to what empirical testing can reveal. Many of the things we've discovered about this trivial program could not have been caught by code review; many of them aren't documented or are poorly documented both in the program and in the Ruby literature. We couldn't look them up, and in many cases we couldn't have anticipated them if they hadn't emerged from testing.

There are other examples of Ellis Island bugs. A correspondent, Brent Lavelle, reports that he's seen a bug in which 50,00 gets converted to 5000, even if the user is from France or Germany (in those countries, a comma rather than a period denotes the decimal, and they use spaces where we use commas).

Now: boundary tests may reveal some Ellis Island bugs. Other Ellis Island bugs defy boundary testing, because there's a catch: many such tests would require you to know what the boundary is and what is supposed to happen when it is crossed. From the outside, that's not at all clear. It's not even clear to the programmer, when libraries are doing the work. That's why it's insufficient to test at the boundaries that we know about already; that's why we must explore.

Testing and Management Parallels

michael.a.bolton@gmail.com (Michael Bolton http://www.developsense.com) — Thu, 04 Feb 2010 16:50:00 +0000

Rikard Edgren, Henrik Emilsson and Martin Jansson collaborate on blog called thoughts from the test eye. In a satirical post from this past summer called "Scripted vs Exploratory Testing from a Managerial Perspective", Martin proposes that "From a managerial perspective without knowing too much about testing, your sole experience comes from the scripted test environment…" But I think that from a managerial perspective, there is another place you could look to understand skilled testing: managing. I'll follow the points in Martin's post.

If you're a capable manager, and you're managing other managers, you know that there are things for which scripting doesn't work:

Control. Managers guide the managers working under them, but everyone involved knows that managers don't have complete control over what they're managing. No script can capture the esssence of management work. (If scripts could do that, we'd have automated management by now.) Managers know that when they have some written guidance on how workers are to perform certain tasks, effective workers and managers alike must adapt to the situation and use their judgement. If, as a manager, you could script workers' actions completely, they wouldn't come to your office to ask for help, and you wouldn't have to assist, guide, motivate, or reprimand them. You, the manager, have to observe a variety of things that cannot be anticiapted, and respond to what actually happens. You might have checklists, but you don't have a list of scripted tasks. You recognize that knowing when management work will end for a particular project can be anticipated but not predicted with certainty. Indeed, that's a function of the risks that you're hired to manage and the problems you're hired to solve. As a manager, you're managing many things simultaneously. You have the freedom and responsibility to carry out your work in the manner you think best, and you grant similar freedom and responsibility to your people. Isn't all that like being a tester, and like managing testers?

Hierarchy. There is a structure to management, with different roles playing their part in the system. No competent manager supervising other managers would characterize management as "some people to do the thinking and others execute". That would suggest that some managers think and other managers execute. As a manager, you recognize that all managers worthy of the name both think and execute, with the recognition that an organization is stronger as a collaborative network. Isn't that like being a tester, and like managing testers?

Scalability. You know that in management, you can't easily bring in people who can execute management scripts that other managers have written. Managers need to own their processes. Getting new managers in the middle of a project would derail it, and you can't take just anyone. Isn't that like being a tester, and like managing testers?

Management Software As a manager, you know that no tool—even one that costs several million dollars—can replace your judgment. At best, it collate data and can generate excellent reports, but the decision-making is yours. As a manager, you're leery of having your work overly mediated. When you have important but mundane tasks to perform, you hand off the non-sapient parts to computing machinery, but you apply sapience to planning, designing, and programming the tools—and you apply sapience to observing the results, to determining their meaning and signifiance, and to your response. When you have to delegate sapient work, you know that it can't be performed by a machine. So you hire someone—a person, not a machine—to do the work with your collaboration and guidance. Isn't that like being a tester, and like managing testers?

Education. You look back on how you learned, and you realize that, whether you had years of schooling or learned on the job, you don't believe in mail-order management courses, and you harbour no illusions that a two-day course accompanied by a piece of paper can teach you how to be a manager; nor can you trust that someone brandishing a similar piece of paper is ready for a management job until you know a lot more about him. Isn't that like being a tester, and like managing testers?

What does Exploratory Testing (ET) include? Well, it's kind of like management, isn't it?

Flatness In Organization. Managers perform management actions as they go along. Managers do not need people to design their actions for them. Managers foster leadership by empowering people to use their skills; guiding, but not controlling; granting freedom and requiring responsibility. Isn't that like being a tester, and like managing testers?

Chaos Can Be Tamed. You have no idea on how you are going to manage, nor on how the managers reporting to you will manage. You have not planned everything out in detail before you start managing; you can't, and you know you'd be fooling yourself if you pretended to do so. You cannot report exactly how long time you need, since you don't know everything in advance. In fact, discovering what needs to be done is a key aspect of your work. You recognize that management is a holistic process, not a linear one. You will use your skills, combined with all of the information available to inform your decisions on time, scope, quality, innovation, skill, and learning. You will use feedback from your surroundings to gather the information you need to make decisions. Isn't that like being a tester, and like managing testers?

Scalability. When you're hiring people to be managers who report to you , you only want managers. If they're not ready for that, but show promise, you'll train and mentor them into the role. Not anyone can be a manager. It is hard to just get anyone to help out since you cannot use just anyone from the organisation. They need to learn real management skills to be effective., which means that, among other things they must be given the freedom to make mistakes that can be observed and corrected in an empowering, fault-tolerant environment. When looked at this way, management does scale. Isn't that like being a tester, and like managing testers?

Skills-based Education Multiple-choice based certification for managers is insufficient. Better: there are degree programs, and there are shorter skill-based courses that involve simulations, open discussion, and testing actual software. Good courses are valuable supplements to an environment that fosters learning and innovation; courses that teach only management nomenclature are a waste of time and money. Isn't that like being a tester, and like managing testers?

Management Software Isn't Management. Management isn't done by software. Major software vendors have tools for this, but they don't replace managers. Customer relationship management software is not customer relationship management; enterprise resource management software isn't enterprise resource management. A real manager knows that it is what she thinks and what she does is important; that for her real work--the analysis and decision making--her paper notepad is as just as valid a tool as an Excel spreadsheet, and that no tool, no matter how big or how expensive or how powerful, is anything more than a tool. Isn't that like being a tester, and like managing testers?

Excellent testing skill has much in common with excellent management skill. As testers, maybe we can use the similarities between them to help explain the work that we do.

Exploratory Testing IS Accountable

michael.a.bolton@gmail.com (Michael Bolton http://www.developsense.com) — Wed, 27 Jan 2010 19:37:00 +0000

In this blog post, my colleague James Bach talks about logging and its importance in support of exploratory testing. Logging takes care of one part of the accountability angle, and in an approach like session-based test management (developed by James and his brother Jon), the test notes and the debrief take care of another part of it.

Logging records what happened from the perspective of the test system. Good logging relieves the tester from having to record specific actions in detail; the machine does that. The tester is thereby free to record test notes—a running account of the tester's ideas, questions, and results as he tested, or what happened from the perspective of the tester. Those notes form the meat of the session sheet, which also includes

coverage data
who did the testing
when they started
how long it took
the proportion of time spent on test design and execution, bug investigation and reporting, and setup
the proportion of the time spent on on-charter work vs. opportunity work
references to log files, data files, and related material such as scenarios, help files, specifications, standards, and so forth
and, of course, bugs discovered and issues identified.

After the session or at the end of the day, the tester presents a report—the session sheet combined with an oral account—in the debrief, a conversation between the tester and the test lead or test manager. In the debrief, the test lead reviews—that is, tests—the tester's experience and his report. The question "What happened?" gets addressed; the oral and written aspects of the report get discussed and evaluated; the session charter is confirmed or revised; holes are discovered and, where needed, plugged with followup testing; bug reports get reviewed; issues get brought up; coaching happens; mentoring happens; learning happens; knowledge gets transferred. The goal here is for the tester and the test lead to be able to say, "we can vouch for what was tested".

The session sheet is structured in such a way that it can be scanned by a text-parsing tool written in Perl. The measurements (in particular the coverage measurements) are collected and collated automatically into reports in the form of sortable HTML tables. Session sheets are kept for later review, if they're needed.

If logging in the program isn't available right away, screen recording tools (like BB Test Assistant, Camtasia, Spector, ...) can provide a retrospective account of what happened. (An over-the-shoulder video camera works too.) Note that these tools simply record video (and, optionally, sound—which is good for narration). Programmatic repetition of the session isn't the point. Nor is the point to have a supervisor review the screen capture obsessively; that wastes time, and besides, nobody likes working for Big Brother. The idea is to use the video only when necessary—to aid in recollection where it's needed, and to help in troubleshooting hard-to-reproduce bugs.

We suggest, where it doesn't get in the way, taking the test notes on the same machine as the application under test, and using the text editor window popping up as a way to link the execution of the application with bugs, test ideas or questions. For bugs that don't appear to be state-critical you can also take very brief notes for later followup. Include a time stamp, where the time stamp is an index into the recording; then revisit the recording later if more detail is called for. (In Notepad, you can press F5; in TextPad, Edit/Insert/Time, and it's macroable; other text editors almost certainly have a similar feature.)

Between a charter, the session sheet, the oral report, data files, and the logs and the debrief, it's hard for me to imagine a more accountable way of working. Each aspect of the reporting structure reinforces the others. This is why I get confused when test managers talk about exploratory testing being "unaccountable" or "unmanageable" or "unstructured": when I ask them what accountability and management means to them, they point lamely to a pile of scripts or spreadsheets full of overspecified actions that were written weeks or months before the software was built, or they mumble something about not knowing what goes on in a tester's head.

Any testing approach is manageable when you choose to manage it. If you want structure think about what you mean (maybe this guide to the structures of exploratory testing will help), identify the structures that are important to you, and develop those structures in your testers, in your team, and in your approaches. If you want accountability, provide structures for it (like session-based test management), and then require accountability. If you find that your testers aren't sufficiently skilled, train them and mentor them. (And if you don't know how to do that rapidly and effectively, we can help you.)

If there's something you don't like about the results you're getting, manage: observe what's going on in your system of testing, and put in a control action where you want to change something. If you want to know what's going on in a tester's head, observe her directly and interview her as she's testing; have her pair with another tester or a test lead; critique her notes; debrief her and coach her, until you get the results that you seek. If you want to supercharge the efficiency of your testers, work with the programmers and their managers to focus on testability, with special attention paid to scriptable interfaces, logging, and at least some programmer testing. (It might help to identify the information-hiding and feedback-loop-lengthening costs of the absence of testability). If you find individual debriefs taking too long, or if you want to share information more broadly within the test team, try group debriefs at the end of one day or the beginning of the next. If you want to add features to the reporting protocol, add them; if you want to drop them, drop them. Experiment, re-evaluate, and tune your testing as you see fit.

And if you have a more manageable and accountable approach than this for fostering the discovery of important problems in the product, please let us know (me, or James, or Jon). We'd really like to hear about it.

Disposable Time

michael.a.bolton@gmail.com (Michael Bolton http://www.developsense.com) — Sun, 17 Jan 2010 07:18:00 +0000

In our Rapid Testing class, James Bach and I like to talk about an underappreciated tester resource: disposable time. Disposable time is the time that you can afford to waste without getting into trouble.

Now, we want to be careful about what we mean by "waste", here. It's not that you want to waste the time. You probably want to spend it wisely. It's just that you won't suffer harm if you do happen to waste it. Disposable time is to your working hours what disposable income is to your total personal income. (In fact, even that's not quite correct, strictly speaking; we actually mean discretionary income: the money that's left over after you've paid for all of the things that you must pay for—food, shelter, basic clothing, medical, and tax expenses. The money that people call disposable income is more properly called discretionary income; as Wikipedia says, "the amount of 'play money' left to spend or save." Oh well. We'll go with the incorrect but popular interpretation of "disposable" here. )

You're never being scrutinized every minute of every day. Practically everyone has a few moments when no one important is watching. In that time, you might

try a tiny test that hasn't been prescribed.

try putting in a risky value instead of a safe value.

pretend to change your mind, or to make a mistake, and go back a step or two; users make mistakes, and error handling and recovery are often the most vulnerable parts of the program.

take a couple of moments to glance at some background information relevant to the work that you're doing.

write in your journal.

see if any of your colleagues in technical support have a hot issue that can inform some test ideas.

steal a couple of moments to write a tiny, simple program that will save you some time; use the saved time and the learning to extend your programming skills so that you can solve increasingly complex programming problems.

spend an extra couple of minutes at the end of a coffee break befriending the network support people.

sketch a workflow diagram for your product, and at some point show it to an expert, and ask if you've got it right.

snoop around in the support logs for the product.

add a few more lines to a spreadsheet of data values

help someone else solve a problem that they're having.

chat with a programmer about some aspect of the technology.

even if you do nothing else, at least pause and look around the screen as you're testing. Take a moment or two to recognize a new risk and write down a new question or a new test idea. Report on that idea later on; ask your test lead, your manager, or a programmer, or a product owner if it's a risk worth investigating. Hang on to your notes. When someone asks "Why didn't you find that bug," you may have an answer for them.

If it turns out that you've made a bad investment, oh well. By defintion, however large or small the period, disposable is time that you can afford to blow without suffering consequences.

On the other hand, you may have made a good investment. You may have found a bug, or recognized a new risk, or learned something important, or helped someone out of a jam, or built on a professional relationship, or surprised and impressed your manager. You may have done all of these things at once. Even if you feel like you've wasted your time, you've probably learned enough to insulate yourself from wasting more time in the same way. When you discover that an alley is blind, you're unlikely to return there when there are other things to explore.

In The Black Swan, Nassim Nicholas Taleb proposes an investment strategy wherein you put the vast bulk of your money, your nest egg, in very safe securities. You then invest a small amount—an amount that you can afford to lose—in very speculative bets that have a chance of providing a spectacular return. He call that very improbable high-return event a positive Black Swan. Your nest egg is like the part of your job that you must accomplish. Disposable time is like your Black Swan fund; you may lost it all, but you have a shot at a big payoff. But there's an important difference, too: since learning is an almost inevitable product of using your disposable time, there's almost always some modest positive outcome.

We encourage test managers to allow disposable time explicitly for their testers. As an example, Google provides its staff with Innovation Time Off. Engineers are encouraged to spend 20% of their time pursuing projects that interest them. That sounds like a waste, until one learns that Google projects like Gmail, Google News, Orkut, and AdSense came of these investments.

What Google may not know is that even within the other 80% of the time that's ostensibly on mission, people still have, and are still using, non-explicit disposable time. People have that almost everywhere, whether they have explicit disposable time or not.

If you're working in an environment where you're being watched so closely that none of this is possible, and where you're punished for learning or seeking problems, my advice is to make sure that slavery has been abolished in your jurisdiction. Then find a job where your testing skills are valued and your managers aren't wasting their time by watching your work instead of doing theirs. But when you've got a few moments to fill, fill them and learn something!

Defect Detection Efficiency: An Evaluation of a Research Study

michael.a.bolton@gmail.com (Michael Bolton http://www.developsense.com) — Fri, 08 Jan 2010 00:11:00 +0000

Over the last several months, B.J. Rollison has been delivering presentations and writing articles and blog posts in which he cites a paper Defect Detection Efficiency: Test Case Based vs. Exploratory Testing, [DDE2007] by Juha Itkonen, Mika V. Mäntylä and Casper Lassenius (First International Symposium on Empirical Software Engineering and Measurement, pp. 61-70; the paper can be found here).

I appreciate the authors’ intentions in examining the efficiency of exploratory testing. That said, the study and the paper that describes it have some pretty serious problems.

Some Background on Exploratory Testing

It is common for people writing about exploratory testing to consider it a technique, rather than an approach. “Exploratory” and “scripted” are opposite poles on a continuum. At one pole, exploratory testing integrates test design, test execution, result interpretation, and learning into a single person at the same time. At the other, scripted testing separates test design and test execution by time, and typically (although not always) by tester, and mediates information about the designer’s intentions by way of a document or a program.As James Bach has recently pointed out, the exploratory and scripted poles are like “hot” and “cold”. Just as there can be warmer or cooler water, there are intermediate gradations to testing approaches. The extent to which an approach is exploratory is the extent to which the tester, rather than the script, is in immediate control of the activity. A strongly scripted approach is one in which ideas from someone else, or ideas from some point in the past, govern the tester’s actions. Test execution can be very scripted, as when the tester is given an explicit set of steps to follow and observations to make; somewhat scripted, as when the tester is given explicit instruction but is welcome or encouraged to deviate from it; or very exploratory, in which the tester is given a mission or charter, and is mandated to use whatever information and ideas are available, even those that have been discovered in the present moment.

Yet the approaches can be blended. James points out that the distinguishing attribute in exploratory and scripted approaches is the presence or absence of loops. The most extreme scripted testing would follow a strictly linear approach; design would be done at the beginning of the project; design would be followed by execution; tests would be performed in a prescribed order; later cycles of testing would use exactly the same tests for regression

Let's get more realistic, though. Consider a tester with a list of tests to perform, each using a data-focused automated script to address a particular test idea. A tester using a highly scripted approach would run that script, observe and record the result, and move on to the next test. A tester using a more exploratory approach would use the list as a point of departure, but upon observing an interesting result might choose to perform a different test from the next one on the list; to alter the data and re-run the test; to modify the automated script; or to abandon that list of tests in favour of another one. That is, the tester's actions in the moment would not be directed by earlier ideas, but would be informed by them. Scripted approaches set out the ideas in advance, and when new information arrives, there's a longer loop between discovery and the incorporation of that new information into the testing cycle. The more exploratory the approach, the shorter the loop. Exploratory approaches do not preclude the use of prepared test ideas, although both James and I would argue that our craft, in general, places excessive emphasis on test cases and focusing techniques at the expense of more general heuristics and defocusing techniques.

The point of all this is that neither exploratory testing nor scripted approaches are testing techniques, nor bodies of testing techniques. They're approaches that can be applied to any testing technique.

To be fair to the authors of [DDE2007], since publication of their paper there has been ongoing progress in the way that many people—in particular Cem Kaner, James Bach, and I—articulate these ideas, but the fundamental notions haven’t changed significantly.

Literature Review

While the authors do cite several papers on testing and test design techniques, they do not cite some of the more important and relevant publications on the exploratory side. Examples of such literature include “Measuring the Effectiveness of Software Testers” (Kaner, 2003; slightly updated in 2006); and “Software engineering metrics: What do they measure and how do we know?” (Kaner & Bond, 2004); and "Inefficiency and Ineffectiveness of Software Testing: A Key Problem in Software Engineering” (Kaner 2006; to be fair to the authors, this paper may have been published too late to inform [DDE2007]), General Functionality and Stability Test Procedure (for Microsoft Windows 2000 Application Certification) (Bach, 2000); Satisfice Heuristic Test Strategy Model (Bach, 2000); How To Break Software (Whittaker, 2002).

The authors of [DDE2007] appear also to have omitted literature on the subject of exploration and its role in learning. Yet there is significant material on the subject, in both popular and more academic literature. Examples here include Collaborative Discovery in a Scientific Domain (Okada and Simon; note that the subjects are testing software); Exploring Science: The Cognition and Development of Discovery Processes (David Klahr and Herbert Simon); Plans and Situated Actions (Lucy Suchman); Play as Exploratory Learning (Mary Reilly); How to Solve It (George Polya); Simple Heuristics That Make Us Smart (Gerg Gigerenzer); Sensemaking in Organizations (Karl Weick); Cognition in the Wild (Edward Hutchins); The Social Life of Information (Paul Duguid and John Seely Brown); Sciences of the Artificial (Herbert Simon); all the way back to A System of Logic, Ratiocinative and Inductive (John Stuart Mill, 1843).

These omissions are reflected in the study and the analysis of the experiment, and that leads to a common problem in such studies: heuristics and other important cognitive structures in exploration are treated as mysterious and unknowable. For example, the authors say, “For the exploratory testing sessions we cannot determine if the subjects used the same testing principles that they used for designing the documented test cases or if they explored the functionality in pure ad-hoc manner. For this reason it is safer to assume the ad-hoc manner to hold true.” [DDE2007, p. 69] Why assume? At the very least, one could at least observe the subjects and debrief them, asking about their approaches. In fact, this is exactly the role that the test lead fulfills in the practice of skilled exploratory testing. And why describe the principles only as "ad-hoc"? It's not like the principles can't be articulated. I talk about oracle heuristics in this article, and talk about stopping heuristics here; Kaner's Black Box Software Testing course talks about test design heuristics; James Bach's work talks about test strategy heuristics (especially here); James Whittaker's books talk about heuristics for finding vulnerabilities...

Tester Experience

The study was performed using testers who were, in the main, novices. “27 subjects had no previous experience in software engineering and 63 had no previous experience in testing. 8 subjects had one year and 4 subjects had two years testing experience. Only four subjects reported having some sort of training in software testing prior to taking the course.” ([DDE2007], p. 65 my emphasis) Testing—especially testing using an exploratory approach—is a complex cognitive activity. If one were to perform a study on novice jugglers, one would likely find that they drop an approximately equal number of objects, whether they were juggling balls or knives.

Tester Training

The paper notes that “subjects were trained to use the test case design techniques before the experiment.” However, the paper does not make note of any specific training in heuristics or exploratory approaches. That might not be surprising in light of the weaknesses on the exploratory side of the literature review. My experience, that of James Bach, and anecdotal reports from our clients suggests that even a brief training session can greatly increase the effectiveness of an exploratory approach.

Cycles of Testing

Testing happens in cycles. In a strongly scripted testing, the process tends to the linear. All tests are designed up front; then those tests are executed; then testing for that area is deemed to be done. In subsequent cycles, the intention is to repeat the original tests to make sure that bugs are fixed to check for regression. By contrast, exploratory testing is an organic and iterative process. In an exploratory approach, the same area might be visited several times, such that learning from early “reconnaissance” sessions informs further exploration in subsequent “deep coverage” sessions. The learning from those (and from ideas about bugs that have been found and fixed) informs “wrap-up sessions”, in which tests may be repeated, varied, or cut from new cloth. No allowance is made for information and learning obtained during one round of testing to inform later rounds. Yet such information and learning is typically of great value.

Quantitative vs. Qualitative Analysis

In the study, there is a great deal of emphasis placed on quantifying results, on experimental and on mathematical rigour. However, such rigour may be misplaced when the products of testing are qualitative, rather than quantitative.

Finding bugs is important, finding many bugs is important, and finding important bugs is especially important. Yet bugs and bug reports are by no means the only products of testing. The study largely ignores the other forms of information that testing may provide.

The tester might learn something about test design, and feed that learning into her approach toward test execution, or vice versa. The value of that learning might be realized immediately (as in an exploratory approach) or over time (as in a scripted approach).

The tester, upon executing a test, might recognize a new risk or missing coverage. That recognition might inform ideas about the design and choices of subsequent tests. In a scripted approach, that’s a relatively long loop. In an exploratory approach, upon noticing a new risk, the tester might choose to note findings for later on. On the other hand, the discovery could be cashed immediately: she might choose to repeat the test, she might perform a variation on the same test, or might alter her strategy to follow a different line of investigation. Compared to a scripted approach, the feedback loop between discovery and subsequent action is far shorter. The study ignores the length of the feedback loops.

In addition to discovering bugs that threaten the value of the product, the tester might discover issues—problems that threaten the value of the testing effort or the development project overall.

The tester who takes an exploratory approach may choose to investigate a bug or an issue that she has found. This may reduce the total bug count, but in some contexts may be very important to the tester’s client. In such cases, the quality of the investigation, rather than the number of bugs found, would be important.

More work products from testing can be found here.

“Efficiency” vs. “Effectiveness”

The study takes a very parsimonious view of “efficiency”, and further confuses “efficiency” with “effectiveness”. Two tests are equally effective if they produce the same effects. The discovery of a bug is certainly an important effect of a test. Yet there are other important effects too, as noted above, but they're not considered in the study.

However, even if we decide that bug-finding is the only worthwhile effect of a test, two equally effective tests might not be equally efficient. I would argue that efficiency is a relationship between effectiveness and cost. An activity is more efficient if it has the same effectiveness at lower cost in terms of time, money, or resources. This leads to what is by far the most serious problem in the paper...

Script Preparation Time Is Ignored

The authors’ evaluation of “efficiency” leaves out the preparation time for the scripted tests! The paper says that the exploratory testing sessions took 90 minutes for design, preparation, and execution. The preparation for the scripted tests took seven hours, where the scripted test execution sessions took 90 minutes, for a total of 8.5 hours. This fact is not highlighted; indeed, it is not mentioned until the eighth of ten pages. (page 68). In journalism, that would be called burying the lead. In terms of bug-finding alone, the authors suggest that the results were of equivalent effectiveness, yet the scripted approach took, in total, 5.6 times longer than the exploratory approach. What other problems could the exploratory testing approaches find given seven additional hours?

Conclusions

The authors offer these four conclusions at the end of the paper:

“First, we identify a lack of research on manual test execution from other than the test case design point of view. It is obvious that focusing only on test case design techniques does not cover many important aspects that affect manual testing. Second, our data showed no benefit in terms of defect detection efficiency of using predesigned test cases in comparison to an exploratory testing approach. Third, there appears to be no big differences in the detected defect types, severities, and in detection difficulty. Fourth, our data indicates that test case based testing produces more false defect reports.”

I would offer to add a few other conclusions. The first is from the authors themselves, but is buried on page 68: “Based on the results of this study, we can conclude that an exploratory approach could be efficient, especially considering the average 7 hours of effort the subjects used for test case design activities.” Or, put another way,

During test execution
unskilled testers found the same number of problems, irrespective of the approach that they took, but
preparation of scripted tests increased testing time approximately by a factor of five
and appeared to add no significant value.

Now: as much as I would like to cite this study as a significant win for exploratory testing, I can’t. There are too many problems with it. There’s not much value in comparing two approaches when those approaches are taken by unskilled and untrained people. The study is heavy on data but light on information. There are no details about the bugs that were found and missed using each approach. There’s no description of the testers’ activities or thought processes; just the output numbers. There is the potential for interesting, rich stories on which bugs were found and which bugs were missed by which approaches, but such stories are absent from the paper. Testing is a qualitative evaluation of a product; this study is a quantitative evaluation of testing. Valuable information is lost thereby.

The authors say, “We could not analyze how good test case designers our subjects were and how much the quality of the test cases affected the results and how much the actual test execution aproach.” Actually, they could have analyzed that. It’s just that they didn’t. Pity.

Handling an Overstructured Mission

michael.a.bolton@gmail.com (Michael Bolton http://www.developsense.com) — Sat, 26 Dec 2009 20:36:00 +0000

Excellent testers recognize that excellent testing is not merely a process of confirmation, verification, and validation. Excellent testing is a process of exploration,discovery, investigation, and learning.

A correspondent that I consider to be an excellent tester (let's call him Al) works in an environment where he is obliged by his managers to execute overly structured, highly confirmatory scripted tests. Al wrote to me recently, saying that he now realizes why that's frustrating for him: every time he runs through a scripted test, he gets five new ideas that he wants to act upon. I think that's a wonderful thing, but when he acts on those ideas and fulfills his implicit mission (finding important problems in the product), it diverts him from his explicit mission (to complete some number of scripted tests per day), and he gets heat from his manager about that. At the end of a couple of days, the manager wants to know why Al is behind schedule—even if Al has revealed important problems along the way—because the manager is focused on test effort in terms of test cases completed, rather than test ideas explored.

I suggested to Al (as I suggest to you, if you're in that kind of situation) a workaround: don't act on the new test ideas; but do note them. Jot them down in handwritten notes or a text file, and especially note your motivation for them—ideas about risk, coverage, oracles, strategies, and the like. Tell your test manager or test lead that you didn't run tests associated with those ideas, and then ask, "Are you okay with us NOT running them?"

In addition, check in with your manager more often than once every two days. Deliver a report, including new ideas, at one- to two-hour intervals. If direct personal contact isn't available, try instant messages or email. If those don't work, batch them, but note the time at which you started and/or stopped a burst of testing activity.

Al was excited about that. "Wow!" he said. "That also means defects arising from the new ideas are noted down. Currently, my management is under the impression that test cases are the things that reveal problems, but it's my acting on my test ideas that really reveals the problems." He also noted, "There's another bad that comes from that. If the test cases don't reveal problems, we take the problems that we've found and create a test case for them so that those problems aren't missed next time." I've seen that happen a lot, too. On the face of it, it doesn't sound like a bad idea—except that specific problems that are fixed and verified tend to remain fixed. Repeating those tests is an opportunity cost against new tests that would reveal previously undiscovered problems.

So: the idea here is to make certain aspects of our work visible. Scripted test cases often reveal problems as those cases are developed. When those problems get fixed, the script loses its power. Thus variation on the script, rather than following the script rigourously, tends to reveal the actual problem. However, unless we're clear that this is happening, managers will mistakenly give credit to the wrong thing—namely, the script—rather than to the mindset and the skill set of the tester.

Selena Delesie on Exploratory Test Chartering

michael.a.bolton@gmail.com (Michael Bolton http://www.developsense.com) — Fri, 18 Dec 2009 15:36:00 +0000

A little while ago, I mentioned that I'd be writing more about session-based test management (SBTM). For me, one thing that's great about having a community of students and colleagues is that they can save me lots of time and work.

Selena Delesie took the Rapid Software Testing course from me a few years back (that is, she was a student). Since then, she has taken Rapid Testing and its practices, including SBTM, and made them her own. This is exactly what James Bach and I aim for. We want to help testers, test leads, and managers realize the the most important factor in excellent testing, bar none, is the mindset and the skill set of the individual tester. This means taking the ideas in the course and internalizing them, adopting them, developing them, experimenting with them, altering them to fit your context. We get people started by making them feel powerful, mostly by helping them to recognize the power and skills that they already have. Then, after the class, they can feel confident in doing the heavy lifting on their own. Selena is by no means our only student who has done that, but she's a paradigmatic example of what's possible.

This post from her blog is a nice account of her appreciation of exploratory testing and of her career growth. That on its own would be good enough, but she's now blogged a post on chartering sessions, and it's excellent. It identifies some of the common traps and misconceptions about chartering, and provides some sharp advice on how to avoid them. It talks not merely about how to charter, but how to do it in a way that affords the tester the freedom and responsibility to do his or her best work. Highest recommendation.

Structures of Exploratory Testing: Resources

michael.a.bolton@gmail.com (Michael Bolton http://www.developsense.com) — Mon, 14 Dec 2009 05:19:00 +0000

In a Webinar that he did for uTest on December 10, James Whittaker mused aloud about what a great idea it would be to structure exploratory testing and capture ideas about it in a repository for sharing with others. It seems to me that one ideal version of that would take the form of a bibliography in a book about exploratory testing, but apparently that's not available. Yet I digress.

The fact is, people have been doing exactly that for years. And I do like the idea of having a repository and sharing, so here's a survey of some exploratory testing structures and some writing about them that I hope people will find helpful. There are some excellent books out there, but for now, these ones are all online and free. Expect updates.

Evolving Work Products, Skills and Tactics, ET Polarities, and Test Strategy. James Bach, Jon Bach, and I authored the latest version of the Exploratory Skills and Dynamics list. This is a kind of evolving master list of exploratory testing structures. James describes it here.
Oracles. The HICCUPPS consistency heuristics, which James Bach initiated and which I wrote about in this article for Better Software in 2005. (Actually, at the time it was only HICCUPP—History, Image, Comparable Products, Claims, User Expectations, Purpose, Product—but since then we've also added S, for Standards and Statutes. Mike Kelly also talks about HICCUPP here.
Test Strategy. James Bach's Heuristic Test Strategy Model isn't restricted to exploratory approaches, but certainly helps to guide and structure them.
Data Type Attacks, Web Tests, Testing Wisdom, Heuristics, and Frameworks. Elisabeth Hendrickson's Test Heuristics Cheat Sheet is a rich set of guideword heuristics and helpful reference information.
Context Factors, Information Objectives. Cem Kaner most recently delivered his Tutorial on Exploratory Testing for the QAI Quest Conference in Chicago, 2008. There's a similar, but not identical talk here.
Quick Tests. In our Rapid Software Testing course, James Bach and I talk about quick tests. The course notes are available for free. Fire up Acrobat and search for "Quick Tests".
Coverage (specific). Michael Hunter's You Are Not Done Yet is a detailed set of coverage ideas to help prompt further exploration when you think you're done.
Coverage (general). James Bach wrote this article in 2001, in which he summarizes test coverage ideas under the mnemonic "San Francisco Depot."—Structure, Function, Data, Platform, and Operations. Several years later, I convinced him to add an element to the list, so now it's "San Francisco Depot. The last T is for...
Time. I realized a few years ago that some guideword heuristics might help us to pay attention to the ways in which products related to time, and vice versa. That turned into a Better Software article called "Time for New Test Ideas".
Tours. Mike Kelly's FCC CUTS VIDS Touring Heuristics (note the date) provides a set of structured approaches for touring the application.
Stopping Heuristics. There are structures to deciding when to stop a given test, a line of investigation, or a test cycle. I catalogued them here, and Cem Kaner made a vital addition here.
Accountability, Reporting Progress. James and Jon Bach's description of Session-Based Test Management is a set of structures for making exploratory testing sessions more accountable.
Procedure. The General Functionality and Stability Test Procedure. It was designed for Microsoft in the late 1990s by James Bach, and may be the first documented procedure to guide exploratory test execution and investigation.
Emotions. I gave a talk on emotions as powerful pointers to test oracles at STAR West in 2007. That helped to inspire some ideas about...
Noticing, Observation. At STAR East 2009, I did a keynote talk on noticing, which can be important for exploratory test execution. The talk introduces a number of areas in which we might notice, and some patterns to sharpen noticing.
Leadership. For the 2009 QAI Conference in Bangalore, India, I did a plenary talk in which I noted several important structural similarities between exploratory testing and leadership.

So, there it is: a repository. I'll eventually reproduce it as part of the resources page on my Web site. Feel free to share; comments and suggestions for additions are welcome.

Best Bug... or Bugs?

michael.a.bolton@gmail.com (Michael Bolton http://www.developsense.com) — Wed, 09 Dec 2009 21:55:00 +0000

And now for the immodest part of the EuroSTAR 2009 Test Lab report: I won the Best Bug award, although it's not clear to me which bug got the nod, since I reported several fairly major problems.

I tested OpenEMR. For me, one candidate for the most serious problem would have been a consistent pattern of inconsistency in input handling and error checking. I observed over a dozen instances of some kind of sloppiness.

This reminded me of a problem that we testers often see in project work, the problem of measuring by counting things—counting bugs, counting bug reports, counting requirements. When the requirement is to defend the application against overflowing text fields and vulnerability to input constraint attacks by hackers, how should we count? How many mentions of that should there be? One, in a statement of general principles at the beginning of a requirements document? Hundreds, in a statement of specific purpose for each input field in a functional specification? How many requirements are there to make sure that fields don't overflow? How many requirements that they support only the characters, numbers, or date ranges that they're supposed to? What about traceability? If this is a genuine problem, and the requirements documents don't mention a particular requirement explicitly, should we refrain from reporting on a problem with that implicit requirement?

When I report an issue—for example, that practically all of the input fields in OpenEMR have some kind of problem with them—should that count as one bug report? Since it applies to hundreds of fields, should it count as hundreds of bug reports? When such a pervasive overall problem exists, should the tester make a report for each and every field in which he observes a problem? And if you want to answer Yes, to that question: is it worth the opportunity cost to do that when there are plenty of other problems in the product?

So again, there were so many instances of unconstrained and unchecked input that I stopped recording specifics and instead reported a general pattern in the bug tracking system. My decision to do this was an instance of the Dead Horse stopping heuristic; reporting yet another instance of the same class of problem would be like flogging a dead horse. I could have wasted a lot of time and energy reporting each instance of each problem I observed, along with specific symptoms and possible ramifications of each one. Yet I'm very skeptical that this would serve the project well. In my experience as a program manager for a product whose code was being developed outside our company, I found that there was steadily diminishing return in value for many reports of the same ilk. When, in testing, we identified a general pattern of failure, we stopped looking for more instances. We sent the product back to the development shop, and required the programmers and their testers to review the product through-and-through for that kind of problem.

If I were to be evaluated on the number of bugs that I found, I'd find it hard to resist the easy pickings of yet another input constraint attack bug report. Yet when I'm testing, every moment of bug investigation and reporting is, by some reckoning, another moment that I can't spend on obtaining more test coverage (more about that here). By focusing on investigating and reporting on input problems (and thereby increasing my bug count), am I missing opportunities to design and perform tests on scheduling conflict-resolution algorithms, workflows, database integrity,...?

There were two other fairly serious problems that I observed. One was that the Chinese version of the product showed a remarkable number of English words, presumably untranslated, interspersed among the ideograms; I expected to see no English at all. I treated that problem in the same way as the input constraint problem: with a single report of a general problem.

The second serious problem was that searches of various kinds would place a link in the address bar. The link represented a command to a CGI script of some kind, which evidently constructed and forwarded a query to an underlying SQL database. Backspacing over the last digit in the address bar and replacing it with a slash caused a lovely SQL error message to appear on the screen, unhandled by any of OpenEMR's code. The message could have been used, said our local product owner, to expose the structure of the database to snoops or hackers. I found that problem by a defocusing heuristic—looking at the browser, rather than the browser window.

I don't know which of these problems took Best Bug honours. I'm not sure that the presenters specified which bug they were crediting with Best Bug. That makes a certain kind of sense, since I can't tell which of these problems is the most serious either. After all, a problem isn't its own thing; it's a relationship between a person and a product or a situation. There are plenty of ways to address a problem. You could fix the product or the situation. You could change the perspective or the perception of the person observing the problem, say by keeping the problem as it is but providing a workaround. You could choose to ignore the problem yourself, which underscores the fact that a problem for some person might not be a problem for you. That's why it's not helpful to count problems.

Managers: do you see how evaluating testers based on test cases or bug counts, rather than the value of reporting, will lead to distortion at best, and more likely to dysfunction? Do you see how providing overstructured test scripts or test cases could reduce the diversity—and therefore the quality—of testing? Do you see how the notion of "one test per requirement" or "one positive and one negative test per requirement" is misleading?

Testers: do you see how being evaluated on bug counts could lead to inattentional blindness with respect to the more serious problems than the low-hanging fruit affords? Do you see how focusing on bugs, rather than focusing on test coverage, could reduce the value of your testing?

Instead of counting things, let's consider evaluating testing work in a different way. Let's consider the overall testing story and its many dimensions. Let's think about the story around each bug, and each bug report—not just the number of reports, but the meaning and significance of each one. Let's look at the value of the information to stakeholders, primarily to programmers and to product owners. Let's think about the extent to which the tester makes things easier for others on the team, including other testers. Let's look at the diversity of problems discovered, the diversity of approaches used, and the diversity of tools and techniques applied. And rather than using this information to reward or punish testers, let's use it to guide coaching, mentoring, and training such that the focus is on developing skill for everyone.

The dimensions above are qualitative, rather than quantitative. Yet if our mission is to provide information to inform decisions about quality, we of all people should recognize that expressing value in terms of numbers often removes important information rather than adding it.

Additional reading:

Measuring and Managing Performance in Organizations (Robert D. Austin)
Software Engineering Metrics: What Do They Measure and How Do We Know? (Kaner and Bond)
Quality Software Management, Vol. 2: First Order Measurement (Weinberg)
Perfect Software (and Other Illusions About Testing) (Weinberg)

EuroSTAR's Test Lab: Bravo!

michael.a.bolton@gmail.com (Michael Bolton http://www.developsense.com) — Wed, 09 Dec 2009 17:42:00 +0000

One of the coolest things about EuroSTAR 2009 was the test lab set up by James Lyndsay and Bart Knaack.

James and Bart (who self-identified as Test Lab Rats) provided testers with the opportunity to have a go at two applications, FreeMind (an open-source mind-mapping program) and OpenEMR (an open-source product for tracking medical records). The Lab Rats did a splendid job of setting things up and providing the services and information that participants needed to get up and running quickly.

Sponsorship in the form of five laptop computers was provided through the good graces of Steve Green at Test Partners, Stuart Noakes at Transition Consulting Ltd., and Bart Knaack at Logica. James Lyndsay also lent a server and a router to the event.

Sponsorship was also provided by tool vendors (here in alphabetical order) Andagon, Microsoft, MicroFocus, Neotys, and Testing Technologies. These sponsors had their tools installed on the laptops, and presented their demos by applying them to OpenEMR and FreeMind as they were installed in the Test Lab. On a loose schedule, some of the presenters did talks and demonstrations of how they tested.

The aforementioned Stuart Noakes and Mieke Gievers gave advice and assistance to the Lab Rats.

Well, that's all very nice, but what was it like?

As someone who spent a couple of hours in the lab, exploring the applications and listening in on the presentations, I'd say it was terrific (although the prospect that OpenEMR is being used in actual medical practices seemed faintly alarming). Both applications were sophisticated enough for some reasonably serious testing, and had interesting problems to discover and report.

Interestingly, none of the certificationists or the standardization folks sat in the lab and tested, to my knowledge.

Bravo to James and Bart, to the sponsors, to the conference organizers and to the program committee for putting this together. Let's see more actual testing at testing conferences!

Why Is Testing Taking So Long? (Part 2)

michael.a.bolton@gmail.com (Michael Bolton http://www.developsense.com) — Wed, 25 Nov 2009 18:17:00 +0000

Yesterday I set up a thought experiment in which we divided our day of testing into three 90-minute sessions. I also made a simplifying assumption that bursts of testing activity representing some equivalent amoun of test coverage (I called it a micro-session, or just a "test") take two minutes. Investigating and reporting a bug that we find costs an additional eight minutes, so a test on its own would take two minutes, and a test that found a problem would take ten.

Yesterday we tested three modules. We found some problems. Today the fixes showed up, so we'll have to verify them.

Let's assume that a fix verification takes six minutes. (That's yet another gross oversimplification, but it sets things up for our little thought experiment.) We don't just perform the original microsession again; we have to do more than that. We want to make sure that the problem is fixed, but we also want to do a little exploration around the specific case and make sure that the general case is fixed too.

Well, at least we'll have to do that for Modules B and C. Module A didn't have any fixes, since nothing was broken. And Team A is up to its usual stellar work, so today we can keep testing Team A's module, uninterrupted by either fix verifications or by bugs. We get 45 more micro-sessions in today, for a two-day total of 90.

Module	Fix Verifications	Bug Investigation and Reporting (time spent on tests that find bugs)	Test Design and Execution (time spent on tests that don't find bugs)	New Tests Today	Two-Day Total
A	0 minutes (no bugs yesterday)	0 minutes (no bugs found)	90 minutes (45 tests)	45	90

Team B stayed an hour or so after work yesterday. They fixed the bug that we found, tested the fix, and checked it in. They asked us to verify the fix this afternoon. That costs us six minutes off the top of the session, leaving us 84 more minutes. Yesterday's trends continue; although Team B is very good, they're human, and we find another bug today. The test costs two minutes, and bug investigation and reporting costs eight more, for a total of ten. In the remaining 74 minutes, we have time for 37 micro-sessions. That means a total of 38 new tests today—one that found a problem, and 37 that didn't. Our two-day today for Module B is 79 micro-sessions.

Module	Fix Verifications	Bug Investigation and Reporting (time spent on tests that find bugs)	Test Design and Execution (time spent on tests that don't find bugs)	New Tests Today	Two-Day Total
A	0 minutes (no bugs yesterday)	0 minutes (no bugs found)	90 minutes (45 tests)	45	90
B	6 minutes (1 bug yesterday)	10 minutes (1 test, 1 bug)	74 minutes (37 tests)	38	79

Team C stayed late last night. Very late. They felt they had to. Yesterday we found eight bugs, and they decided to stay at work and fix them. (Perhaps this is why their code has so many problems; they don't get enough sleep, and produce more bugs, which means they have to stay late again, which means even less sleep...) In any case, they've delivered us all eight fixes, and we start our session this afternoon by verifying them. Eight fix verifications at six minutes each amounts to 48 minutes. So far as obtaining new coverage goes, today's 90-minute session with Module C is pretty much hosed before it even starts; 48 minutes—more than half of the session—is taken up by fix verifications, right from the get-go. We have 42 minutes left in which to run new micro-sessions, those little two-minute slabs of test time that give us some equivalent measure of coverage. Yesterday's trends continue for Team C too, and we discover four problems that require investigation and reporting. That takes 40 of the remaining 42 minutes. Somewhere in there, we spend two minutes of testing that doesn't find a bug. So today's results look like this:

Module	Fix Verifications	Bug Investigation and Reporting (time spent on tests that find bugs)	Test Design and Execution (time spent on tests that don't find bugs)	New Tests Today	Two-Day Total
A	0 minutes (no bugs yesterday)	0 minutes (no bugs found)	90 minutes (45 tests)	45	90
B	6 minutes (1 bug yesterday)	10 minutes (1 test, 1 bug)	74 minutes (37 tests)	38	79
C	48 minutes (8 bugs yesterday)	40 minutes (4 tests, 4 bugs)	2 minutes (1 test)	5	18

Over two days, we've been able to obtain only 20% of the test coverage for Module C that we've been able to obtain for Module A. We're still at less than 1/4 of the coverage that we've been able to obtain for Module B.

Yesterday, we learned one lesson:

Lots of bugs means reduced coverage, or slower testing, or both.

From today's results, here's a second:

Finding bugs today means verifying fixes later, which means even less coverage or even slower testing, or both.

So why is testing taking so long? One of the biggest reasons might be this:

Testing is taking longer than we might have expected or hoped because, although we've budgeted time for testing, we lumped into it the time for investigating and reporting problems that we didn't expect to find.

Or, more generally,

Testing is taking longer than we might have expected or hoped because we have a faulty model of what testing is and how it proceeds.

For managers who ask "Why is testing taking so long?", it's often the case that their model of testing doesn't incorporate the influence of things outside the testers' control. Over two days of testing, the difference between the quality of Team A's code and Team C's code has a profound impact on the amount of uninterrupted test design and execution work we're able to do. The bugs in Module C present interruptions to coverage, such that (in this very simplified model) we're able to spend only one-fifth of our test time designing and executing tests. After the first day, we were already way behind; after two days, we're even further behind. And even here, we're being optimistic. With a team like Team C, how many of those fixes will be perfect, revealing no further problems and taking no further investigation and reporting time?

And again, those faulty management models will lead to distortion or dysfunction. If the quality of testing is measured by bugs found, then anyone testing Module C will look great, and people testing Module A will look terrible. But if the quality of testing is evaluated by coverage, then the Module A people will look sensational and the Module C people will be on the firing line. But remember, the differences in results here have nothing to do with the quality of the testing, and everything to do with the quality of what is being tested.

There's a psychological factor at work, too. If our approach to testing is confirmatory, with steps to follow and expected, predicted results, we'll design our testing around the idea that the product should do this, and that it should behave thus and so, and that testing will proceed in a predictable fashion. If that's the case, it's possible—probable, in my view—that we will bias ourselves towards the expected and away from the unexpected. If our approach to testing is exploratory, perhaps we'll start from the presumption that, to a great degree, we don't know what we're going to find. As much as managers, hack statisticians, and process enthusiasts would like to make testing and bug-finding predictable, people don't know how to do that such that the predictions stand up to human variability and the complexity of the world we live in. Plus, if you can predict a problem, why wait for testing to find it? If you can really predict it, do something about it now. If you don't have the ability to do that, you're just playing with numbers.

Now: note again that this has been a thought experiment. For simplicity's sake, I've made some significant distortions and left out an enormous amount of what testing is really like in practice.

I've treated testing activities as compartmentalized chunks of two minutes apiece, treading dangerously close to the unhelpful and misleading model of testing as development and execution of test cases.
I haven't looked at the role of setup time and its impact on test design and execution.
I haven't looked at the messy reality of having to wait for a product that isn't building properly.
I haven't included the time that testers spend waiting for fixes.
I haven't included the delays associated with bugs that block our ability to test and obtain coverage of the code behind them.
I've deliberately ignored the complexity of the code.
I've left out difficulties in learning about the business domain.
I've made a highly simplistic assumptions about the quality and relevance of the testing and the quality and relevance of the bug reports, the skill of the testers in finding and reporting bugs, and so forth.
And I've left out the fact that, as important as skill is, luck always plays a role in finding problems.

My goal was simply to show this:

Problems in a product have a huge impact on our ability to obtain test coverage of that product.

The trouble is that even this fairly simple observation is below the level of visibilty of many managers. Why is it that so many managers fail to notice it?

One reason, I think, is that they're used to seeing linear processes instead of organic ones, a problem that Jerry Weinberg describes in Becoming a Technical Leader. Linear models "assume that observers have a perfect understanding of the task," as Jerry says. But software development isn't like that at all, and it can't be. By its nature, software development is about dealing with things that we haven't dealt with before (otherwise there would be no need to develop a new product; we'd just reuse the one we had). We're always dealing with the novel, the uncertain, the untried, and the untested, so our observation is bound to be imperfect. If we fail to recognize that, we won't be able to improve the quality and value of our work.

What's worse about managers with a linear model of development and testing is that "they filter our innovations that the observer hasn't seen before or doesn't understand" (again, from Becoming a Technical Leader.) As an antidote for such managers, I'd recommend Perfect Software, and Other Illusions About Testing and Lessons Learned in Software Testing as primers. But mostly I'd suggest that they observe the work of testing. In order to do that well, they may need some help from us, and that means that we need to observe the work of testing too. So over the next little while, I'll be talking more than usual about Session-Based Test Management, developed initially by James and Jon Bach, which is a powerful set of ideas, tools and processes that aid in observing and managing testing.

Why Is Testing Taking So Long? (Part 1)

michael.a.bolton@gmail.com (Michael Bolton http://www.developsense.com) — Tue, 24 Nov 2009 20:55:00 +0000

If you're a tester, you've probably been asked, "Why is testing taking so long?" Maybe you've had a ready answer; maybe you haven't. Here's a model that might help you deal with the kind of manager who asks such questions.

Let's suppose that we divide our day of testing into three sessions, each session being, on average, 90 minutes of chartered, uninterrupted testing time. That's four and a half hours of testing, which seems reasonable in an eight-hour day interrupted by meetings, planning sessions, working with programmers, debriefings, training, email, conversations, administrivia of various kinds, lunch time, and breaks.

The reason that we're testing is that we want to obtain coverage; that is, we want to ask and answer questions about the product and its elements to the greatest extent that we can. Asking and answering questions is the process of test design and execution. So let's further assume that we break each session into average two-minute micro-sessions, in which we perform some test activity that's focused on a particular testing question, or on evaluating a particular feature. That means in a 90-minute session, we can theoretically perform 45 of these little micro-sessions, which for the sake of brevity we'll informally call "tests". Of course life doesn't really work this way; a test idea might a couple of seconds to implement, or it might take all day. But I'm modeling here, making this rather gross simplification to clarify a more complex set of dynamics. (Note that if you'd like to take a really impoverished view of what happens in skilled testing, you could say that a "test case" takes two minutes. But I leave it to my colleague James Bach to explain why you should reject the concept of test cases.)

Let's further suppose that we'll find problems every now and again, which means that we have to do bug investigation and reporting. This is valuable work for the development team, but it takes time that interrupts test design and execution—the stuff that yields test coverage. Let's say that, for each bug that we find, we must spend an extra eight minutes investigating it and preparing a report. Again, this is a pretty dramatic simplification. Investigating a bug might take all day, and preparing a good report could take time on the order of hours. Some bugs (think typos and spelling errors in the UI) leap out at us and don't call for much investigation, so they'll take less than eight minutes. Even though eight minutes is probably a dramatic underestimate for investigation and reporting, let's go with that. So a test activity that doesn't find a problem costs us two minutes, and a test activity that does find a problem takes ten minutes.

Now, let's imagine one more thing: we have perfect testing prowess; that if there's a problem in an area that we're testing, we'll find it, and that we'll never enter a bogus report, either. Yes, this is a thought experiment.

One day we come into work, and we're given three modules to test.

The morning session is taken up with Module A, from Development Team A. These people are amazing, hyper-competent. They use test-first programming, and test-driven design. They work closely with us, the testers, to design challenging unit checks, scriptable interfaces, and log files. They use pair programming, and they review and critique each other's work in an egoless way. They refactor mercilessly, and run suites of automated checks before checking in code. They brush their teeth and floss after every meal; they're wonderful. We test their work diligently, but it's really a formality because they've been testing and we've been helping them test all along. In our 90-minute testing session, we don't find any problems. That means that we've performed 45 micro-sessions, and have therefore obtained 45 units of test coverage.

Module	Bug Investigation and Reporting (time spent on tests that find bugs)	Test Design and Execution (time spent on tests that don't find bugs)	Total Tests
A	0 minutes (no bugs found)	90 minutes (45 tests)	45

The first thing after lunch, we have a look at Team B's module. These people are very diligent indeed. Most organizations would be delighted to have them on board. Like Team A, they use test-first programming and TDD, they review carefully, they pair, and they collaborate with testers. But they're human. When we test their stuff, we find a bug very occasionally; let's say once per session. The test that finds the bug takes two minutes; investigation and reporting of it takes a further eight minutes. That's ten minutes altogether. The rest of the time, we don't find any problems, so that leaves us 80 minutes in which we can run 40 tests. Let's compare that with this morning's results.

Module	Bug Investigation and Reporting (time spent on tests that find bugs)	Test Design and Execution (time spent on tests that don't find bugs)	Total Tests
A	0 minutes (no bugs found)	90 minutes (45 tests)	45
B	10 minutes (1 test, 1 bug)	80 minutes (40 tests)	41

After the afternoon coffee break, we move on to Team C's module. Frankly, it's a mess. Team C is made up of nice people with the best of intentions, but sadly they're not very capable. They don't work with us at all, and they don't test their stuff on their own, either. There's no pairing, no review, in Team C. To Team C, if it compiles, it's ready for the testers. The module is a dog's breakfast, and we find bugs practically everywhere. Let's say we find eight in our 90-minute session. Each test that finds a problem costs us 10 minutes, so we spent 80 minutes on those eight bugs. Every now and again, we happen to run a test that doesn't find a problem. (Hey, even dBase IV occasionally did something right.) Our results for the day now look like this:

Module	Bug Investigation and Reporting (time spent on tests that find bugs)	Test Design and Execution (time spent on tests that don't find bugs)	Total Tests
A	0 minutes (no bugs found)	90 minutes (45 tests)	45
B	10 minutes (1 test, 1 bug)	80 minutes (40 tests)	41
C	80 minutes (8 tests, 8 bugs)	10 minutes (5 tests)	13

Because of all the bugs, Module C allows us to perform thirteen micro-sessions in 90 minutes. Thirteen, where with the other modules we managed 45 and 41. Because we've been investigating and reporting bugs, there are 32 micro-sessions, 32 units of coverage, that we haven't been able to obtain on this module. If we decide that we need to perform that testing (and the module's overall badness is consistent throughout), we're going to need at least three more sessions to cover it. Alternatively, we could stop testing now, but what are the chances of a serious problem lurking in the parts of the module we haven't covered? So, the first thing to observe here is:

Lots of bugs means reduced coverage, or slower testing, or both.

There's something else that's interesting, too. If we are being measured based on the number of bugs we find (exactly the sort of measurement that will be taken by managers who don't understand testing), Team A makes us look awful—we're not finding any bugs in their stuff. Meanwhile, Team C makes us look great in the eyes of management. We're finding lots of bugs! That's good! How could that be bad?

On the other hand, if we're being measured based on the test coverage we obtain in a day (which is exactly the sort of measurement that will be taken by managers who count test cases; that is, managers who probably have an even more damaging model of testing than the managers in the last paragraph), Team C makes us look terrible. "You're not getting enough done! You could have performed 45 test cases today on Module C, and you've only done 13!" And yet, remember that in our scenario we started with the assumption that, no matter what the module, we always find a problem if there's one there. That is, there's no difference between the testers or the testing for each of the three modules; it's solely the condition of the product that makes all the difference.

This is the first in a pair of posts. Let's see what happens tomorrow.

"Merely" Checking or "Merely" Testing

michael.a.bolton@gmail.com (Michael Bolton http://www.developsense.com) — Tue, 10 Nov 2009 18:14:00 +0000

The distinction between testing vs. checking got a big boost recently from James Bach at the Øredev conference in Malmö, Sweden. But a recent tweet by Brian Marick, and a recent conversation with a colleague have highlighted an issue that I should probably address.

My colleague suggested that somehow I may have underplayed the significance or importance or the worth of checking. Brian's tweet said,

"I think the trendy distinction between "testing" and "checking" is a power play: which would you preface with "mere"? http://bit.ly/2Cuyj

As a consequence, I was worried that I had ever said "mere checking" or "merely checking" in one of my blog postings or on Twitter, so I researched it. Apparently I had not; that was a relief. However, the fact that I was suspicious even of myself suggests that some maybe I need to clarify something.

The distinction between testing and checking is a power play, but it's not a power play between (say) testers and programmers. It's a power play between the glorification of mechanizable assertions over human intelligence. It's a power play between sapient and non-sapient actions.

Recall that the action of a check has three parts to it. Part one is an observation of a product. Part two is a decision rule, by which we can compare that empirical observation of the product with an idea that someone had about it. Part three is the setting of a bit (pass or fail, yes or no, true or false) that represents the non-sapient application of both the observation and the decision rule. Note, too, that this means that a check can be performed by one of two agencies: 1) a machine. 2) A sufficiently disengaged human; that is, a human who has been scripted to behave like a machine, and who has for whatever reason accepted that assignment.

So checks can be hugely important. Checks are the means by which a programmer, engaged in test-driven development, tests his idea. Checks are a valuable product (a by-product, some would say) of test-driven development. Checks are change detectors, tools that allow programmers to refactor with confidence. Checks built into continuous integration are mechanisms to make sure that our builds can work well enough to be tested—or, if we're confident enough in the prior quality of our testing, can work well enough to be deployed. Checks tend to shortens the loop between the implementation of an idea and the disovery of a problem that the checks can detect, since the checks are typically designed and run (a lot, iteratively) by the person doing the implementation. Checks tend to speed up certain aspects of the post-programmer testing of the product, since good checks will find the kind dopey, embarrassing errors that even the best programmers can make from time to time. The need for checks sometimes (alas, not always) prompts us to create interfaces that can be used by programmers or testers to aid in later exploration.

Checking represents the rediscovery of techniques that were around at least in 1957. "The first attack on the checkout problem may be made before coding has begun." D. D. McCracken, Digital Computer Programming, 1957 (Thanks to Ben Simo for inspiring me to purchase a copy of this book.) In 2007, I had dinner with Jerry Weinberg and Josh Kerievsky. Josh asked Jerry if he did a lot of unit testing back in the day. Jerry practically did a spit-take, saying "Yes, of course. Computer time was hugely expensive, but we programmers were cheap. Getting the program right was really important, so we had to test a lot." Then he added something that hadn't occurred to me. "There was another reason, too. Apart from everything else, we tested because the machinery was so unreliable. We'd run a test program, then run the program we wrote, then run the test program again to make sure that we got the same result the second time. We had to make sure that no tubes had blown out."

So, in those senses, checking rocks. Checking has always rocked. It seems that in some places, people forgot how much it rocks, and that the Agilists have rediscovered them.

Yet it's important to note that checks on their own don't deliver value unless there's sapient engagement with them. What do I mean by that?

As James Bach says here, "A sapient process is any process that relies on skilled humans." Sapience is the capacity to act with human intelligence, human judgment, and some degree of human wisdom.

It takes sapience to recognize the need for a check—a risk, or a potential vulnerability. It takes sapience—testing skill—to express that need in terms of a test idea. It takes sapience—more test design skill—to express that test idea in terms of a question that we could ask about the program. Sapience—in terms of testing skill, and probably some programming skill—is needed to frame that question as a yes-or-no, true-or-false, pass-or-fail question. Sapience, in the form of programming skill, is required to turn that question into executable code that can implement the check (or, far more expensively and with less value, into a test script for execution by a non-sapient human). We need sapience—testing skill again—to identify an event or condition that would trigger some agency to perform the check. We need sapience—programming skill again—to encode that trigger into executable code so that the process can be automated.

Sapience disappears while the check is being performed. By definition, the observation, the decision rule, and the setting of the bit all happen without the cognitive engagement of a skilled human.

Once the check has been performed, though, skill comes back into the picture for reporting. Checks are rarely done on their own, so they must be aggregated. The aggregation is typically handled by the application of programming skill. To make the outcome of the check observable, the aggregated results must be turned into a human-readable report of some kind, which requires both testing and programming skill. The human observation of the report, intake, is by defintion a sapient process. Then comes interpretation. The human ascribes meaning to the various parts of the report, which requires skills of testing and of critical thinking. The human ascribes significance to the meaning, which again takes testing and critical thinking skill. Sapient activity by someone—a tester, a programmer, or a product owner—is needed to determine the response. Upon deciding on significance, more sapient action is required—fixing the application being checked (by the production programmer); fixing or updating the check (by the person who designed or programmed the check); adding a new check (by whomever might want to do so) or getting rid of the check (by one or more people who matter, and who have decided that the check is no longer relevant).

So: the check in and of itself is relatively trivial. It's all that stuff around the check—the testing and programming and analysis activity—that's important, supremely important. And as is usual with important stuff, there are potential traps.

The first trap is that it might be easy to do any of the sapient aspects of checking badly. Since the checks are at their core software, there might be problems in requirements, design, coding, or interpretation, just as there might be with any software.

The second trap is that it can be easy to fall asleep somewhere between the report and interpretations stages of the checking process. The green bar tells us that All Is Well, but we must be careful about that. All is well with respect to the checks that we've programmed is a very different statement. Red tends to get our attention, but green is an addictive and narcotic colour. A passing test is another White Swan, confirmation of our existing beliefs, proof by induction. Now, we can't live without proof by induction, but induction can't alert us to new problems. Millions of repeated tests, repeated thousands of times, don't tell us about the bugs that elude them. We only need one Black Swan to bump into a devastating effect.

The third trap is that we might believe that checking a program is all there is to testing it. Checking done well incorporates an enormous amount of testing and programming skill, but some quality attributes of a program are not machine-decidable. Checks are the kinds of tests that aren't vulnerable to the halting problem.Someone on a mailing list once said, "Once all the (automated) acceptance test pass (that is, all the checks), we know we're done." I liked Joe Rainsberger's reply, "No, you're not done; you're ready to give it to a real tester to kick the snot out of it." That kicking is usually expressed with greater emphasis on exploration, discovery, and investigation, and rather less on confirmation, verification, and validation.

The fourth trap is a close cousin of the third trap: at certain points, we might pay undue attention to the value of checking with respect to its cost. Cost vs. value is a dominating problem with any kind of testing, of course. One of the reasons that the Agile emphasis on testing remains exciting is that excellent checking lowers the cost of testing, and both help to defend the value of the program. Yet checks may not be Just The Thing for some purposes. Joe has expressed concerns in his series Integration Tests are a Scam, and Brian Marick did too, a while ago, An Alternative to Business-Facing TDD. I think they're both making important points here, thinking of checks as a means to an end, rather than as a fetish.

Fifth: upon noting the previous four traps (and others), we might be tempted to diminish the value of checking. That would be a mistake. Pretty much any program is made more testable by someone removing problems before someone else sees them. Every bug or issue that we find could trigger investigation, reporting, fixing, and retesting, and that gives other (and potentially more serious) problems time to hide. Checking helps to prevent those unhappy discoveries. Excellent checking (which incorporates excellent testing) will tend to reduce the number of problems in the product at any given time, and thereby results in a more testable program. James Bach points out that a good manual test could never be automated (he'd say "sapient" now, I believe). But note, in that same post that he says, that "if you can truly automate a manual test, it couldn’t have been a good manual test", and "if you have a great automated test, it’s not the same as the manual test that you believe you were automating". The point is that there are such things as great automated tests, and some of them might be checks.

So the power play is over which we're going to value: the checks ("we have 50,000 automated tests") or the checking. Mere checks aren't important; but checking—the activity required to build, maintain, and analyze the checks—is. To paraphrase Eisenhower, with respect to checking, the checks are nothing; the checking is everything. Yet the checking isn't everything; neither is the testing. They're both important, and to me, neither can be appropriately preceded with "mere", or "merely".

There's one exception, though: If you're only doing one or the other, it might be important to say, "You're merely been testing the program; wouldn't you be better off checking it, too?" or "That program hasn't been tested; it's merely been checked."

Testing, Checking, and Convincing the Boss to Explore

michael.a.bolton@gmail.com (Michael Bolton http://www.developsense.com) — Tue, 10 Nov 2009 04:04:00 +0000

How is it useful to make the distinction between testing and checking? One colleague (let's call him Andrew) recently found it very useful indeed. I've been asked not to reveal his real name or his company, but he has very generously permitted me to tell this story.

He works for a large, globally distributed company, which produces goods and services in a sector not always known for its nimbleness. He's been a test manager with the company for about 10 years. He's had a number of senior managers who have allowed him and his team to take an exploratory approach, almost a skunkworks inside the larger organization. Rather than depending on process manuals and paperwork, he manages by direct interaction and conversation. He hires bright people, trains them, and grants them a fairly high degree of autonomy, balanced by frequent check-ins.

Recently, on a Thursday the relatively new CEO came to town and held an all-hands meeting for Andrew's division. Andrew was impressed; the CEO seemed genuinely interested in cutting bureaucracy and making the organization more flexible, adaptable, and responsive to change. After the CEO's remarks, there was a question-and-answer period. Andrew asked if the company would be doing anything to make testing more effective and more efficient. The CEO seemed curious about that, and jotted down a note on a piece of paper. Andrew was given the mandate of following up with the VP responsible for that area.

Late that afternoon, Andrew called me. We chatted for a while on the phone. He hadn't read my series on testing vs. checking, but he seemed intrigued. I suggested that he read it, and that we get together and talk about it.

As luck would have it, there was occasion to bring a few more people into the picture. That weekend, we had a timely conversation with Fiona Charles who reminded us to focus the issue of risk. Rob Sabourin, happened to be visiting on Saturday evening, so he, Andrew, and I sat down to compose a letter to the VP. Aside from changing the names that would identify the parties involved, this is an unedited version what we came up with:

[Our Letter]

Dear [Madam VP]...

[Mr. CEO] asked me to send you this email as a follow up to a question that I posed during his recent trip to the [OurTown] office on [SomeDate] on the current state of the testing at [OurCompany] and how our testing effectiveness should be improved.

The [OurTown]-based [OurDivision] test team has been very successful in finding serious issues with our products with a fairly small test team using exploratory test approaches. As an example, a couple of weeks ago one of my testers found a critical error in an emergency fix within his two days of exploratory testing in a load that had passed four person-weeks of regression testing (scripted checking) by another team.

Last week a Project Lead called me and asked if my team could perform a regression sweep on a third party delivery. I replied that we could provide the requested coverage with two person-days of effort without disrupting our other commitments. He seemed surprised and delighted. He had come to us because [OurCompany]'s typical approach yielded a four-to-six person-week effort which would have caused a delay in the project's subsequent release.

Our experience using exploratory testing in [OurDivision] has demonstrated improved flexibility and adaptability to respond to rapid changes in priorities.

Testing is not checking. Checking is a process of confirmation, validation, and verification in which we are comparing an output to a predicted and expected result. Testing is something more than that. Testing is a process of exploration, discovery, investigation, learning, and analysis with the goal of gathering information about risks, vulnerabilities, and threats to the value of the product. The current effectiveness of many groups' automated scripts is quite excellent, yet without supplementing these checks with "brain-engaged" human testing we run the risk of serious problems in the field impacting our customers, and the consequential bad press that follow these critical events.

At [OurCompany] much of our "testing" is focused on checking. This has served us fairly well but there are many important reasons for broadening the focus of our current approach. While checking is very important, it is vulnerable to the "pesticide paradox". As bacteria develop a resistance to antibiotics, software bugs are capable of avoiding detection by existing tests (checks), whether executed once or repeated over and over. In order to reduce our vulnerability to field issues and critical customer incidents, we must supplement our existing emphasis on scripted tests (both manual and automated) with an active search for new problems and new risks.

There are several strong reasons for integrating exploratory approaches into our current development and testing practices:

Scripted tests are perceived to be important for compliance with [regulatory] requirements. They are focused on being repeatable and defensible. Mere compliance is insufficient—we need our products to work.

Scripted checks take time and effort to design and prepare, whether they are run by machines or by humans. We should focus on reducing preparation cost wherever possible and reallocating that effort to more valuable pursuits.

Scripted checks take far more time and effort to execute when performed by a human than when performed by a machine. For scripted checks, machine execution is recommended over human execution, allowing more time for both human interaction with the product, and consequent observation and evaluation.

Exploratory tests take advantage of the human capacity for recognizing new risks and problems.

Exploratory testing is highly credible and accountable when done well by trained testers. The findings of exploratory tests are rich, risk-focused, and value-centered, revealing far more knowledge about the system than simple pass/fail results.

The quality of exploratory testing is based upon the skill set and the mindset of the individual tester. Therefore, I recommend that testers and managers across the organization be trained in the structures and disciplines of excellent exploratory testing. As teams become trained, we should systematically introduce exploratory sessions into the existing testing processes, observing and evaluating the results obtained from each approach.

I have been actively involved in improving testing, in general, outside of [OurCompany]. I am on the board of a testing association and I have been attending, organizing and facilitating meetings of testers for many years.

During this time, I have been exposed to much of the latest developments in software testing and I have led the implementation of Session Based Exploratory Testing within my department. In addition, over the past four years, I have been providing instruction in software testing both to the testers within my business unit and to companies outside of [OurCompany].

I look forward to the opportunity to talk with you about this further.

[/Our Letter]

Now, I thought that was pretty strong. But the response was far more gratifying than I expected. Andrew sent the message on Sunday afternoon. The VP responded by 8:45am on Monday morning. Her reply was in my mailbox before 10:00am. The reply read:

[The VP's Reply]

Dear Andrew

Thanks very much for the email. I find this very intriguing! I believe the distinction you make between testing and checking is quite insightful and I would like to connect with you to see how we can build these concepts and techniques into our quality management services as well as my central team verification tests. I will get a call together with [Mr. Bigwig] and [Mr. OtherBigwig] so that we can figure out the best way to incorporate your ideas. Again, many thanks!!

[/The VP's Reply]

A couple of key points:

The letter was much stronger thanks to collaboration. Any one of the four of us could have written a good letter; the result was better than any of us could have done on our own.

The letter is sticky, in the sense that Chip and Dan Heath talk about in their book Made to Stick: Why Some Ideas Survive and Others Die. It's not a profound book, but it contains some useful points to ponder. The letter starts with two stories that are simple, unexpected, concrete, credible, and emotional (remember, the product manager was surprised and delighted). Those initials can be rearranged to SUCCES, which is the mnemonic that the Heaths use for successful communication.

The testing vs. checking distinction is simpler and memorable than "exploratory approaches" vs. "confirmatory scripted approaches". The explanation is available (and in most cases necessary), but "testing" and "checking" roll off the tongue quickly after the explanation has been absorbed.

We managed to hit some of the most important aspects of good testing: cost vs. value, risk focus, diversification of approaches, flexibility and adaptability, and rapid service to the larger organization.

After reviewing this post, Andrew said, "I like the post a lot. Let's hope we end up helping a lot of people with it." Amen. You are, of course, welcome to use this letter as a point of departure for your own letter to the bigwigs. If you'd like help, please feel free to drop me a line.

Comment on a Not-So-Good Article on Exploratory Testing

michael.a.bolton@gmail.com (Michael Bolton http://www.developsense.com) — Fri, 30 Oct 2009 08:59:00 +0000

An article from a while back on StickyMinds entitled How To Choose Between Scripted and Exploratory Testing refers to a bunch of factors in making choices between scripted testing and exploratory testing. The problems start early: "As a test manager or engineer, you may be considering whether or not to use exploratory testing on your next project."

If you're not planning on investigating any problems that you find, I suppose that you could choose not to use exploratory testing (investigating a bug is not a scripted or scriptable process). If you don't allow the tester to control his or her actions, or to feed information from the last test into the thought process and action behind the next test, then you could get away without taking an exploratory approach. If the testing ideas were all declared and written down at the beginning of the project, you could successfully avoid exploratory testing. And if you used an all-checking strategy, maybe exploratory testing wouldn't be an approach you'd take. But would you really want all of your testing to be confirmatory? All validation and verification? All checking to make sure that existing beliefs are correct, rather than probing the product to disconfirm those beliefs, with the goal of showing that they're inaccurate or incomplete?

This isn't the worst article ever written on exploratory testing. It is, however, typical of a certain kind of bad article. It's full of assertions that aren't supported in the view of exploratory testing that a bunch of us (including Jon Bach, who, at the bottom of the article provides an unusually stern comment by his usual standards) have been developing for years. Let's have a look at some of the points. Quotes from the original article are in italics.

Tester domain knowledge...If a tester does not understand the system or the business processes, it would be very difficult for him to use, let alone test, the application without the aid of test scripts and cases.

A few years back, a former co-worker contracted me to to document a network analysis tool that was being rebranded, fixed, and upgraded by his new company. The only documentation available was not only weak but also a couple of versions out of date. In addition, I didn't know much about network analysis tools at all. I reviewed some literature on the topic, and I interviewed the company's lead programmer. But for me, the most rapid, relevant, and powerful learning came from interacting with the product, figuring out how it worked (and, at several points, how it appeared to fail), and building the story of the product in my mind and in the documentation that I created. Without that interaction with the product, my learning would have not have been rooted in experience, and would have been much slower. And the documentation that I produced was deemed excellent by the client.

How can I test a product in a domain that I don't know much about? The answer is that through testing, I learn, and I learn far more rapidly than by other means. Anyone learning a complex cognitive activity learns far more, far more quickly, in the doing of it than in the reading about it. Preschool kids do the same thing. Watch them; then watch how they tend to slow down and follow the schools' processes instead of their own.

System complexity...End-to-end testing can be accomplished with exploratory testing; however, the capabilities and skill sets required are typically those of a more experienced test engineer.

People often argue that exploratory testing needs capable, experienced, and skilled testers. I agree. The argument suggests (and often states outright) that scripted testing doesn't need capable, experienced, and skilled testers. To some degree that's true; when the scripted action is checking, by definition skill and sapience aren't required for the moment of the check. But sapience and skill are required for the design and construction of the check, and for the analysis and interpretation of the results, so the argument that scripted testing doesn't need skilled testers doesn't hold water if you want to do scripted testing well. Indeed, good testing of any kind requires skill. If you or your testers are genuinely unskilled, training, coaching, mentoring, and a fault-tolerant environment that fosters learning will help to build skills quickly.

I'm also puzzled by the argument in another way: the (claimed) lack of a need for skilled testers is often presented as a virtue of scripted testing. This is like saying that the lack of a requirement for medical training is a virtue of quack medicine.

Level of documentation. Scripted testing generally flows from business requirements documents and functional specifications. When these documents do not exist or are deficient, it is very difficult to conduct scripted testing in a meaningful way. The shortcuts that would be required, such as scripting from the application as it is built, are accomplished as efficiently using exploratory testing.

Actually, there are several ways to do scripted testing in a meaningful way without business requirements documents or functional specifications being present or perfect. A standout example is test-driven development, in which checks are developed prior to developing the application code. Another: unit tests, in which checks are prepared at some point after the application code has been developed. Both forms of checks support refactoring. Another example: many Agile development shops use Fitnesse to develop, explore, and discover many of the requirements of the product, and to create scripted checks as development proceeds.

In fact, no one of whom I am aware has ever seen perfect requirements documents; they're always deficient in some way for some person's purpose, and always to some degree that's intentional. It's infeasible and wasteful to document every assumption; for programmers and testers with appropriate judgment and skill, it's also somewhat insulting. Thus, whether developing either scripted or exploratory test ideas, we should treat imperfect documents as a default assumption.

Timeframes and deadlines.The lead-in time to test execution determines whether you can conduct test design before execution. When there is little or no time, you might at least need to start with exploratory testing while documented tests are prepared after the specifications become available.

Here is the implication, once again, that exploratory tests aren't documented. Yet exploratory ideas can be documented in advance, in the form of checklists or as charters. Existing data files—a form of documentation—can be available in advance. Exploratory tests can be guided by marked-up diagrams, by user documentation, or even by a functional specification. Exploratory testing can be informed by any kind of documentation you like. One key distinction between exploratory and scripted approaches is not whether documentation is available in advance; with ET the difference is that the tester, rather than the document, is the primary agency driving the design and execution of the test. Another key distinction is that in an exploratory approach, detailed documentation of the tester's actions tends to be de-emphasized prior to test execution, and tends to be produced during or very shortly after testing.

Available resources. The total number of person-days of effort can determine which approach you should take. Formal test documentation has significant overhead that can be prohibitive where resources and budgets are tight.

Well, at least maybe we agree on something. But the number of person-days of effort doesn't determine which approach you choose; people do that. It's unclear here as to whether the person-days are a given, or whether the test manager gets to choose. Moreover, I wouldn't use the word "formal" here, since documentation associated with exploratory approaches can be quite formal (look at session-based test management as an example). The word I'd use instead of "formal" above is "excessive" or "wasteful" or "overly detailed". I'd also suggest that it's fairly rare for testers to feel as though resources and budgets are ample; most testers I speak with maintain that resources and budgets are always tight, for them.

Skills required. The skill sets of your team members can affect your choices. Good test analysts may not necessarily be effective exploratory testers and vice versa. A nose for finding bugs quickly and efficiently is not a skill that is easily learned...

Perhaps not. James Bach, Jon Bach, James Lyndsay, and I (to name but four) have a lot of experience training testers, and our experience is that the capacity to find bugs can be learned more quickly than many people believe. But habitutate testers to working from detailed manual test scripts and I'll guarantee that they don't develop skill quickly. In fact, a huge part of learning to find problems quickly lies in the tester being given the freedom and responsibility to explore and to develop his own mental models of the product.

Verification. Verification requires something to compare against. Formal test scripts describe an expected outcome that is drawn from the requirements and specifications documents. As such, we can verify compliance. Exploratory testing is compared to the test engineer’s expectations of how the application should work.

Many kinds of testing—not just verification—require something to compare against. That "something" is called an oracle, a principle or mechanism by which we recognize a problem. The claim here is that scripted testing is a good thing because we can verify "compliance" (to what?) based on requirements and specifications documents. Actually, what we'd be verifying here is consistency between some point in a test script and a specific claim made in one of the documents. There are two problems here. One is the problem of inattentional blindness; focusing a tester's attention on a single observation drives the tester's attention away from other possible observations that might represent an issue with the product. The second problem relates to the fact that requirements and specification documents (as above) can be presumed to contain errors, misinterpretations, and outdated information.

Exploratory approaches defend against these two problems by their emphasis on the tester's skill set and mindset. Rather than depending upon a single oracle (that of consistency with a specification), an exploratory approach emphasizes apply several oracles, including consistency with the product's history, with the image the development organization wants to project, with comparable products, with reasonable user expectations, with the intended purpose of the product, with other elements within the product, and with relevant standards, regulations, or laws. Claims—statements in the requirement documents—are taken seriously by the exploring tester, of course, but the other kinds of oracles provide a rich set of possible comparisons, rather than the relatively impoverished single set provided by a script. By keeping the tester cognitively engaged and under her own control, we lessen the risk of script-induced inattentional blindness. Statements like "Exploratory testing is compared to the test engineer’s expectations of how the application should work" are not only inaccurate, but also trivialize the complex set of heuristics that skilled testers bring to the game.

At this point, I'm in agreement with Jon Bach. Refuting the rest of the article would take more time than I've got at the moment. So, to keep the curious and eager people occupied:

Web Resources

Evolving Understanding of Exploratory Testing (Bolton)
A Tutorial in Exploratory Testing (Kaner)
The Nature of Exploratory Testing (Kaner)
The Value of Checklists and the Danger of Scripts: What Legal Training Suggests for Testers
Exploratory Testing Dynamics (James Bach, Jon Bach, Michael Bolton)
General Functionality and Stability Test Procedure (for Microsoft Windows 2000 Application Certification) (James Bach)
Experiments in Problem Solving (Jerry Weinberg)
Collaborative Discovery in a Scientific Domain (Okada and Simon)
- a study of collaborative problem-solving. Notice the subjects are testing software.

Books

Exploring Science: The Cognition and Development of Discovery Processes (David Klahr)
Plans and Situated Actions (Lucy Suchman)
Play as Exploratory Learning (Mary Reilly)
Exploratory Research in the Behavioral Sciences
Naturalistic Inquiry
How to Solve It (George Polya)
Simple Heuristics That Make Us Smart (Gerg Gigerenzer)
Blink (Malcolm Gladwell)
Gut Feelings (Gerg Gigerenzer)
Sensemaking in Organizations (Karl Weick)
Cognition in the Wild (Edward Hutchins)
The Social Life of Information (Paul Duguid and John Seely Brown)
A System of Logic and Ratiocination (John Stuart Mill)
Sciences of the Artificial (Herbert Simon)

Yes, learning about exploratory testing in a non-trivial way might take some practice and study. That goes for anything worth doing expertly, doesn't it?

Maturity Models Have It Backwards

michael.a.bolton@gmail.com (Michael Bolton http://www.developsense.com) — Wed, 28 Oct 2009 01:43:00 +0000

At a couple of recent conferences, some people have asked me about one "maturity" model or another. As one of the few people who has read the CMMI book from cover to cover, here's what I think: In process-speak, the notion of maturity is backwards.

A mature entity, in biology, is one that can survive and thrive without parental support. A mature being is one that has achieved an age and stage where it can reproduce, mutate, and diversify. In general, diversity and sustainability come from interaction with other mature creatures. An animal or plant attains and sustains maturity either by adapting to its environment, or by being adapted to it already, but no species is adapted to all environments.

A mature person is one who is highly conscious of when it's appropriate to follow rules and when to break them. A mature person is largely self-guided. Only in exceptional circumstances does a mature person need to refer to or appeal to a rulebook at all. In such cases, the issue is that someone believes that the rulebook isn't working, and in such cases, consensus between mature individuals and organizations—not the rulebook itself—makes the determination as to what should happen next. Mature people know that rulebooks need to be interpreted.

We don't consider a person mature when he says or does the same thing over and over again, when he answers by rote, when he appeals to authority, or when he goes through the motions. We consider a person mature when he is able to think and perform independently, when he behaves responsibly and respectfully towards others, and when he accepts responsibility for his actions. We also don't mind when a mature person screws around every once in a while, as long as little or no harm comes of it.

We generally think of mature people as being relaxed and easy-going, rather than rigid and uptight. Mature people who can't get their way immediately don't stamp their feet, shout, or hold their breath until they turn blue. Mature people recognize the possibility that other people's values, actions, languages, cultures, and means of expression are worthy of respect. Mature people question themselves at least as quickly and as closely as they question others. Mature people are forgiving and fault-tolerant, recognizing that immature people often need to make mistakes in order to learn. Mature parents don't scare the kids, don't yell at them, don't try to protect them from every peril. Mature parents provide a supportive environment where kids can make mistakes and learn from them.

One more thing about mature beings: they eventually get old, and they die, to be replaced by other mature beings.

So what does all this mean for "process maturity"?

If maturity means the same thing for processes as for other living things, a genuinely mature process, whether for individuals or for groups, should incorporate freedom, responsibility, diversity, adaptability, and self-sufficiency. A genuinely mature process shouldn't emphasize repeatability as a virtue unto itself, but rather as something set up to foster appropriate variation, sustainability, and growth. A mature process should encourage risk-taking and mistakes while taking steps to limit the severity and consequence of the mistakes, because without making mistakes, learning isn't possible.

What do you think? How do process "maturity" models stack up?

Note: This post was inspired by Jerry Weinberg's writing, in particular the material on "maturity" in Quality Software Management, Volume 1: Systems Thinking. For an unusually sharp passage on the subject, see pages 20 and 21.

The Testers' Christmas Present

michael.a.bolton@gmail.com (Michael Bolton http://www.developsense.com) — Mon, 19 Oct 2009 21:10:00 +0000

So the holidays are coming up, and you're wondering what to get for your tester friends, or (if you're a tester) for your kids.

Let me be the first this season to recommend I Am A Bug, a perfectly charming little book by Robert Sabourin, and illustrated by his daughter Catherine, who was between 11 and 12 years old as the book was being published. It's been around for several years.

The secret is that I Am A Bug is a serious testing book, cleverly disguised as a children's book. It's one of the wisest books on testing I've ever read. Each page begins with a big message, illustrated and elaborated below. Here's a littl sample:

A bee sting may hurt a bit

The same bug can be found in different computer programs. In one program the bug may not cause much damage...

But it can kill you if you're allergic to bees.

...but in another program it could be fatal.

The whole book is online, at Rob's Web site, http://www.amibug.com. (The book is found under "Presentations".) It's fun to read there, but order the dead-tree version for a very reasonable price. You can get it from the usual online booksellers. Highly recommended.

When Do We Stop Testing? One More Sure Thing

michael.a.bolton@gmail.com (Michael Bolton http://www.developsense.com) — Tue, 13 Oct 2009 15:14:00 +0000

Not too long ago, I posted a list of stopping heuristics for testing. As usual, such lists are always subjective, subject to refinement and revision, and under scutiny from colleagues and other readers. As usual, James Bach is a harsh critic (and that's a compliment, not a complaint). We're still transpecting over some of the points; eventually we'll come out with something on which we agree.

Joe Harter, in his blog, suggests splitting "Pause That Refreshes" into two: "Change in Priorities" and "Lights are Off". The former kicks in when we know that there's still testing to be done, but something else is taking precedence. The latter is that we've lost our sense of purpose—as I suggested in the original post we might be tired, or bored, or uninspired to test and that a break will allow us to return to the product later with fresh eyes or fresh minds. Maybe they're different enough that they belong in different categories, and I'm thinking that they are. Joe provides a number of examples of why the lights go out; one feels to me like "customary conclusion", another looks like "Mission Accomplished". But his third point is interesting: it's a form of Parkinson's Law, "work expands to fill the time available for its completion". Says Joe, "The test team might be given more time than is actually necessary to test a feature so they fill it up with old test cases that don’t have much meaning." I'm not sure how often people feel as though they have more time than they need, but I am sure that I've seen (been in) situations where people seem to be bereft of new ideas and simply going through the motions. So: if that feeling comes up, one should consider Parkinson's Law and a Pause That Refreshes. Maybe there's a new one there. But as Joe himself points out, "In the end it doesn’t matter if you use [Michael's] list, my list or any list at all. These heuristics are rules of thumb to help thinking testers decide when testing should stop. The most important thing is that you are thinking about it."

For sure, however, there is a glaring omission in the original list. Cem Kaner pointed it out to me—and that shouldn't have been necessary, because I've used this heuristic myself. It focuses on the individual tester, but it might also apply to a testing or development team.

Mission Rejected. We stop testing when we perceive a problem for some person—in particular, an ethical issue—that prevents us from continuing work on a given test, test cycle, or development project.

Would you continue a test if it involved providing fake test results? Lying? Damaging valuable equipment? Harming a human, as in the Milgram Experiment or the Stanford Prison Experiment? Maybe the victim isn't the test subject, but the client: Would you continue a test if you believed that some cost of what you were doing—including, perhaps, your own salary—were grossly disproportionate to the value it produced? Maybe the victim is you: Would you stop testing if you believed that the client wasn't paying you enough?

The consequences of ignoring this heuristic can be dire. Outside the field of software testing, but in testing generally, a friend of mine worked in a science lab that did experiments on bone regeneration. The experimental protocol involved the surgical removal of around one inch of bone from both forelegs of a dog (many dogs, over the course of the research), treating one leg as an experiment and the other as a control. Despite misgivings, my friend was reasonably convinced of the value of the work. Later, when he found out that these experiments had been performed over and over, and that no new science was really being done, he suffered a nervous breakdown and left the field. Sometimes testing doesn't have a happy ending.

Context-free Questions For Testing and Checking

michael.a.bolton@gmail.com (Michael Bolton http://www.developsense.com) — Wed, 30 Sep 2009 20:42:00 +0000

After a presentation on exploratory approaches and on testing vs. checking yesterday, a correspondent and old friend writes:

Although the presentation made good arguments for exploratory testing, I am not sure a small QA department can spare the resources unless a majority of regression checking can be moved to automation. Particularly in situations with short QA cycles.

(Notice that he and I are using "testing" and "checking" in this specific way.)

Any time someone makes an observation about what is or isn't possible, irrespective of the kind of testing (or checking) that they're doing, it suggests some questions for the testing, programming, and management teams. I'd ask my old friend

1) How much checking do you need to do?

2) What, specifically, suggests that checking needs to be done? What happens when you do it? What doesn't happen when you do it? What happens when you don't do it? What doesn't happen when you don't do it?

3) What specifically, might suggest that the testers are the best people to do the checking? What, specifically, might suggest that they aren't the best people to do it?

4) Where do your testers spend their time? When you speak with the people who are actually testing, do they feel the time that they're spending on checking is worthwhile? Do they have things to say about what slows down testing (or checking)?

5) What are the risks that checking addresses well? What risks are not addressed well by checking?

These are open questions that all teams can ask, regardless of the approach they're using now. Feel free to replace the word "checking" with "testing", and vice versa, wherever you like.

I encourage and, when asked, help people to ask and answer these questions, and others like them. I have no specific answers from the outset; I don't know you, and I don't know your context. But you do. Maybe the questions can be helpful to you. I hope so.

A Letter To The Programmer

michael.a.bolton@gmail.com (Michael Bolton http://www.developsense.com) — Tue, 29 Sep 2009 19:51:00 +0000

This is a letter that I would not show to a programmer in a real-life situation. I've often thought of bits of it at a time, and those bits come up in conversation occasionally, but not all at once.

This is based on an observation of the chat window in Skype 4.0.0.226.

Dear Programmer,

I discovered a bug today. I'll tell you how I found it. It's pretty easy to reproduce. There's this input field in our program. I didn't know what the intended limit was. It was documented somewhere, but that part of the spec got deleted when the CM system went down last week. I could have asked you, but you were downstairs getting another latte.

Plus, it's really quick and easy to find out empirically; quicker than looking it up, quicker than asking you, even if you were here. There's this tool called PerlClip that allows me to create strings that look like this

*3*5*7*9*12*15*18*21*24*27*30*33*36*39*42*45*48*51*54*57*60*...

As you'll notice, the string itself tells you about its own length. The number to the left of each asterisk tells you the offset position of that asterisk in the string. (You can use whatever character you like for a delimiter, including letters and numbers, so that you can test fields that filter unwanted characters.)

It takes a handful of keystrokes to generate a string of tremendous length, millions of characters. The tool automatically copies it to the Windows clipboard, whereupon you can paste it into an input field. Right away, you get to see the apparent limit of the field; find an asterisk, and you can figure out in a moment exactly how many characters it accepts. It makes it easy to produce all kinds of strings using Perl syntax, which saves you having to write a line of Perl script to do it and another few lines to get it into the clipboard. In fact, you can give PerlClip to a less-experienced tester that doesn't know Perl syntax at all (yet), show them a few examples and the online help, and they can get plenty of bang for the buck. They get to learn something about Perl, too. This little tool is like a keychain version of a Swiss Army knife for data generation. It's dead handy for analyzing input constraints. It allows you to create all kinds of cool patterns, or data that describes itself, and you can store the output wherever you can paste from the clipboard. Oh, and it's free.

You can get a copy of PerlClip here, by the way. It was written by James Bach and Danny Faught. The idea started with a Perl one-liner by Danny, and they build on each other's ideas for it. I don't think it took them very long to write it. Once you've had the idea, it's a pretty trivial program to implement. But still, kind of a cool idea, don't you think?

So anyway, I created a string a million characters long, and I pasted it into the chat window input field. I saw that the input field apparently accepted 32768 characters before it truncated the rest of the input. So I guess your limit is 32768 characters.

Then I pressed "Send", and the text appeared in the output field. Well, not all of it. I saw the first 29996 characters, and then two periods, and then nothing else. The rest of the text had vanished.

That's weird. It doesn't seem like a big deal, does it? Yet there's this thing called representativeness bias. It's a critical thinking error, the phenomenon that causes us to believe that a big problem always looks big from every angle, and that an observation of a problem with little manifestations always has little consequences.

Our biases are influenced by our world views. For example, last week when that tester found that crash in that critical routine, everyone else panicked, but you realized that it was only a one-byte fix and we were back in business within a few minutes. It also goes the other way, though: something that looks trivial or harmless can have dire and shocking consequences, made all the more risky because of the trivial nature of the symptom. If we think symptoms and problems and fixes are all alike in terms of significance, when we see a trivial symptom, no one bothers to investigate the problem. It's only a little rounding error, and it only happens on one transaction in ten, and it only costs half a cent at most. When that rounding error is multiplied over hundreds of transactions a minute, tens of thousands an hour... well you get the point.

I'm well aware that, as a test, this is a toy. It's like a security check where you rattle the doorknob. It's like testing a car by kicking the tires. And the result that I'm seeing is like the doorknob falling off, or the door opening, or a tire suddenly hissing. For a tester, this is a mere bagatelle. It's a trivial test. Yet when a trivial test reveals something that we can't explain immediately, it might be good idea to seek an explanation.

A few things occurred to me as possibilities.

The first one is that someone, somewhere, is missing some kind of internal check in the code. Maybe it's you; maybe it's the guy who wrote the parser downstream, maybe it's the guy that's writing the display engine. But it seems to me as though you figured that you could send 32768 bytes, someone else has a limit of 29998 bytes. Or 29996, probably. Well, maybe.

Maybe one of you isn't aware of the published limits of the third-party toolkits you're using. That wouldn't be the first time. It wouldn't necessarily be negligence on your part, either—the docs for those toolkits are terrible, I know.

Maybe the published limit is available, but there's simply a bug in one of those toolkits. In that case, maybe there isn't a big problem here, but there's a much bigger problem that the toolkit causes elsewhere in the code.

Maybe you're not using third-party toolkits. Maybe they're toolkits that we developed here. Mind you, that's exactly the same as the last problem; if you're not aware of the limits, or if there's a bug, who produced the code has no bearing on the behaviour of the code.

Maybe you're not using toolkits at all, for any given function. Mind you, that doesn't change the nature of the problems above either.

Maybe some downstream guy is truncating everything over 29996 bytes, placing those two dots at the end, and ignoring everything else, and and he's not sending a return value to you to let you know that he's doing it.

Maybe he is sending you a return value, but the wrong one.

Maybe he's sending you a return value, and you're ignoring it.

Maybe he's sending you a return value, and you are paying attention to it, but there's some confusion about what it means and how it should be handled.

Maybe you're truncating the last two and a half kilobytes or so of data before you send it on, and we're not telling the user about it. Maybe that's your intention. Seems a little rude to me to do that, but to you, it works as designed. To some user, it doesn't work—as designed.

Maybe there's no one else involved, and it's just you working on all those bits of the code, but the program has now become sufficiently complex that you're unable to keep everything in your head. That stands to reason; it is a complicated program, with lots of bits and pieces.

Maybe you're depending on unit tests to tell you if anything is wrong with the individual functions or objects. But maybe nothing is wrong with any particular one of them in isolation; maybe it's the interaction between them that's problemmatic.

Maybe you don't have any unit tests at all.

Maybe you do have unit tests for this stuff. From right here, I can't tell. If you do have them, I can't tell whether your checks are really great and you just missed one this time, or if you missed a few, or if you missed a bunch of them, or whether there's a ton of them and they're all really lousy.

unit tests

For any one of the cases above, since it's so easy to test and check for these things, I would think that if you or anyone else knew about this problem, your sense of professionalism and craftsmanship would tell you to do some testing, write some checks, and fix it. After all, as Uncle Bob Martin said, you guys don't want us to find any bugs, right?

But it's not my place to say that. All that stuff is up to you. I don't tell you how to do your work; I tell you what I observe, in this case entirely from the outside. Plus it's only one test. I'll have to do a few more tests to find out if there's a more general problem. Maybe this is an aberration.

Now, I know you're fond of saying, "No user would ever do that." I think what you really mean is no user that you've thought of, and that you like, would do that on purpose. But it might be a thought to consider users that you haven't thought of, however unlikely they and their task might be to you. It could be a good idea to think of users that neither one of us like, such as hackers or identity thieves. It could also be important to think of users that you do like who would do things by accident. People make mistakes all the time. In fact, by accident, I pasted the text of this message into another program, just a second ago.

So far, I've only talked about the source of the problem and the trigger for it. I haven't talked much about possible consequences, or risks. Let's consider some of those.

A customer could lose up to 2770 bytes of data. That actually sounds like a low-risk thing, to me. It seems pretty unlikely that someone would type or paste that much data in any kind of routine way. Still, I did hear from one person that they like to paste stack traces into a chat window. You responded rather dismissively to that. It does sound like a corner case.

Maybe you don't report truncated data as a matter of course, and there are tons of other problems like this in the code, in places that I'm not yet aware of or that are invisible from the black box. Not this problem, but a problem with the same kind of cause could lead to a much more serious problem than this unlikely scenario.

Maybe there is a consistent pattern of user interface problems where the internals of the code handle problems but don't alert the user, even though the user might like to know about them.

Maybe there's a buffer overrun. That worries me more—a lot more—than the stack trace thing above. You remember that this kind of problem used to be dismissed as a "corner case" back when we worked at Microsoft—and then how Microsoft shut down new product development spent two months on investigating these kinds of problems, back in the spring of 2002? Hundreds of worms and viruses and denial of service attacks stem from problems whose outward manifestation looked exactly as trivial as this problem. There are variations on it.

Maybe there's a buffer overrun that would allow other users to view a conversation that my contact and I would like to keep between ourselves.

Maybe an appropriately crafted string could allow hackers to get at some of my account information.

Maybe an appropriately crafted string could allow hackers to get at everyone's account information.

Maybe there's a vulnerability that allows access to system files, as the Blaster worm did.

Maybe the product is now unstable, and there's a crash about to happen that hasn't yet manifested itself. We never know for sure if a test is finished.

Here's something that I think is more troubling, and perhaps the biggest risk of all. Maybe, by blowing off this report, you'll discourage testers from reporting a similarly trivial symptom of a much more serious problem. In a meeing a couple of weeks ago, the last time a tester reported something like this, you castigated her in public for the apparently trivial nature of the problem. She was embarrassed and intimidated. These days she doesn't report anything except symptoms that she thinks you'll consider sufficiently dramatic. In fact, just yesterday she saw something that she thought to be a pretty serious performance issue, but she's keeping mum about it. Some time several weeks from now, when we start to do thousands or millions of transactions, you may find yourself wishing that she had felt okay about speaking up today. Or who knows; maybe you'll just ask her why she didn't find that bug.

NASA calls this last problem "the normalization of deviance". In fact, this tiny little inconsistency reminds me of the Challenger problem. Remember that? There were these O-rings that were supposed to keep two chambers of highly-pressurized gases separate from each other. It turns out that on seven of the shuttle flights that preceded the Challenger, these O-rings burned through a bit and some gases leaked (they called this "erosion" and "blow-by"). Various managers managed to convince themselves that it wasn't a problem, because it only happened on about a third of the flights, and the rings, at most, only burned a third of the way through. Because these "little" problems didn't result in catastrophe the first seven times, NASA managers used this as evidence for safety. Every successful flight that had the problem was taken as reassurance that NASA could get away with it. In that sense, it was like Nassim Nicholas Taleb's turkey, who increases his belief in the benevolence of the farmer every day... until some time in the week before Thanksgiving.

Richard Feynman, in his Appendix to the Rogers Commission Report on the Space Shuttle Challenger Accident, nailed the issue:

The phenomenon of accepting for flight, seals that had shown erosion and blow-by in previous flights, is very clear. The Challenger flight is an excellent example. There are several references to flights that had gone before. The acceptance and success of these flights is taken as evidence of safety. But erosion and blow-by are not what the design expected. They are warnings that something is wrong. The equipment is not operating as expected, and therefore there is a danger that it can operate with even wider deviations in this unexpected and not thoroughly understood way. The fact that this danger did not lead to a catastrophe before is no guarantee that it will not the next time, unless it is completely understood. When playing Russian roulette the fact that the first shot got off safely is little comfort for the next.

That's the problem with any evidence of any bug, at first observation; we only know about a symptom, not the cause, and not the consequences. When the system is in an unpredicted state, it's in an unpredictable state.

Software is wonderfully deterministic, in that it does exactly what we tell it to do. But, as you know, there's sometimes a big difference between what we tell it to do and what we meant to tell it to do. When software does what we tell it to do instead of what we meant, we find ourselves off the map that we drew for ourselves. And once we're off the map, we don't know where we are.

According to Wikipedia, Feynman's investigations also revealed that there had been many serious doubts raised about the O-ring seals by engineers at Morton Thiokol, which made the solid fuel boosters, but communication failures had led to their concerns being ignored by NASA management. He found similar failures in procedure in many other areas at NASA, but singled out its software development for praise due to its rigorous and highly effective quality control procedures - then under threat from NASA management, which wished to reduce testing to save money given that the tests had always been passed.

At NASA, back then, the software people realized that just because their checks were passing, it didn't mean that they should relax their diligence. They realized that what really reduced risk on the project was appropriate testing, lots of tests, and paying attention to seemingly inconsequential failures.

I know we're not sending people to the moon here. Even though we don't know the consequences of this inconsistency, it's hard to conceive of anyone dying because of it. So let's make it clear: I'm not saying that the sky is falling, and I'm not making a value judgment as to whether we should fix it. That stuff is for you and the project managers to decide upon. It's simply my role to observe it and report it.

I think it might be important, though, for us to understand why the problem is there in the first place. That's because I don't know whether the problem that I'm seeing is a big deal. And the thing is, until you've looked at the code, neither do you.

As always, it's your call. And as usual, I'm happy to assist you in running whatever tests you'd like me to run on your behalf. I'll also poke around and see if I can find any other surprises.

Your friend,

The Tester

P.S. I did run a second test. This time, I used PerlClip to craft a string of 100000 instances of :). That pair of characters, in normal circumstances, results in a smiley-face emoticon. It seemed as though the input field accepted the characters literally, and then converted them to the graphical smiley face. It took a long, long time for the input field to render this. I thought that my chat window had crashed, but it hadn't. Eventually it finished processing, and displayed what it had parsed from this odd input. I didn't see 32768 smileys, nor 29996, nor 16384, nor 14998. I saw exactly two dots. Weird, huh?

Should We Call Test-Driven Development Something Else?

michael.a.bolton@gmail.com (Michael Bolton http://www.developsense.com) — Mon, 28 Sep 2009 18:42:00 +0000

In the first post in this series, I proposed "that those things that we usually call 'unit tests' be called 'unit checks'." I stand by the proposal, but I should clarify something important about it. See, it's all a matter of timing. And, of course, sapience.

After James Bach's blog post titled "Sapience and Blowing Peoples’ Minds", Joe Rainsberger commented:

Sadly, the distinction between testing and checking makes describing test-driven development (TDD) somewhat awkward, because it’s a test when I write it and a check after I run it for the first time. Am I test-driving or check-driving?

Joe has put his finger on something that's important: that in the mangle of practice, things are constantly changing, and so are our perspectives on them.

In The Elements of Testing and Checking, I broke down the process of developing, performing, and analyzing a check. The most important thing to note is that the check (an observation linked to a decision rule) can be performed non-sapiently, but that everything surrounding it—the development and analysis of the check—is sapient, and is testing. Test-driven development is first and foremost development, and development is a sapient process. The interactive process of developing a check and analyzing its outcome is a sapient process; the development cycle includes having an idea, testing it and responding to the information revealed by the test (the whole process), even when the result is supplied by a check (an atomic part of the test process). TDD is an exploratory, heuristic process. You don't know in advance what your solution is going to look like; you explore the problem space and build your solution iteratively, and you stop when you decide to stop.

Several years ago, James and Jon Bach produced a set of exploratory skills, tactics, and dynamics:

Modeling
Resourcing
Chartering
Observing
Manipulating
Pairing (now called Collaborating)
Generation and Elaboration
Overproduction and Abandonment
Abandonment and Recovery
Refocusing (Focusing and Defocusing)
Alternating
Branching and Backtracking
Conjecturing
Recording
Reporting

(I believe that several other people have made contributions to the original list, including Jonathan Kohl and Mike Kelly. I'd also include tooling—building tools, rather than merely obtaining or resourcing them, and orienteering—figuring out where you are in relation to where you want to be. I think James disagrees. That's okay; good colleagues do that. The cool thing about such lists is that they can evolve as we think and learn more, and disagreeing helps us to figure out what's important eventually. Maybe I'll drop them, or maybe James will adopt them.)

The point is that these exploratory skills, tactics and dynamics apply not only to testing, but to practically any open-ended heuristic process. Note how TDD, done well, incorporates practically all of the stuff from James and Jon's original list, which was focused on testing.

So the answer to the question in the title of this blog is this: No; there's no need to rename TDD. It really is test-driven development.

As James replied to Joe,

Strictly speaking you are "doing testing" by "writing checks", but not actually "writing tests." If you run the checks unattended and accept the green bar as is, then that is not testing. It requires absolutely no testing skill to do that, just as you wouldn't say someone is doing programming just because they invoke a compiler and verify that the compile was successful. If the bar is NOT green, the process of investigating is testing, as well as debugging, programming, etc.

If you watch the tests (James means checks here, I think --MB) run or ponder the deeper meaning of the green bar, you are doing testing along with the checking.

Think of "test" as a verb rather than a noun, and it becomes clear that test-driven design is truly test-driven design, although the testing is rather simplistic, based primarily on those little checks. Once the design is done the automated checks become useful as change detectors against the risk of regression. They certainly aid the testing process, despite not being tests.

Checks definitely do NOT drive development. Development is never a rote and non-sapient process. It's far better to say test-driven, because the design of the checks is a thoughtful process.

So what of the earlier business about calling unit tests "unit checks"?

For me, the distinction lies in the artifact—that xUnit thingy, or that rSpec assertion—and the way that you approach it. A minor gloss on Joe's comment: the thingy might not be a check after you run it the first time, especially if it doesn't pass. At that point, it is still very much part of your conscious interaction with the business of creating working code; it's figure, rather than ground.

After you've solved the problem that your unit of code is intended to solve, the thingy's prominence fades from figure into ground. You're no longer really paying much attention to it. There's no design activity going on with respect to it, it gets performed automatically and non-sapiently, and its result gets ignored, especially when the result is positive and aggregated with dozens, hundreds, or thousands of other positive results. At that point, it's no longer shedding any particular cognitive light on what you're doing, and its testing power has faded into a single pixel in a pale green glow. It's now a check, no longer a test but a change detector. In fact, you might think of "check" as an abbreviation for "change detector". The change from a test to a check is a kind of reverse metamorphosis, as though an intriuging, fluttering butterfly has turned into a not-very-interesting, ponderous little green caterpillar. That's not to say that it's not an important part of the Great Chain of Being; just that we tend not to pay much attention to it. However, we might pay more attention to the caterpillar when it's red.

As I've said repeatedly, what you call them is less important than how you think of them. As James says,

I wouldn't insist that people change their ordinary language. I see no problem calling whales "fish" or spiders "insects" in everyday life. But sometimes it matters to be precise. We should be ABLE to make strict distinctions when trying to help ourselves, or others, master our craft.

At Agile 2009, Joe pointed out that if we can produce more code with fewer errors in it, we can get our products to real testing, and then to market more quickly. And that means that we can get paid sooner. So I agree with Joe here, too:

I have to admit I like the pun of Check-Driven Development, even if it only works in American English.

The Testing vs. Checking Series

1. Testing vs. Checking
2. Transpection and the Three Elements of Checking
3. Pass vs. Fail vs. Is There a Problem Here?
4. Elements of Testing and Checking
5. Testing, Checking, and Changing the Language
6. Tests vs. Checks: The Motive for Distinguishing
7. A Tester Asks About Checking

Related: James Bach on Sapience and Blowing People's Minds

A Tester Asks About Checking

michael.a.bolton@gmail.com (Michael Bolton http://www.developsense.com) — Mon, 21 Sep 2009 14:22:00 +0000

In a previous comment, Sunjeet asks

Does not testing encompass checking? Can testing alone be efficient without doing any checking?

As I hope I made it clear in Elements of Testing and Checking, the development and analysis of checks is surrounded by plenty of testing activity, and testing may include a good deal of checking. Testing, I think, can be vastly more efficient if we consider the ways in which checking can be helpful. Cem Kaner, in his 2004 paper The Ongoing Revolution in Software Testing, said this:

I think the recent advances in unit testing (Astels, 2003; Rainsberger, 2004) have been the most exciting progress that I've seen in testing in the last 10 years.

With programmer-created and programmer-maintained change detectors (that is, checks -MB):
• There is a near-zero feedback delay between the discovery of a problem and awareness of the programmer who created the problem
• There is a near-zero communication cost between the discover of the problem and the programmer who will either fix the bug or fix the [check]
• The [check] is tightly tied to the underlying coding problem, making troubleshooting much cheaper and easier than system-level testing.

And I agree.

Should testers shun checking? Why not call checking as "confirmative testing"?

There is a role for testers to program and perform checks where cost is low and value is high, but I think that if practices associated with XP really begin to take hold, in the long run it behooves testers to get out of the checking business. That's because the vast bulk of the checking work will be done by programmers; because checks at the system level tend to be time-consuming, error-prone, and expensive when performed by humans (and expensive to automate when not performed by humans); because they drive humans to inattentional blindness.

But there's another reason: "confirmative testing" isn't really testing; it's confirming. It's looking at a white swan and saying, "All swans are white;" at another and saying, "See? All swans are white;" and at yet another and saying "Just like I told you; all swans are white." There's an analogy in software, "It works on my machine." "See? It works on my machine." "Just like I told you; it works on my machine." To find problems in a product, which is one of the key goals of testing, we need to get out of the confirmatory mindset.

I confirm that the exact problem is fixed - by exactly executing the steps mentioned in the bug report - by this I confirm that the bug and only that bug is fixed or not. Brainless? Yes... a machine could have done the same... yes BUT is this required ...YES we might not be deriving new quality value from it but we are CONFIRMING existing quality info from it...

Let me suggest an alternative way of looking at this.

In an environment where the bug report is vague or your programmers are known or suspected of being unreliable with their bug fixes or you know that the programmer has not created an automated check, then what you're describing might indeed be a very good idea. (When testing the fix, I might start by trying to reproduce the problem as exactly as I could, but I might also start with a slight variation on the problem to see if the general case has been fixed. Either way, I'll likely end up performing a check to see if the special case of the problem has been fixed.)

Task 2 i do is ...i look out for side effects ...regression...new test ideas ...execute more tests etc i.e. all the pillars of ET... How is task 2 useful without 1?

If you're following exactly the steps mentioned in the bug report and the programmer has fixed the problem and the programmer has already set up an automated check for the problem, then what you're doing is reproducing the programmer's effort (and the machine's effort) in performing a check, when it might be much more valuable to use your sapient skills and test. Note that I cannot decide either way. I'm not in your context or your immediate situation. But you can. To me, there's at least one clear circumstance in which it would be greatly more valuable for you to focus on Task 2: When someone has already done Task 1.

2. Should i tell my lead/manager...buddy I am just a tester find a checker to do this! or get a machine to do this? if a machine needs to do this who is going to code/script a machine to do this ...wont that be a tester himself?

In answer to the first question, let me ask this: Are you a tester or a checker? Again, I don't know. The quality of the questions you're asking suggest to me that you're a tester; that is, you're not accepting what I'm saying blindly, nor are you rejecting it out of hand. You're thinking critically about the idea, even if I may have blown your mind at first.

Either way, I wouldn't advise telling your lead or your manager that you're refusing to do work. But thinking in terms of testing vs. checking, if you so choose, might trigger a productive conversation between you about the relative cost and value of the activities that you (and the programmers) are doing. It might indeed make more sense to get a machine to do the checking work—and for the programmers to insert some change detectors and take greater responsibility for the quality of their code, an idea that was one of the triggers for Extreme Programming and the Agile movement.

As for who does the programming, note the passage from the very posting upon which you commented:

"When someone asks, 'Can't we hire pretty much any programmer to write our test automation code?', we can point out that the quality of checking is conditioned largely by the quality of the testing work that surrounds it, and emphasize that creating excellent checks requires excellent testing skill, in addition to programming skill."

Thank you for your comments and questions.

The Testing vs. Checking Series

1. Testing vs. Checking
2. Transpection and the Three Elements of Checking
3. Pass vs. Fail vs. Is There a Problem Here?
4. Elements of Testing and Checking
5. Testing, Checking, and Changing the Language
6. Tests vs. Checks: The Motive for Distinguishing
7. A Tester Asks About Checking
8. Tests vs. Checks: Should We Call Test-Driven Development Something Else?

Related: James Bach on Sapience and Blowing People's Minds

Tests vs. Checks: The Motive for Distinguishing

michael.a.bolton@gmail.com (Michael Bolton http://www.developsense.com) — Fri, 18 Sep 2009 14:55:00 +0000

The word "criticism" has several meanings and connotations. To criticize, these days, often means to speak reproachfully of someone or something, but criticism isn't always disparaging. Way, way back when, I studied English literature, and read the work of many critics. Literary critics and film critics aren't people who merely criticize, as we use the word in common parlance. Instead, the role of the critic is to contextualize—to observe and evaluate things, to shine light on some work so as to help people understand it better.

So when I say that Dale Emery is a critic, that's a compliment. On the subject of testing vs. checking, Dale recently remarked to me, "I think I understand the distinction. I don't yet understand what problem you're trying to solve with your specific choice of terminology. Not the implications, but the problem." That's an excellent critical statement, in that Dale is not disparaging, but he's trying to tell me something that I need to recognize and deal with.

My answer is that sometimes having different vocabulary allows us to recognize a problem and its solution more easily. As Jerry Weinberg says, "A rose by any other name should smell as sweet, yet nobody can seriously doubt that we are often fooled by the names of things." (An Introduction to General Systems Thinking, p. 74). He also says "If we have limited memories, decomposing a system into noninteracting parts may enable us to predict behavior better than we could without the decomposition. This is the method of science, which would not be necessary were it not for our limited brains." (ibid, p. 134).

The problem I'm trying to address, then, is that the word test lumps a large number of concepts into a single word, and testing lumps a similarly large number of activities together. As James Bach suggests, compiling is part of the activity of programming, yet we don't mistake compiling for programming, nor do we mistake the compiler for the programmer.

If we have a conceptual item called a check, or an activity called checking, I contend that we suddenly have a new observational state available to us, and new observations to be made. That can help us to resolve differences in perception or opinion. It can help us to understand the process of testing at a finer level of detail, so that we can make better decisions about strategy and tactics.

In the Agile 2009 session, "Brittle and Slow: Replacing End-To-End Testing", Arlo Belshee and James Shore took this as a point of departure:

End-to-end tests appear everywhere: test-driven development, story-test-driven development, acceptance testing, functional testing, and system testing. They're also slow, brittle, and expensive.

This was confusing to me. My colleague Fiona Charles specializes in end-to-end system testing for great big projects. The teams that she leads are fast, compared to others that I've seen. Their tests are painstaking and detailed, but they're flexible and adaptable, not brittle.

During the session, one person (presumably a programmer, but maybe not) said, "Manual testing sucks." There was a loud murmur of agreement from both the testers and the programmers in the room.

I thought that was strange too. I love manual testing. I like operating the product interactively and making observations and evaluations. I like pretending that I'm a user of the program, with some task to accomplish or some problem to solve. I like looking at the program from a more analytical perspective, too—thinking about how all the components of the product interact with one another, and where the communication between them might be vulnerable if distorted or disturbed or interrupted. I like playing with the data, trying to figure out the pathological cases where the program might hiccupp or die on certain inputs. In my interaction with the program, I discover lots of things that appear to be problems. Upon making such a discovery, I'm compelled to investigate it. As I investigate it, sometimes I find that it's a real problem, and sometimes I find that it isn't. In this process, I learn about the system, about the ways in which it can work and the ways in which it might fail. I learn about my preconceptions, which are sometimes right and sometimes wrong. As I test, I recognize new risks, whereupon I realize new test ideas. I act on those test ideas, often right away. (By the way, I'm trying to get out of the habit of calling this stuff manual testing; I learning to call it sapient testing, because it's primarily the eyes and the brain, not the hands, that are doing the work. Whatever you call it, manual testing doesn't suck; it rocks.

So are the programmer in question and all the people who applauded ignorant? That seems unlikely. They're smart people, and they know tons about software development. Are they wrong? Well, that's a value judgment, but it would seem to me that as smart people who solve problems for a living, it would be very surprising if they weren't engaged by exploration and discovery and investigation and learning. So there must be another explanation.

Maybe when they're talking about manual testing, they're talking about something else. Maybe they're talking about behaving like an automaton and precisely following a precisely described set of steps, the last of which is to compare some output of the program to a predicted, expected value. For a thinking human, that process is slow, and it's tedious, and it doesn't really engage the brain. And in the end, almost all the time, all we get is exactly what we expected to get in the first place.

So if that's what they're talking about, I agree with them. Therefore: if we're going to understand each other more clearly, it would help to make the distinction between some kinds of manual testing and other kinds. The think that we don't like, that none of us like apparently, is manual checking.

Maybe Arlo and James were talking about end-to-end system checks being brittle and slow. Maybe it's integration checks, rather than integration tests, that are a scam, as Joe (J.B.) Rainsberger puts it here, here, here, and here.

So having a handle for a particular concept may make it easier for us to make certain observations and to carry on certain conversations.

If we can differentiate between manual testing and manual checking, we might be more specific about what, specifically sucks.

If we can comprehend the difference between automated tests and automated checks, we can understand the circumstances in which one might be more valuable than the other.

If we tease out the elements of developing, performing, and evaluating a check (as I attempted to do here) we might better see specific opportunities for increasing value or reducing cost.

If we can recognize when we're checking, rather than testing, we can better recognize the opportunity to hand the work over to a machine.

If we can recognize that we're spending inordinate amounts of time and money preparing scripts directing outsourced testers in other countries to check, rather than test, we can recognize a waste of energy, time, money, and human potential, because testers are capable of so much more than merely checking. (We might also detect the odour of immorality in asking people in developing countries to behave like machines, and instead consider giving a modicum of freedom and responsibility to them so that they can learn things about the product—things in which we might be very interested.)

If we can recognize that checking alone doesn't yield new information, we can better recognize the need to de-emphasize checking and emphasize testing when that's appropriate.

If we can recognize when testing is pointing us to areas of the product that appear to be vulnerable to breakage, we might choose to emphasize inserting more and/or better checks, so as to draw our attention to breakage should it occur ("change detectors", as Cem Kaner calls them).

If we can distinguish between testing and checking, we can penetrate "the illusion that software systems are simple enough to define all the checks before any code is written", as my colleague Ben Simo recently pointed out—never mind all the tests.

When someone asks, "Why didn't testing find that bug when we spent all that money on all those automation tools?", maybe we can point to the fact that the tools foster checking far more than they foster testing.

Maybe we can recognize that checking tends to be helpful in preventing bugs that we can anticipate, but not so helpful at finding problems that we didn't anticipate. For that we need testing. Or, alas, sometimes, accidental discovery.

Maybe we'd be able to recognize that testing (but not checking) can reveal information on novel ways of using the product, information that can add to the perceived value of the product.

When someone asks, "Can't we hire pretty much any programmer to write our test automation code?", we can point out that the quality of checking is conditioned largely by the quality of the testing work that surrounds it, and emphasize that creating excellent checks requires excellent testing skill, in addition to programming skill.

If we're interested in improving the efficiency and capacity of the test group, we can point out that test automation is far more than just check automation. Test automation is, in James Bach's way of putting it, any use of tools to support testing. Testing tools help us to generate test data; to probe the internals of an application or an operating system; to produce oracles that use a different algorithm to produce a comparable result; to produce macros that automate a long sequence of actionas in the application so that the tester can be quickly delivered to place to start exploring and testing; to rapidly configure or reconfigure the application; to parse, sort, and search log files; to produce blink oracles for blink testing...

When a programmer says to a tester, "You should only test this stuff; here are the boundary conditions," the tester can respond "I will check that stuff, but I'm also going to test for boundary conditions that you might not have been aware of, or that you've forgotten to tell me about, and for other possible problems too."

When we see a test group that is entirely focused on confirming that a product conforms to some requirements document, rather than investigating to discover things that might threaten the value of the product to its users, we can point out that they may be checking, but they're not testing.

Here's a passage from Jerry Weinberg, one that I find inspiring and absolutely true

"One of the lessons to be learned ... is that the sheer number of tests performed is of little significance in itself. Too often, the series of tests simply proves how good the computer is at doing the same things with different numbers. As in many instances, we are probably misled here by our experiences with people, whose inherent reliability on repetitive work is at best variable. With a computer program, however, the greater problem is to prove adaptability, something which is not trivial in human functions either. Consequently we must be sure that each test does some work not done by previous tests. To do this, we must struggle to develop a suspicious nature as well as a lively imagination." Jerry Weinberg, Computer Programming Fundamentals, 1961.

To me, that's a magnificent paragraph. But just in case, let's paraphrase it to make it (to my mind, at least) even clearer:

"One of the lessons to be learned ... is that the sheer number of checks performed is of little significance in itself. Too often, the series of checks simply proves how good the computer is at doing the same things with different numbers. As in many instances, we are probably misled here by our experiences with people, whose inherent reliability on repetitive work is at best variable. With a computer program, however, the greater problem is to prove adaptability, something which is not trivial in human functions either. Consequently we must be sure that each test does some work not done by previous checks. To do this, we must struggle to develop a suspicious nature as well as a lively imagination.

Thank you to Dale for your critical questions, and to the others who have asked questions about the motivation for making the distinction and hanging a new label on it. I hope this helps. If it doesn't, please let me know, and we'll try to work it out. In any case, there will be more to come.

The Testing vs. Checking Series

1. Testing vs. Checking
2. Transpection and the Three Elements of Checking
3. Pass vs. Fail vs. Is There a Problem Here?
4. Elements of Testing and Checking
5. Testing, Checking, and Changing the Language
6. Tests vs. Checks: The Motive for Distinguishing
7. A Tester Asks About Checking
8. Tests vs. Checks: Should We Call Test-Driven Development Something Else?

Related: James Bach on Sapience and Blowing People's Minds

Upcoming Events: KWSQA and STAR West

michael.a.bolton@gmail.com (Michael Bolton http://www.developsense.com) — Wed, 16 Sep 2009 18:36:00 +0000

I'm delighted to have been asked to present a lunchtime talk at the Kitchener-Waterloo Software Quality Association, Wednesday September 30. I'll be giving a reprise of my STAR East keynote talk, What Haven't You Noticed Lately? Building Awareness in Testers. (The title has been pinched from Mark Federman, who got it from Terence McKenna, who may have got it from Marshall McLuhan, but maybe not.)

The following week, it's STAR West in Anaheim, California. I'll be giving a half-day workshop, Tester's Clinic: Dealing with Tough Questions and Testing Myths and a track session, The Skill of Factoring: Identifying What to Test.

I'll also be giving a bonus session, Using the Secrets of Improv to Improve Your Testing. I've done this one at Agile 2008 in Toronto, and at the AYE Conference in 2006, and it's fun, but because so much of the learning comes from the participants, in the moment, it's also been remarkably insightful both times. Improv is about being aware of your actions, the actions of others, and how they relate to each other—immediately. Even dipping one's toe in it is very exciting. Adam White talks compellingly about his experience of a couple of rounds of classes with Second City, and he did a well-regarded improv session at CAST 2008.

There's an official panel discussion hosted by Ross Collard on Wednesday at 6:30, and there's an official Meet-The-Presenter session Thursday morning. The rest of the time, James Bach and I will be holding unofficial versions of both of those things. We'll be bringing testing toys and testing games, and workshopping old and new exercises with whomever wants to come. He'll likely be talking about his new book, Secrets of a Buccaneer Scholar, a terrific memoir and guide to self-education.

I'd like to meet you at the conference, but I'm not sure who you are. If you'd like to do some hands-on testing puzzles, have chat about testing vs. checking, or to discuss anything you like, drop me a line—michael at developsense.com.