Blog Posts from February, 2006

Exploratory Testing: Building the Skill

Saturday, February 25th, 2006

I spent the last couple of days of January and the first few days of February attending two workshops hosted by Cem Kaner and James Bach in Melbourne, Florida.

The first was the Exploratory Testing Research Summit, attended by

James Bach
Jonathan Bach
Scott Barber
Michael Bolton
Elisabeth Hendrickson
Cem Kaner
Mike Kelly
Jonathan Kohl
James Lyndsay
Robert Sabourin

As I expected, it was a very rich and useful discussion. The mandate was to begin building a community of expert exploratory testers; to bring up consensus and differences about what exploratory testing is and how to do it better; to exchange models and teaching exercises; and to begin to figure out ways of building the reputation of exploratory testing as a central skill of our profession.

Each person above is someone that I consider to be a key figure in the E.T. community. The thing that they have in common: the work of each is valuable, interesting, and distinctive. For now, check out their Web sites; in the next little while, I’ll be posting bits and pieces of an experience report.

New Look

Saturday, February 25th, 2006

A little while hacking away at CSS, and there’s a new style for the blog. Hope you like it.

The Simplest Things Can Possibly Fail

Friday, February 10th, 2006

Jonathan Kohl and I were chatting recently as we often do. (Chatting with Jonathan is so interesting and enjoyable and productive that it sometimes has a serious impact on other forms of productivity.)

We’re observing together the evolution of agile development, in terms of what I’m proposing to call “mythodology“. (I invented this term independently, and was sorry to see via a Google search that I wasn’t the first person in the world to use it. However, I’d like to go down in history as the person who coined the term with respect to software development; please feel free to help me in my goal. But I digress.)

Most recently, Jonathan was bemoaning what he observes to be an excess of YAGNI (You Ain’t Gonna Need It) and STTCPW (the Simplest Thing That Could Possibly Work), two allegedly essential agile principles. Specifically, he was observing what he considered to be (and which testing demonstrated to be) sloppy work, and the rationale that was being presented for it was that it was STTCPW. Recall that, as James Bach says, “when a developer says ‘it works’, he really means, ‘it appears to meet some requirement to some degree.'” STTCPW becomes toxic when “it works” fails to include some pretty fundamental requirements–for example, releasing a resource that you’re finished with. STTCPW might NRW (Not Really Work); applied thoughtlessly, it can slip into OLLIW (Only Looks Like It Works).

As with any mythodology, principles that begin with reasonable grouding can be misapplied and corrupted. This reminded me of James’ observation that the Traditionalists and the Agilists are both instances of the Routine or Factory School of Software Testing (as Bret Pettichord describes in his wonderful paper Four Schools of Software Testing). I may disagree with James to the degree that I don’t think the early or smarter Agilists intended to behave in a Routine-School manner, but I think he’s right to point out that a large and growing number of their disciples appear to have slipped into Routine behaviour. The common element between Bad Traditionalist processes and Bad Agile Processes is that both want to remove that irritating thinking part.

In my experience, I’ve known process-followers and I’ve known experts. The distinction between them is that the experts can follow processes, but they don’t follow them blindly. Instead, they incorporate critical thinking–and in particular self-critical thinking–into everything that they do. The experts demonstrate humility perfectly balanced with arrogance, and confidence with fallibility. They tend to pause and think for an extra moment before deciding that they’re done. Rather than simply assuming that YAGNI, they consider the possibility that you might really need it, or that STTCPW might be too simple and might not work. Agility in the face of excess bureaucracy is welcome, but it’s strengthened when it’s tempered with reflection.

Scott Ambler at TASSQ

Friday, February 3rd, 2006

Some time in December 2006, I asked Scott Ambler if he would provide us with a presentation at TASSQ, the Toronto Association of System and Software Quality (I’m the program chair for that organization). I was delighted that he agreed to present. I was aware that his remarks would probably be controversial, and that I would agree with some of them and disagree strongly with others.

Scott gave his talk to TASSQ on January 31, 2006. I wasn’t able to attend; I was at an Exploratory Testing summit meeting and the Workshop on Teaching Software Testing, both in Melbourne, Florida (about which more in a later blog entry). However, I heard reports back from a couple of people, one of whom pointed me to a blog post, here:

I reviewed the speaker notes, and I’ve seen a similar presentation from Scott. He also participated in the Toronto Workshop on Software Testing in June of 2005, which I convened with Fiona Charles. So although I didn’t attend this presentation, I’ve got a fairly good picture of what must have gone on. If I say “Scott said,” I’m mererly assuming and alleging that he did, for convenience in addressing his arguments.

Scott, like other Agilistas, proposes that XP and other agile processes will result in higher quality software. I hope they’re right; I’m all for that. I agree with Scott when he says that test-driven development (TDD) is a technique that is likely to result in better-designed and more reliable code. If a developer writes a failing test, and then writes code to pass it, we have some assurance that the code at least fulfills that requirement. Morever, after that code is written, we have a unit test that helps the developer to modify the code with confidence; since all tests are run after each modification, the developer gets instant feedback if the code happens to break in some way.

TDD is a strong design technique. According to Ward Cunningham, who is generally credited by Beck and others with inventing the approach, TDD is really all about design. The TDD tests provide an excellent regression suite too; it provides confirmation that things are still working OK.

But I think Scott misses the mark if he believes that TDD is an effective overall testing technique. A product’s functional units can be functioning perfectly, but interactions between them might not be. TDD deliberately isn’t focussed on testing the interactions between functional units. In order to be effective, the unit tests suites need to be compact, and they need to run quickly. If the tests cover too much or take too long to run, developers won’t run them. TDD tests are therefore intentionally lightweight and low-level. That’s fine, but no sane person should mistake a suite of unit tests with good testing coverage.

More complex, user-oriented tests in agile environments are often handled by requirements-oriented tools such as Fit or FitNesse. With these tools, requirements writers describe the product in Word documents or on a Wiki, and provide examples of how certain functions should behave by inserting tables that contain examples of inputs and expected outputs. Developers provide fixtures–interfaces between these tables and the product’s actual code. At the push of a button, FIT/FitNesse runs the code associated with the documents, and turns the cells in the tables green where the tests achieve the predicted result, and red where there have been failures. This is pretty powerful, since it’s easy to see where some of the product’s code is running successfully from moment to moment.

For me, as a tester, the real limitation with TDD and FIT/FitNesse is that they’re focussed on confirmation, and not on investigation. Confirmation is a necessary and useful part of testing, but it’s not the whole story. Functionality or capability are not the only quality-related aspects of the product. Other classes of tests are important too–system testing, end-to-end testing, stress testing, flow testing (in which complete transactions are run, one after another, without the system being reset), performance testing, high-volume random testing. FIT/FitNesse generally has dreadful support for the user interface of the product under test, although other tools such as WATIR (for Web applications) can step in to some degree.

The most pernicious (and often unstated) aspect of the arguments about testing in agile environments is the suggestion that automated tests are an unalloyed good, and that manual tests are unquestionably bad. Some in the agile community decree that all tests must be automated, or that a manual test is something to be despised. This is a farcical notion, and bespeaks a fundamental ignorance of the purposes and practices of the kind of testing that I perform and promote. I’m not omniscient–not even close–but I do have the power to operate and observe a product in ways that a machine cannot. The machine has speed and precision going for it; the human has cognitive capabilities. Automated tests are very poor at evaluating certain kinds of things that humans can assess with ease. A tester with a trained brain, turned on, can use exploratory skills to find new problems in usability, security, compatibility, supportability, localization, prompted by nothing more than a thoughtful question that she asks herself.

Automated tests can’t suspect a problem and then act on the knowledge. An automated test typically operates using a single oracle (an oracle is a principle or mechanism by which we recognize a problem). At any given instant, it’s possible for a human to work through dozens of oracles; that’s what we do. Some of us are better at that than others, but we can all improve our skills.

I agree with Scott when he argues that we testers are going to have to improve our skills in order to bring value. I’ve maintained that for years; that’s why I’m a trainer and a consultant, and that’s why, for the last few years, I’ve spent six to eight weeks a year getting training and attending conferences. That’s not the only way to improve skills; there are books, online resources, and testing communities to draw upon. I agree when he says that jobs are at risk for those who don’t sharpen the saw. But I don’t agree when Scott suggests that testers are going to be slain by agile development’s silver bullet. We need intelligent, thoughtful people to ask challenging questions about software products; we need versatile, skilled people to ask those questions, in the form of manual and automated tests; and we’ll need such people forever. The question is: what kind of tester do you want to be? I’ll have much more to say in the near future.

—Michael B.