Archive for the ‘Tester Skill’ Category

« Older Entries
Newer Entries »

Questioning Test Cases, Part 2: Testers Learn, But Test Cases Don’t Teach

Wednesday, April 6th, 2011

In the last post, my LinkedIn correspondent provided a couple of reasons why she liked writing test cases, and why she thought it necessary to write them down in the early stages of the project. Then she gave a third reason:

When I’m on a project and I am the only one who knows how to test something, then I can’t move on to something new. I’d still be testing Cut/Copy/Paste if I were not able to pass that script on easily to a new recruit.

I often hear this kind of justification for test cases: “We need test cases so that testers can learn quickly how to test the product.” To me, that’s like saying that we need driving cases so that people can learn quickly how to drive in a new city. In a new city, a map or a general set of ideas might be helpful. Driving cases won’t, because to drive through a new city well, you have to know how to drive. In the same way, test cases are unlikely to be anything more than negligbly helpful. In order to learn quickly how to test a new program well, you have to know how to test.

It also seemed to me that she considers that writing to be the One True Way to impart information, and that reading to be the One True Way to receive it. If I’m wrong about that, it’s because I’ve misinterpreted something she’s written. Fancy that!

So for the sake of discussion, let’s say I were to to hire (say) an older domain expert, making a career change, who had no computer experience, and let’s use Cut/Copy/Paste as an example. With the new recruit, I’d do what I’ve done many times in coaching sessions. I’d sit with her down with a copy of Microsoft Word, and open up a document that had some text and some images in it. Then I’d say something like this:

“We’re going to learn how to move text around, cutting and pasting and making copies of it so we can save time on retyping stuff that the machine can do for us. Okay. Put your cursor there, and press Shift-RightArrow there. Good. See what happened?”

(There would be lots of pauses. I’d pause after each question to give the trainee time to answer; frequently to let the trainee experiment; and often to let the trainee ask questions of her own. I won’t do that explicitly here; I’ll let you put most of the pauses in yourself, as you read. For sure, assume that there’s one at least after every question, and usually after every sentence.)

“So try doing that a few more times. What happened? So you’ve highlighted the text; people call that marking or selecting the text. What happens when you add the Ctrl key, and press Ctrl-Alt-RightArrow at the same time? Right: it selects a word at a time. Now, try pressing Ctrl-X.

“Whoa, what happened just there?! Right: pressing Ctrl-X that makes the selected text go away; you can see that. What you might not know is that the text goes onto the Windows Clipboard, a kind of temporary storage area for all kinds of stuff. So you haven’t erased it; it’s more like you’ve cut it out, and the computer still has a copy of it stashed somewhere. Now, put your cursor here, and hit Ctrl-V. What did that do? Yeah; it pasted the text you cut before. That is, it retrieved the text from the clipboard, and made a copy right where you pressed Ctrl-V. So, Ctrl-X cuts, and Ctrl-V pastes a copy of the thing you cut. Try that again. What happened? Right; another copy. So it’s a little different from regular cutting and pasting, in that you can make many copies of the same thing. When might that come in handy? Good idea! Yep, that’s a good idea too. Try it a few more times, in other places. What does Ctrl-V do? What does Ctrl-X do?

“Try selecting something else, and pressing Ctrl-X again. What happens when you paste now? Can you get the old one back? Well, there might be special add-ons that allow you to keep copies of lots of things you pasted, but normally, you can only paste the last thing you put on the Clipboard.

“How are you going to remember that Ctrl-X means “cut”? That’s a great idea! Myself, I think of the X as a pair of open scissors, ready to cut—but your idea is great, and better than that, it’s your own, so you’ll remember it. How are you going to remember Ctrl-V as paste? Yeah, it’s tough to come up with a way to remember that, isn’t it? I think of it like the tip of one of those old glue bottles from grade school. On the other hand, what’s in between X and V on the keyboard? Now, go select some text again. Remember how? Good. This time, try pressing Ctrl-C, instead of Ctrl-X. Right; nothing seems to happen. And yet… move your cursor here, and try pasting. Yeah; it’s a copy, even though the original stays where it was. So Ctrl-C is Copy–C for copy, which makes some kind of sense, for a change—right there in between cut and paste. And there’s another way to remember; try pressing Alt-E. E stands for Edit, that’s right. What do you see listed on the menu, there?

“Now… there’s some documentation for these cut-and-paste features; it’s in the Windows documentation. Press F1. Great. Now look for “Keyboard Shortcuts”. There’s a list. How many different ways can you find of marking, cutting, and pasting text? Make me a list, and I’ll be back in ten minutes. Be ready to show me your report, and to demonstrate the ones you’ve discovered. Oh—and one more thing: create a list of at least five circumstances in which you might use this stuff, and we’ll talk about that too.”

It’s taken quite a while to type this example. It would take considerably less time to do it. Moreover, the trainee would learn more quickly and more deeply, because she’d be interacting with the product herself, experiencing the actions in her body, literally incorporating them. The questions, the answers, and the interactions make the learning more sticky. The practice makes it more sticky still. I try to encourage people to create mnemonics to help remember. While conversing with the trainee, I’d also be observing her. She’ll be making plenty of mistakes; people do that when they’re learning. In fact, people should be encouraged to do that when they’re learning. (Brian Marick’s book Everyday Scripting in Ruby encourages deliberate mistakes, which is one of the reasons I like it so much.) When I see mistakes, I’ll give her a chance to feel the mistake, puzzle out a way around it, and then—if she needs it—help her get out of it. (“Let’s figure out how to fix that. Hit Alt-E again. What’s listed on the Edit menu? So what’s the keyboard shortcut for Undo?” “Oops, you pressed Ctrl-Z one time too many… What does Ctrl-Y do?”). After a while, I won’t tell her how to do everything right away. I’ll start asking her to figure out some things on her own, and only help out when she’s stuck. That way she’ll be learning it for herself, which is more challenging but more affecting. I’ll keep trying to raise the bar, giving her more challenging assignments, increasing her independence while also increasing her responsibility to report. When I see her make a mistake, I might correct her right away, or I might let her continue with the mistake for a bit until she encountered some kind of minor consequence. That makes the learning even more sticky. Finally, I’d cut the conversation short if she told me that she already knew how to cut and paste—but I’d get her to show me that she knew it, and I’d give her some more exercises so that I could observe her.

The trouble that I see with “passing on a script to a new recruit” is that most test scripts I’ve seen are very long on what to do or how to do it, and correspondingly short on why to do it and how to generalize it. That is, they’re short on motivation, and they’re short on risk, and they rarely include a mission to learn. In our teaching, when we’re working with test groups, and when we’re coaching individual testers, James and I put less emphasis on techniques. Instead, we focus on making sure that people have the skills to know what to do when they receive a testing mission, whether that mission is written down in detail or not.

If there are specific things that I want a tester to look for, I’ll provide specific instructions (either written or spoken), but mostly I want to set out on their own to discover things. I want to encourage testers to vary their work and their observations. Variation makes it more that they will find problems that would go off hiding, were we to focus on the scripts. As a bonus, giving testers a concise, open-ended task keeps writing to a minimum, which saves on development cost, maintenance cost, and opportunity cost.

We’ll finish up this series in the next post.

Posted in Accountability, Test Framing, Tester Skill, Training | 9 Comments »

Questioning Test Cases, Part 1

Monday, April 4th, 2011

Over the years, LinkedIn seems to have replaced comp.software.testing as the prime repository for wooly thinking and poorly conceived questions about testing.

Recently I was involved in a conversation with someone who, at least, seemed to be more articulate than most of the people on LinkedIn. Alas, I’ve since lost the thread, and after some searching I’ve been unable to find it. No matter: the points of the discussion are, to me, are still worth addressing. She was asking a question about how much time to allocate to writing test cases before starting testing, and I questioned the usefulness of doing that. The first part of her reply went like this:

I would like to point out that I doubt anyone wants to write something that they don’t need to write.

I agree that most people probably don’t want to write things they don’t need to write. But they often feel compelled, or are compelled to write things they don’t need to write.

I find value in writing test cases for a number of reason. One is that I train more junior engineers in testing and it is a good method to have them execute tests that I have written so they learn how a good test plan is put together.

If that were so, wouldn’t your junior engineers learn even more from writing test cases themselves, and getting feedback on their design and their writing? There’s a feedback loop in the design of a test, the execution of a test, the interpretation of a test result, and the learning that happens between them; wouldn’t it be a good idea to keep the feedback loop—and the learning—as rapid as possible? Wouldn’t your junior engineers learn still more from actually testing—under your close supervision, at first, and then with the freedom and responsibility to act more independently as they gain skill? You might want to have a look at this article: http://www.developsense.com/articles/2008-06-KnowWhereYourWheelsAre.pdf.

There’s a common misconception that testing happens in the characters of a written test case. It doesn’t. Testing happens in the mind and actions of the tester. It happens in the design of the test, in the execution of the test, in the observation and interpretation of the outcome of the test. Testing happens in the discovery of problems, in the investigation of those problems, and the learning about those problems and the product. At most, a fraction of this can be written down.

A test is far less something you execute, and far more a line of inquiry that you follow. To me, a good test case is idea-stuff; it’s a question that we want to ask of the program, based on some motivating idea about discovery or risk. In my observation, in writing test cases, people generally write down the least important stuff. They appear to be trying to program the tester’s actions, rather trying than to prime the tester’s thinking and observation.

Moreover, a test plan something quite different from a pile of test cases.

Secondly, [writing test cases] communicates the testing coverage with everyone involved in developing the software. If you are a contractor, this is very important since you want to leave with the client feeling like you did your job and they have the documentation to prove that they have done due diligence if they shop their company around or look for for VC money.

Were you, as a contractor, given the mission to produce test scripts specifically, or is that a mission that you have inferred? Bear in mind, I’ve been witness to many takeovers, as a program manager at a company where our senior managers were ambitiously acquiring technologies, products and companies, and as a consultant to several companies that were taken over by larger companies. In no case did anyone ever ask to see any test case documentation. Those who are investigating the company to be acquired typically don’t go to that level of detail. In my experience, alas, due diligence largely doesn’t happen at all. I’m puzzled, too, by the appeal to the least likely instances in which people might interact with test documention, rather than the everyday.

Meanwhile, there are many ways to communicate test coverage. (For example, see here, here, and here.) There are also many ways to fool yourself (and others) into believing that the more documentation the more coverage, or the more specific the documentation the more coverage—especially when that documentation is prospective, rather than retrospective. In Rapid Testing, we don’t encourage people to eliminate documentation. We encourage people to reduce documentation to what is actually necessary for the mission, and to eliminate wasteful documentation.

We focus on documenting test ideas concisely; on producing coverage outlines and risk lists that can be used to guide, rather than control a tester and her line of inquiry; on producing records of the tester’s thought process, observations, risk ideas, motivations, and so forth (see below). The goal is to capture things far more important than the tester’s mechanical actions. If someone wants a record of that, we recommend video capture software (tools like BB Test Assistant or Camtasia). An automatic log allows the tester to focus on testing the product and recording the ideas, rather than splitting focus between operating the product and writing about operating the product.

You can find examples of the kind of test documentation I’m talking about here, in the appendices to the Rapid Software Testing class, starting at page 47. Note that each example has varying degrees of polish and formal structure. Sometime it’s highly informal, used to aid memory, to frame a conversation, or trigger ideas. Sometimes it’s more formal and polished, when the audience is outside the test group. The over-riding point is to fit the documentation to the mission and to the task at hand.

More to come…

Posted in Accountability, Test Framing, Tester Skill, Training | 6 Comments »

Why Do Some Testers Find The Critical Problems?

Saturday, February 5th, 2011

Today, someone on Twitter pointed to an interesting blog post by Alan Page of Microsoft. He says:

“How do testers determine if a bug is a bug anyone would care about vs. a bug that directly impacts quality (or the customers perception of quality)? (or something in between?) Of course, testers should report anything that may annoy a user, but learning to differentiate between an ‘it could be better’ bug and a ‘oh-my-gosh-fix-this’ bug is a skill that some testers seem to learn slowly. … “So what is it that makes some testers zero in on critical issues, while others get lost in the weeds?”

I believe I have some answers to this. My answers are based on roughly 20 years of observation and experience in consulting, training, and working with other testers. The forms of interaction have included in-class training; online coaching via video, voice, and text; face-to-face conversation in workplaces, conferences, and workshops; direct collaboration with other working testers in mass-market commercial software, financial services, retail services, specialized mathematical applications, and several other domains.

My first answer is that testing, for a long time and in many places, has been myopically focused on functional correctness, rather than on value to people. Cem Kaner discusses this issue in his talk Software Testing as a Social Science, and later variations on it. This problem in testing is a subset of a larger problem in computer science and software engineering. Introductory texts often observe that a computer program is “a set of instructions for a computer”. Kaner’s defintion of a computer program as “a communication among several humans and computers, distributed over distance and time, that contains instructions that can be executed a computer” goes some distance towards addressing the problem; his explication that “the point of the program is to provide value to the stakeholders” goes further still. When the definition of programming is reduced to producing “a set of instructions for a computer”, it misses the point—value to people—and when testing is reduced to the checking of those instructions, the “testing” will miss the same point. I’ve suggested in recent talks that testing is “the investigation of systems composed of people, computer programs, related products and services.” Successful testers avoid a fascination with functional correctness, and focus on ways in which people might obtain value from a program—or have their value unfulfilled or threatened.

This first answer gives rise to my second: that when testing is focused on functional correctness, it becomes a confirmatory, verification-oriented task, rather than an exploratory, discovery-oriented set of processes. This is not a new problem. It’s old enough that Glenford Myers tried (more or less unsuccessfully, it seems) to argue against it in The Art of Software Testing in 1979. Myers’ point was the testing should be premised on trying to expose the program’s failures, rather than on trying to confirm that it works. Psychological research before and since Myers’ book (in particular Klayman and Ha’s paper on confirmation bias) shows that the positive test heuristic biases people towards choosing tests that demonstrate fit with a working hypothesis (showing THAT it works), rather than tests that drive towards final rule discovery (showing how it works, and more important, how it might fail). Worse yet, I’ve heard numerous reports of development and test managers urging testers to “make sure the tests pass”. The trouble with passing tests is that they don’t expose threats to value. Every function in the program code might be checked and found correct, but the product might be unusable. As in Alan’s example, the phone might make calls perfectly, but unless we model the way people actually use the product—talking for more than three minutes at a time, say—we will miss important problems. Every function might work perfectly, but we might fail to observe missing functionality. Every function might work perfectly, but we might miss terrible compatibility problems. Functional correctness is a very important thing in computer software, but it’s not the only thing. (See the “Quality Criteria” section of the Heuristic Test Strategy Model for suggestions.) Testers “who zero in critical issues” avoid the confirmation trap.

My third answer (related to the first two) is that when testing is focused on confirming functional correctness, a lot of other information gets left lying on the table. Testing becomes a search for finding errors, rather than on finding issues. That is, testers become oriented towards reporting bugs, and less oriented towards the discovery of issues—things that aren’t bugs, necessarily, but that threaten the value of testing and of the project generally. I’ve written recently about issues here. Successful testers recognize issues that represent obstacles to their missions and strategies, and work around them or seek help.

My fourth answer is that many (in my unscientific sample, most) testers are poorly versed in the skills of test framing. This is understandable, at least in part because test framing itself wasn’t known by that name as recently as a year ago as I write. Test framing is the set of logical connections that structure and inform a test. It involves the capacity to follow and express a line of perhaps informal yet reasonably structured logic that directly links the testing mission to the tests and their results. In my experience, most testers are unable to trace this logical line quickly and expertly. There are many roots for this problem. The earlier answers above provide part of the explanation; the mission of value to the customer is overwhelmed by the mission of proving functional correctness. In situations where the process of test design is separated from test execution (as in environments that take a highly scripted approach to testing), the steps to perform the test and observe the results are typically listed explicitly, but the motivation for performing the test is often left out. In situations where test execution, observation of outcomes, and reporting of test results is heavily delegated to automation, motivation is even further disconnected from the mission. In such environments, focus is directed towards getting the automation to follow a script, rather using than automation to assist in probing for problems. In such environments, focus is often on the quantity of tests or the quantity of bug reports, rather than on the quality, the value, of the information revealed by testing. Testers who find problems successfully can link tests, test activities, and test results to the mission. They’re far more concerned about the quality of the information they provide than the quantity.

My fifth answer is that in many organizations there is insufficient diversity of tester skills, mindsets, and approaches for finding the great diversity of problems that might lurk in the product. This problem starts in various ways. In some organizations, testers are drawn exclusively from the business. In others, testers are required to have programming skills before they can be considered for the job. And then things get left out. Testers who need training or experience in the business domain don’t get it, and are kept separated from the business people (that’s a classic example of an issue). Testers aren’t given training in software design, programming, or related skills. They’re not given training in testing, problem reporting and bug advocacy, design of experiments. They’re not given training or education in anthropology, critical thinking, systems thinking, or philosophy and other disciplines that inform excellent testing. Successful testers tend to take on diversified skills, knowledge, and tactics, and when those skills are lacking, they collaborate with people who have them.

Note that I’m not suggesting here that anyone become a Donald Knuth-level programmer, a Pierre Bourdieu-league anthropologist, a Ross Ashby-class systems thinker, a Wittgenstein-grade philosopher. I am suggesting that testers be given sufficient training and opportunity to learn to program to the level of Brian Marick’s Everyday Scripting with Ruby, and that they be given classes, experience, and challenges in observation, the business domain, systems thinking and critical thinking. I am suggesting that people who are testing computer software do need some exposure to core ideas about logic (if we see this, can we justifiably infer that?), about ontology (what are our systems of knowledge about the way things work—especially related to computer programs and to testing), and about epistemology (how do we know what we know?).

I’ve been told by people involved in the design of testing standards that “you can’t expect regular testers to learn epistemology, for goodness’ sake”. Well, I’m saying that we can and that we must at least provide opportunities for learning, to the degree that testers can frame their mission, their ideas about risk, their testing, and their evaluation of the product in the ways that their clients value. Moreover, I’ve worked with testing organizations that have done that, and the results have been impressive. Sometimes I hear people saying “what if we train our testers and they leave?” As one wag on Twitter replied (I wish I knew who), “What if you don’t train them and they stay?”

In our classes, James Bach and I have the experience of inspriring testers to become interested in and excited by these topics. We find that it’s not hard to do that. We remain concerned about the capacity of some organizations to sustain that enthusiasm, often because some middle managers’ misconceptions about the practice and value of testing can squash both enthusiasm and value in a hurry. Testers, to be successful, must be given the freedom and responsibility to explore and to contribute what they’ve learned back to their team and to the rest of the organization.

So, what would we advise?

Read this set of ideas as a system, rather than as a linear list:

  • The purpose of testing is to identify threats to the value of the program. Functional errors are only one kind of threat to the value of the program.
  • Take on expansive ideas about what might constitute—or threaten—the quality of the product.
  • Dynamically manage your focus to exercise the product and test those ideas about value.
  • In hiring, staffing, and training, focus on the mindset and the skill set of the individual tester as a member of a highly diversified team.
  • As an individual tester, develop and diversify your skills and your strategies.
  • Immediately identify report issues that threaten the value of the testing effort and of the project generally. Solve the ones you can; raise team and management awareness of the costs and risks of issues, in order to get attention and help.
  • Learn to frame your testing and to compose, edit, narrate and justify a compelling testing story.
  • Don’t try to control or restrain testers; grant them the freedom—along with the responsibility to discover what they will. Given that… they will.

Posted in Exploratory Testing, Learning, Management, Systems Thinking, Test Framing, Tester Skill, Testing Story, Testing vs. Checking | 15 Comments »

Gaming the Tests

Monday, September 27th, 2010

Let’s imagine, for a second, that you had a political problem at work. Your CEO has promised his wife that their feckless son Ambrose, having flunked his university entrance exams, will be given a job at your firm this fall. Company policy is strict: in order to prevent charges of nepotism, anyone holding a job must be qualified for it. You know, from having met him at last year’s Christmas party, that Ambrose is (how to put this gently?) a couple of tomatoes short of a thick sauce. Yet the policy is explicit: every candidate must not only pass a multiple choice test, but must get every answer right. The standard number of correct answers required is (let’s say) 40.

So, the boss has a dilemma. He’s not completely out to lunch. He knows that Ambrose is (how can I say this?) not the sharpest razor in the barbershop. Yet the boss adamantly wants his son to get a job with the firm. At the same time, the boss doesn’t want to be seen to be violating his own policy. So he leaves it to you to solve the problem. And if you solve the problem, the boss lets you know subtly that you’ll get a handsome bonus. Equally subtly, he lets you know that if Ambrose doesn’t pass, your career path will be limited.

You ponder for a while, and you realize that, although you have to give Ambrose an exam, you have the authority to set the content and conditions of the exam. This gives you some possibilities.

A. You could give a multiple choice test in which all the answers were right. That way, anyone completing the test would get a perfect score.

B. You could give a multiple choice test for which the answers were easy to guess, but irrelvant to the work Ambrose would be asked to do. For example, you could include questions like, “What is the very bright object in the sky that rises in the morning and sets in the evening?” and provide “The Sun” as choice of answer, and the names of hockey players for the other choices.

C. You could find out what questions Ambrose might be most likely to answer correctly in the domain of interest, and then craft an exam based on that.

D. You could give a multiple choice test in which, for every question, one of A, B, or C was the correct answer, and answer D was always “One of the above.”

E. You might give a reasonably difficult multiple-choice exam, but when Ambrose got an answer wrong, you could decide that there’s another way to interpret the answer, and quietly mark it right.

F. You might give Ambrose a very long set of multiple-choice questions (say 400 of them), and then, of his answers, pick 40 correct ones. You then present those questions and answers as the completed exam.

G. You could give Ambrose a set of questions, but give him as much time as he wanted to provide an answer. In addition, you don’t watch him carefully (although not watching carefully is a strategy that nicely supports most of these options).

H. You could ask Ambrose one multiple choice question. If he got it wrong, correct him until he gets it right. Then you could develop another question, ask that, and if he gets it wrong, correct him until he gets it right. Then continue in a loop until you get to 40 questions.

I. This approach is like H, but instead you could give a multiple choice test for which you had chosen an entire set of 40 questions in advance. If Ambrose didn’t get them all right, you could correct him, and then give him the same set of questions again. And again. And over and over again, until he finally gets them all right. You don’t have to publicize the failed attempts; only the final, successful one. That might take some time and effort, and Ambrose wouldn’t really be any more capable of anything except answering these specific questions. But, like all the other approaches above, you could effect a perfect score for Ambrose.

When the boss is clamoring for a certain result, you feel under pressure and you’re vulnerable. You wouldn’t advise anyone to do any of the things above, and you wouldn’t do them yourself. Or at least, you wouldn’t do them consciously. You might even do them with the best of intentions.

There’s an obvious parallel here—or maybe not. You may be thinking of the exam in terms of a certain kind of certification scheme that uses only multiple-choice questions, the boss as the hiring manager for a test group, and Ambrose as a hapless tester that everyone wants to put into a job for different reasons, even though no one is particularly thrilled about the idea. Some critical outsider might come along and tell you point-blank that your exam wasn’t going to evaluate Ambrose accurately. Even a sympathetic observer might offer criticism. If that were to happen, you’d want to keep the information under your hat—and quite frankly, the other interested parties would probably be complacent too. Dealing with the critique openly would disturb the idea that everyone can save face by saying that Ambrose passed a test.

Yet that’s not what I had in mind—not specifically, at least. I wanted to point out some examples of bad or misleading testing, which you can find in all kinds of contexts if you put your mind to it. Imagine that the exam is a set of tests—checks, really. The boss is a product owner who wants to get the product released. The boss’ wife is a product marketing manager. Hapless Ambrose is a program—not a very good program to be sure, but one that everyone wants to release for different reasons, even though no one is particularly thrilled by the idea. You, whether a programmer or a tester or a test manager, are responsible for “testing”, but you’re really setting up a set of checks. And you’re under a lot of pressure. How might your judgement—consciously or subconsciously—be compromised? Would your good intentions bend and stretch as you tried to please your stakeholders and preserve your integrity? Would you admit to the boss that your testing was suspect? If you were under enough pressure, would you even notice that your testing was suspect?

So this story is actually about any circumstance in which someone might set up a set of checks that provide some illusion of success. Can you think of any more ways that you might game the tests… or worse, fool yourself?

Posted in Certification, Management, Measurement, Tester Skill, Testing vs. Checking | 22 Comments »

Encouraging Programmers to be Testers

Monday, September 20th, 2010

A colleague wrote to me recently and asked about a problem that he’s had in hiring. He says…

The kind of test engineers we’re looking for are ones that can think their way around a system and look for all the ways that things can go wrong (pretty standard, so far), and then code up a tool or system that can automatically verify that those things haven’t gone wrong (a bit more rare, especially, for some reason, in the part of the country where I’m working).

The problem is that I suspect there is a larger pool of candidates who fit this description, but think of themselves as software developers. They never consider a software engineer in test role because they think “oh, that’s QA”.

It’s a little more easy on the west coast—in Silicon Valley and the Pacific Northwest, I mean—because big companies like Microsoft, Google, and others define a lot of their test engineering roles as “software developers who build systems that just happen to test other systems as their product”.

Any thoughts on how to shift that perception here to be more closely aligned with the west coast?

My reply went like this:

I don’t like that particular perception, myself. I observe that the “Software Development Engineer in Test” view often presupposes that the SDET role is a consolation prize for not being hired as a Real Programmer on the one hand. On the other hand, the view poses a barrier to entry for non-programmers who want to be testers.

If your prospects think, “Oh, that’s QA” in this dismissive kind of way, what kind of testers are they going to be? Moreover, what kind of programmers are they going to be? To me and to our community, testing is questioning a product in order to evaluate it. Testing is a service to the project, wherein testers help to discover risks and problems that threaten the value of the product and the goals of the business. While testers are specialists in that kind of role, testing is something that any programmer should be doing too, as he or she is writing, reviewing, and revising code. So if prospects think that testing is some kind of second-class role or task, who needs them? Encourage them to take a hike and come back when they’ve learned not to be such prima donnas.

If offered a gaggle of programming enthusiasts to choose from, I’d prefer to hire the person who is most genuinely interested in testing. I’d give that person a toolsmith role, wherein (s)he provides programming services to other testers. In addition, the toolsmith can provide the service of aiding testers in learning to program, while the testers help the toolsmith to sharpen his/her testing skills.

If you’d like to motivate programmer-types to test, here’s a suggestion: Instead of treating testing as a programming exercise, treat it as a more general problem-solving task in which hand-crafted tools might be very helpful. James Bach and I do this by giving people testing puzzles that look simple but that have devious twists and traps. In my experience, most programmers (and all of the best ones) enjoy challenging intellectual problems, irrespective of whether programming is at the centre of solution. In our classes and at conferences and in jam sessions, we show programmers over and over again how they can be fooled by puzzles, just as anyone else can. We hasten to point out that programming can help to solve problems in a manner that can be far more more practical and useful than other approaches for solving some problems in some contexts. But as it goes with production code, so it goes with test code: working out how to solve the problem is the most challenging part, and writing a program is a means to that end, rather than the end in itself. The trick is to help people to recognize when they might want to emphasize writing code and when they might want to emphasize other approaches and other lines of investigation.

As Cem Kaner points out in his talk “Investment Modeling as an Exemplar of Exploratory Test Automation“, the best skilled testers (and especially those with programming skills) have the kind of mindset that we might easily associate with the best quantitative analysts. On Wall Street, they’re called quants, and they make a lot of money. In order to be successful and to reduce risk, both their programs and their models have to be useful, reliable, accurate—which means they have to be tested, and tested well.

Trouble is, many managers treat testing as a rote, clerical, bureaucratic, mechanistic activity. If that were all there is to testing, it would be a dead-end role and ripe for automation, but testing is so much more than coding up a set of checks. It’s up to us to help others to recognize what testing can do, by offering conversation, challenges, and leadership by example—and information about products and projects that people genuinely value. To do that, we need a community of testers who are passionate about their craft and practice it with skill, recognizing that it’s thinking—about risk, cost, value, and learning—that’s central.

But if you put the hammer and hammering at the centre of the task, you’ll get enthusiatic hammerers applying for the job, where I suspect you really want cabinet makers.

Posted in Management, Tester Skill | 2 Comments »

Hire Ben Simo!

Tuesday, August 31st, 2010

I have four or five blog posts in the hopper, each almost ready to go. I’m working on a whole book and a chapter of another one, and I’m on a deadline that I’m about to blow. The kids are still out of school, and I really should be cooking dinner right now. And yet…

As I write, one of the best testers that I know is looking for work. His name is Ben Simo. He lives in Colorado Springs, Colorado (my understanding is that he’s willing to relocate). He’s well-versed in LoadRunner and Performance Center, like many other testers. Unlike (alas!) so many other testers with those bullet points on the résumé, he’s not inclined merely to go through the motions and use tools for checking. He is an astute, passionate critical thinker, entirely focused on investigating and defending the value of the products for which he’s responsible by identifying problems that threaten that value. And yet’s he’s not of the Quality Assurance school; he entirely understands that assuring quality is the responsibility of those who produce and manage the work—programmers, writers, designers, artists, and managers. His job, as he sees it, is to make quality assurance possible. He collaborates with the project community, investigates the product ,and provides the most important, most timely information he can to the people who are producing and managing the work. With that, they can make the decisions they must make, informed by the very best technical information that he can provide.

He’s past President of the Association for Software Testing; he was Conference Chair for the 2009 Conference for the Association for Software Testing; he maintains one blog called Questioning Software and another called Is There A Problem Here?. He was recently given a three-part interview by UTest; you can read that here, and here, and here.

And, as of this writing, he’s available. For someone looking for a tester, he’s like the dream date that you spy across the dance floor whom a friend tells you is single, smart, modest, and well-off. The only issue is that maybe he’s a little too modest. He won’t be on the dance floor for long, because some organization will come along and sweep him off his feet. And that organization will be exceptionally lucky.

And that organization could be yours. His email address is ben at qualityfrog (period) com.

Postscript: Ben’s been hired by a company near Phoenix, AZ. You had your chance!

Posted in Community, Tester Skill | 3 Comments »

Heuristics and Leadership

Friday, May 28th, 2010

In a recent blog post, James Bach discusses the essence of heuristics. A heuristic is a fallible method for solving a problem or making a decision. When used as an adjective, “heuristic” means fallible and conducive to learning. James ends the post by introducing a number of questions in order to test whether someone is teaching you a heuristic effectively. Meeta Prakash, in the comments, remarks “Your questions sound so much like my idea and interpretation of a ‘leader’.”

Yes, Meeta, the asking of those questions sounds like leadership to me too. What is leadership? Leadership is “creating an environment in which everyone is empowered” (that’s Jerry Weinberg’s definition). We believe in that. We believe that excellent testing starts with the skill set and the mindset of the individual tester. Other things might help, but excellence in testing must centre on the tester.

Discussion of the Method is Billy Vaughan Koen’s book on engineering. He describes the engineering method as “the use of heuristics to cause the best change in a poorly understood situation within the available resources”. He notes that the individual engineer, the engineering organization in which he works, and the engineering discipline overall each has a state of the art, which he abbreviates as “sota”. These sotas overlap one another in some places and cover new ground elsewhere. Each leads and lags the others in certain areas. The overall discipline has a sota that is more advanced than that of the organization and the individual in some places. The organization’s sota is aware of things that neither the individual nor the discipline has yet recognized. Each individual’s sota contains some knowledge that is unknown to both the organization and the discipline.

Each sota may advance before that of the other two, and the advances are ongoing. Each sota evolves in a different context. For each stage of evolution, we can’t be certain about what factors might be relevant to success or failure. Thus no practice nor method nor approach can deemed to be best. We don’t know, can’t know, whether some method is best; we don’t know, can’t know, whether someone might invent a better method. We don’t know, can’t know, whether someone has already invented a better method. What we can do, however, is to create environments in which everyone is empowered to discover and apply new heuristics along with those that we already know. That’s important because the discipline and the organization never advance on their own. They advance when someone has the inspiration, the initiative, the courage, and the opportunity to try something new, uncertain as to whether the new approach will work.

Organizations and individuals can foster initiative by empowering people (or at the very least, by leaving them alone). The overall discipline tends not to encourage initiative much, so it seems. With prescriptive, restrictive standards, disciplines often discourage innovation. Why would disciplines do this? One reason is that disciplines are typically led by experts who, as McLuhan said, are heavily invested in their own expertise, and therefore resist change that would threaten that investment. Propaganda and the threat of unemployment are among the crude tools that experts wield.  Want evidence for that in testing? You need look no further than the “experts”, the certifiers, and the standards enthusiasts in our craft; their narrow and flawed models of evidence and measurement; their intolerance of uncertainty; and their resistance to acknowledging the exploratory mindset. Yet without exploration, progress in any domain ceases while the rest of the world rushes past.

“All is heuristic,” declares Koen. That’s an absolute statement which appears to declare that there are no absolutes. But rather than denying the paradox, Koen embraces it. All is heuristic, he says, including the heuristic that all is heuristic. The notion that all is heuristic is pretty robust. Koen points out that even algorithms have contexts in which they work and contexts in which they fail, and that algorithms must be chosen by people applying judgement and skill in uncertain conditions. Yet he leaves open the possibility that someone, somewhere, some day, might discover an infallible method for solving a problem.

James, our colleagues, and I deal with a similar paradox when we contend that all testing is heuristic. There’s no test for infallibility! It might be that there are some occasions when there are infallible methods for testing. It’s just that we’ve never seen one, and that we can’t currently imagine a case in which a process or a standard could—without the application of judgement or skill—be guaranteed to solve some testing problem. That’s because skilled testers (let’s call them “we”, without claiming expertise, but asserting that we are students of the craft) have specific advantages over process models, methods, and tools (“they”):

  • We have situational awareness (they don’t).
  • We work from the assumption that we’re fallible (they don’t).
  • We have the capacity to make judgements on questions of cost vs. value (they don’t).
  • We have the ability to apply a stopping heuristic at any time (they don’t).
  • We have the intelligence to choose which heuristics are applicable and which are not (they don’t).
  • We have the opportunity to consult, in real time, with our clients (they don’t).
  • We have the inventiveness to work around a problem (they don’t).
  • We have the sensory appartus to determine when a heuristic is failing (they don’t).
  • We have the humanity to notice something unexpected that might be a problem for people (they don’t).
  • We have the capacity to learn (they don’t).

And that’s only a partial list. A couple of notes: every one of these capabilities is itself heuristic; and it should be clear that I’m not talking only about skilled testers, but about any skilled discipline.

It’s not that bodies of knowledge or process models or standards never have anything interesting to say (although, in my experience, most of them do tend to be written in a style that induces immediate and profound slumber). It’s that none of these tools have any intrinsic relevance or value unless and until they are applied by people. If the tools are to be applied effectively, we need to recognize that they are all heuristic. We must test these heurstics and their validity with questions like the ones James poses.  We must also recognize the people applying them must have the skills to know how, when, and when not to apply them. To develop those heuristics and those skills, we need to create an environment in which everyone is empowered. That is what we believe.

Posted in Heuristics, Management, Tester Skill, Tools | 5 Comments »

Transpection Transpected

Tuesday, May 25th, 2010

Part of the joy of producing this blog is in seeing what happens when other people pick up the ideas and run with them.  That happened when I posted a scenario on management mistakes a few weeks ago, and Markus Gärtner responded with far more energy and thought than I would have expected. Thanks, Markus.

Last week I posted a transcript of a transpection session between me and James Bach.  The responses and the comments were very gratifying, but Oliver Vilson’s comment has sparked a discussion of its own. Oliver says,

I would have to say it is not only possible to test the clock-in-the-box but actually necessary.

I see it as an exercise when you have to test part of a system which you have no control over.

For example I’ve had problems with integration to the third party systems that gave absolute nonsense errors about things nobody could think of at that time and it messed up the correct behavior of the primary system pretty badly. We could do nothing but to observe what happened. Almost no possible way to change input data by end user. It either happened or not. But it ended up as very useful experience about testing.

I discussed your exercise with my colleague Rasmus and we found at least few ways to test it without giving it direct input itself

1) Expectations – for example: What format does it show time? Is it understandable?
2) End-values – turnover of seconds/minutes/hours where, for example, 59 -> 00
3) Load testing – how much does it starts to lie in 10 seconds, 1 minute, 1 hour, 1 day, 1 month, 1 year etc compared to let’s say quantum clock or NIST-F1.
4) What time zone time is it showing? Can be tricky because look at India’s time zone for example.
5) How long does the battery last before it shuts down? or before it starts to “lie”? How rapidly does it start to lie when batteries are running lower?
6) How are the digits shown? Are they visible via any other angle? Are they too small or too big?

And few ways to have direct input without moving or touching the box itself
1) Put powerful-enough magnet next to the box to see what happens.
2) Set EMP-bomb off near the box to see what happens.

With best regards
Oliver V.

I’ve had the pleasure of meeting Oliver Vilson a couple of times.  I find his thinking to be incisive and insightful, and he has provided me with a couple of excellent stories.  The first thing that Oliver has done here is to help with transfer:  the idea that our odd little thought experiment about the clock can be transferred to real-world contexts.  Oliver is right:  no matter what we test, much of the time we interact with things that are black boxes, closed to us.  Sometimes we have to take the operation of the black boxes on trust.  Other time we have to test them, and as we’re testing them, we’re nastily constrained by our inability to control or influence the factors in the experiment.  Identifying those factors, getting around those constraints (to the degree that we can), and figuring out what and how to observe are all central to testing skill.

As I was reading, it also occurred to me that Oliver’s list of test ideas could provide a very nice example of the way to use the HICCUPPS(F) mnemonic for oracle heuristics and the CRUSSPICSTMPL mnemonic (!) for quality criteria in both a retrospective and in a generative way.

Let’s recall:  HICCUPPS(F) is a mnenomic by which we remember consistency heuristics for oracles, the principles or mechanisms by which we might recognize a problem.  We perceive no problem when all of the following heuristics hold, and we suspect a problem when any one of the following heuristics is violated:

History: The present version of the system is consistent with past versions of itself.
Image: The system is consistent with an image that the organization wants to project.
Comparable Products: The system is consistent with comparable systems.
Claims: The system is consistent with what important people say it’s supposed to be.
Users’ Expectations: The system is consistent with what users want.
Product: Each element of the system is consistent with comparable elements in the same system.
Purpose: The system is consistent with its purposes, both explicit and implicit.
Statutes: The system is consistent with applicable laws.
That’s the HICCUPPS part.  What’s with the (F)?  “F” stands for “Familiar problems”:
Familiarity: The system is not consistent with the pattern of any familiar problem.

That is, we suspect a problem in the item to be tested if we see some consistency with a problem that we’ve seen before.  We perceive “no problem” in the item to be tested when it doesn’t present a familiar problem to us while we’re testing.  I’ve written about an earlier version of this list of oracle heuristics here.

The quality criteria for a product are those aspects of it that would tend to please favoured users—customers, or people who benefit from the efficient and accurate work of that customer.  Quality critieria can also be seen as things that would stymie disfavoured users—users that we don’t like, such as intruders, black hat hackers, snoops, denial-of-service enthusiasts, thieves, and so forth.

In the Rapid Software Testing course, we talk about quality criteria in terms of a set of guideword heuristics—labels for groups of ideas that trigger deeper analysis.  Our quality criteria include:

  • Capability
  • Reliability
  • Usability
  • Security
  • Scalability
  • Performance
  • Installability
  • Compatibility
  • Supportability
  • Testability
  • Maintainability
  • Portability
  • Localizability

These criteria are part of the Heuristic Test Strategy Model, first developed by James Bach.

So let’s look at Oliver’s example in terms of the oracles that are being used and the quality criteria that are being questioned here. I’ll start by tagging each test idea with one or more oracle heuristics and one or more quality criteria.

1) Expectations – for example: What format does it show time? Is it understandable?

Oracles:  user expectations, (implicit) purpose.  Quality critieria:  Usability, localizability

2) End-values – turnover of seconds/minutes/hours where, for example, 59 -> 00

Oracles:  user expectations, relevant standards.  Quality critieria:  capability, reliability.

3) Load testing – how much does it starts to lie in 10 seconds, 1 minute, 1 hour, 1 day, 1 month, 1 year etc compared to let’s say quantum clock or NIST-F1.

Oracles:  History, comparable products; familiar problem (clocks gaining or losing time).  Quality criteria:  reliability, performance.

4) What time zone time is it showing? Can be tricky because look at India’s time zone for example.

Oracles:  User expectations; implicit purpose.  Quality criteria:  usability, localizability.

5) How long does the battery last before it shuts down? or before it starts to “lie”? How rapidly does it start to lie when batteries are running lower?

Oracles:  History, user expectations.  Quality criteria:  Reliability, performance

6) How are the digits shown? Are they visible via any other angle? Are they too small or too big?

Oracles:  User expectations, implicit purpose.  Quality criteria:  Usability, testability.

Now, I’d like you to notice a few things.  First, the classifications that I’ve set here are my own.  They’re arbitrary.  You can agree with them or disagree.  That doesn’t matter so much.

What matters more, I think, is the excerise in which we think about the relationships between the test ideas, the quality criteria, the oracles, and the risks.  For a product of any kind, there’s risk associated with the idea that a relevant quality criterion of some kind will not be fulfilled.    By using the oracle and quality criteria guidewords, we can become conscious of the chaing of logic or “framing” of the test, which in turn helps us to compose, edit, narrate, and justify the product story and the testing story.

After we’ve applied oracle and quality-criteria tags to each of Oliver’s test ideas, we might start to notice some things. First, he has used a number of diverse heuristics by which he might recognize a problem.  In doing that, he has also identified tests that would address a number of quality criteria.  He did that quite spontaneously, without specifications or other documentation.  That is, as we’ve emphasized so often, it’s perfectly possible to test with incomplete or insufficient or inconsistent or ambiguous or out-of-date information, because

When information is missing, testing is a great way to generate it.

In providing a set of test ideas as he’s done, Oliver also brings to the surface a number of ideas and assumptions about the clock.  Whether those assumptions turn out to be right or wrong isn’t so important.  What’s far more important is getting started in observing similarities and differences between the assumptions and the reality.  The process of doing this is central to generating knowledge about the product.  This is very similar to Karl Weick’s observation, responding to a story in which a platoon of soldiers had a map that didn’t match the territory, but found their way home anyway:

“This raises the intriguing possibility that when you’re lost, any old map will do … maybe when you are confused any old strategic plan will do. Strategic plans are a lot like maps. They animate and orient people. Once people begin to act, they generate tangible outcomes in some context, and this helps them discover what is occurring, what needs to be explained, and what should be done next. Managers keep forgetting that it is what they do, not what they plan, that explains their success. They keep giving credit to the wrong thing—namely, the plan—and having made this error, they then spend more time planning and less time acting. They are astonished when more planning improves nothing.“  (Karl Weick, Sensemaking in Organizations, p. 54-55)

Oliver’s list (implicitly) includes test ideas that take advantage of the user expectations, comparable product, purpose, standards, and familiar problem heuristics.  We can see and justify what’s there by comparing it with the HICCUPPS(F) list, and noting that inconsistency with those items would point us to a problem.  “User expectations” seems to dominate the list of oracle heuristics.  One question we could ask is “how might we refine or expand the set of user expectations that we have?”  Another question is “are our ideas about oracles overloaded in the direction of user expectations?”

We can use the HICCUPPS(F) list to see what’s there, but with the list we can also see what might be missing:  questions about history (is there another clock like this?  is this the first one that we’ve ever seen?); about image (who is our client here?  what are possible perceptions that the client might want to project?); claims (what do people say about this clock, anyway? how is it supposed to work?  is there any useful information, whether documented or not, on this?); product (can we learn anything about the product by observing parts of it that should be consistent with one another? does the product include any internal sanity checks?).

Similarly, we can use the quality criteria list to help us generate ideas based on the things that might threaten the value of the product.  We can see some test ideas based on capability, reliability, usability, performance, and localizability.  What other factors might we choose to consider?  Which ones might be more important in our testing mission?  Less important?  Are there any that are crucial, or irrelevant?

Are there security concerns related to the clock?  Why is it in this box?  What would happen if someone were to get inside?  Could the functioning of the clock be affected by heat, cold, light, acceleration, bombardment?  What are the boundaries between the clock, its containers, and other systems?   Scalability:  is this a prototype clock, or are there going to be many like it?  Could it be used for very short-term or long-term measurements of time?  What if large numbers of people need access to the information it provides?  Installability:  How did it get there?  Can it be updated?  How would we get rid of it?  Compatibility:  does the clock interface with anything else?  How?  Supportability:  What do we do if someone has a problem with the clock? Can we get at it then?  How?  And if we can get at it then, why not now?  Testability:  You say that there is no way to provide input to the clock.  Really?  Is there some other way that you might be interpreting “input”?  What interfaces might be available?  What reference material?  What oracles?  Does the clock produce any information other than its display?  Are there any markings on it?  Guides to its internals?  Maintainability:  Supposedly I’m testing this because you want to be able to identify problems with it.  Do you want to be able to fix those problems?  Who would be responsible for doing that?  Is there source code or are there architectural drawings for the program that runs the clock?  Portability:  does that program work on other clocks?  What information can we learn about this clock that might be transferrable to other clocks?

As tools to help us see what’s there and see what’s missing, we can use the HICCUPPS(F) list to evaluate our oracles.  We can use the quality criteria list to evaluate our requirements coverage and make decisions about it.  At some point, we’ll also talk about product elements that point to coverage ideas.  We’ll also talk about the project environment that influences our context and our choices, both of which evolve over time.  But for now, that’s for later.  Thank you to Oliver for providing an excellent example on which, in this space, we could do a little something like transpection.

Posted in Coverage, Oracles, Test Framing, Tester Skill, Testing Story | 4 Comments »

Questions from Listeners (1): Handling Inexperienced Testers

Wednesday, May 19th, 2010

On April 19, 2010, I was interviewed by Gil Broza.  In preparation for that interview, we solicited questions from the listeners, and I promised to answer them either in the interview or in my blog.  Here’s the first one.

How to deal with un-experienced testers? is there a test approach that suits better for them?

Here’s what I’d do: I’d train them. I’d pair them up with more experienced testers (typically test leads or coaches). I’d start them with things that are relatively easy to test, gradually increasing the difficulty of the assignments and the responsibilities of the tester. I’d watch closely to make sure that the test leads and the novices were talking to each other a lot during testing sessions.  I’d debrief them both together and individually.  I’d have the novices read books, articles, and blog posts, and ask them for summary reviews, and then we’d talk about them. I’d give them experiential exercises, in in which they and other members of the team would try to solve a testing puzzle, and then we’d talk about it afterwards. I’d reward them for demonstrating increasing skill: participating in Weekend Testing sessions; writing blog posts or articles for internal or external consumption; building tools and contributing to our development group or to the wider community.

Some might object that this approach is time-consuming. It certainly consumes some time, but it’s the fastest way I know to develop skill, and it’s similar in a general way to the training approaches for doctors, pilots, and skilled trades: involve them in the work, supervise them closely, and make them increasingly responsible for increasingly challenging work.

Here’s what I wouldn’t do: I wouldn’t give them a script to follow unsupervised and then presume that they’re going to do or learn the work. Everything that we know about learning and about testing suggests that both will be highly limited with this hands-off approach.  My friend Joey McAllister recently tweeted “The Barista at this Target Starbucks didn’t know if a mocha came with a shot of espresso in it. And she was working alone.” Most of the products that we’re testing aren’t simple cups of mocha. People are likely to suffer loss or harm if we take the approach that someone took with this barista.

Posted in Tester Skill, Training | 6 Comments »

Testers: Get Out of the Quality Assurance Business

Monday, May 3rd, 2010

The other day, Cory Foy tweeted a challenge: “Having a QA department is a sign of incompetency in your Development department. Discuss.”

Here’s what I think: I’m a tester, and it’s time for our craft to grow up. Whatever the organizational structure of our development shops, it’s time for us testers to get out of the Quality Assurance business.

In the fall of 2008, I was at the Amplifying Your Effectiveness conference (AYE), assisting Fiona Charles and Jerry Weinberg with a session called “Testing Lies”. Jerry was sitting at the front of the room, and as people were milling in and getting settled, I heard him in the middle of a chat with a couple of people sitting close to him. “You’re in quality assurance?” he asked. Yes, came the answer. “So, are you allowed to change the source code for the programs you test?” No, definitely not. “That’s interesting. Then how can you assure quality?”

A good question, and one that immediately reminded me of a conversation more than ten years earlier. In the fall of 1996, Cem Kaner presented his Black Box Software Testing course at Quarterdeck. I was a program manager at the time, but the head of Quality Assurance (that is, testing) had invited me to attend the class. As part of it, Cem led a discussion as to whether the testing group should really be called “Quality Assurance”. His stance was that individuals—programmers and testers alike—could certainly assure the quality of their own work, but that testers could not assure the quality of the work of others, and shouldn’t try it. The quality assurance role in the company, Cem said, lay with the management and the CEO (the principal quality officer in the company), since it was they—and certainly not the testers—who had the authority to make decisions about quality. Over the years he has continued to develop this idea, principally in versions of his presentations and papers on The Ongoing Revolution in Software Testing, but the concept suffuses all of his work. The role for us is not quality assurance; we don’t have control over the schedule, the budget, programmer staffing, product scope, the development model, customer relationships, and so forth. But when we’re doing our best work, we’re providing valuable, timely information about the actual state of the product and the project. We don’t own quality; we’re helping the people who are responsible for quality and the things that influence it. “Quality assistance; that’s what we do.”

More recently, in an interview with Roy Osherove, James Bach also notes that we testers are not gatekeepers of quality. We don’t have responsibility for quality any more than anyone else; everyone on the development team has that responsibility. When he first became a test manager at Apple Computer way back when, James was energized by the idea that he was the quality gatekeeper. “I came to realize later that this was terribly insulting to everyone else on the team, because the subtle message to everyone else on the team is ‘You guys don’t really care, do you? You don’t care like I do. I’m a tester, that means I care.’ Except the developers are the people who create quality; they make the quality happen. Without the developers, nothing would be there; you’d have zero quality. So it’s quite insulting, and when you insult them like that, they don’t want to work with you.”

Last week, I attended the STAR East conference in Orlando. Many times, I was approached by testers and test managers who asked for my help. They wanted me to advise them on how they could get programmers to adopt “best practices”. They wanted to know how they could influence the managers to do a better job. They wanted to know how to impose one kind of process model or another on the development team. In one session, testers bemoaned the quality of the requirements that they were receiving from the business people (moreover, repeating a common mistake, the testers said “requirements” when they meant requirement documents). In response, one fellow declared that when he got back to work, the testers were going to take over the job of writing the requirements.

In answer to the request for advice, the most helpful reply I can give, by far, is this: these are not the businesses that skilled testers are in.

We are not the brains of the project. That is to say, we don’t control it. Our role is to provide ESP—not extra-sensory perception, but extra sensory perception. We’re extra eyes, ears, fingertips, noses, and taste buds for the programmers and the managers. We’re extensions of their senses. At our best, we’re like extremely sensitive and well-calibrated instruments—microscopes, telescopes, super-sensitive microphones, vernier calipers, mass spectrometers. Bomb-sniffing detectors. (The idea that we are the test instruments comes to me from Cem Kaner.) We help the programmers and the managers to see and hear and otherwise sense things that, in the limited time available to them, and in the mindset that they need to do their work, they might not be able to sense on their own.

Listen: if you really want to improve the quality of the code and think that you can, become a programmer. I’ve done that. I can assure you that if you do it too, you’ll quickly find out how challenging and humbling a job it is to be a truly excellent programmer—because like all tools, the computer extends your incompetence as quickly and powerfully as it extends your competence. If you want to manage the project, become a project manager. I’ve done that too, and I was pretty good at it. But try it, and you’ll soon discover that quality is value to some person or persons who matter, and that your own standards of quality are far less important than those who actually use the product and pay the bills. Become a project manager, and you’ll almost immediately realize that the decision to release a product is informed by technical issues but is ultimately a business decision, balancing the value of the product and bugs—threats to the value of the product—against the costs of not releasing it.

In neither case would I have found it helpful for someone who was neither a programmer nor a project manager to whine to me that I didn’t have respect for quality; worse still would have been instruction on how to do my job from people who had never done that kind of work. In both of those roles, what I wanted from testers was information. As a programmer, I wanted to know about problems the testers had found in my code, how they had found them, the ways in which the product might not work, and the steps and clues I needed to point me towards finding and fixing the problem myself. As a program manager, I wanted to know what information the testers had gathered about the product, and how they had configured, operated, observed, and evaluated the product to get that information. I wanted to know how our product worked differently from the ways in which it had always worked. I wanted to know about internal inconsistencies. I wanted to know how our product worked in comparison to other products on the market; how it was worse, how it was better. I wanted to know how the product stacked up against claims that we were making for it.

So:  you want to have an influence on quality, and on the team.  Want to know how to influence the programmers in a positive way?

  • Tell the programmers, as James suggests in the interview, that your principal goal is to help them look good—and then start believing it. Your job is not to shame, or to blame, or to be evil. I don’t think we should even joke about it, because it isn’t funny.
  • You’re often the bearer of bad news. Recognize that, and deliver it with compassion and humility.
  • You might be wrong too.  Be skeptical about your own conclusions.
  • Focus on exploring, discovering, investigating, and learning about the product, rather than on confirming things that we already know about it.
  • Report what you’ve learned about the product in a way that identifies its value and threats to that value.
  • Try to understand how the product works on all of the levels you can comprehend, from the highest to the lowest.  Appreciate that it’s complex, so that when you have some harebrained idea about how simple it is to fix a problem, or to find all the bugs in the code, you can take the opportunity to pause and reflect.
  • Express sincere interest in what programmers do, and learn to code if that suits you. At least, learn a little something about how code works and what it does.
  • Don’t ever tell programmers how they should be doing their work. If you actually believe that that’s your role, try a reality check: How do you like it when they do that to you?

Want to know how to influence the managers?

  • Provide them with the information they need to make informed decisions, and then let them make the decisions.
  • Remain fully aware that they’re making business decisions, not just technical ones.
  • Know that the product doesn’t necessarily have to hew to your standard of quality.
  • It’s not the development manager’s job, nor anyone else’s, to make you happy. It might be part of their job to help you to be more productive.  Help them understand how to do that.  Particularly draw attention to the fact that…
  • Issues that slow down testing are terribly important, because they allow bugs the opportunity to hide for longer and deeper. So report not only bugs in the product, but issues that slow down testing.
  • If you want to provide information to improve the development process, report on how you’re actually spending your time.
  • Note, as is so commonly the case, why testing is taking so long—how little time you’re spending on actually testing the product, and how much time you’re spending on bug investigation and reporting, setup, meetings, administrivia, and other interruptions to obtaining test coverage.
  • Focus on making these reports accurate (say to the nearest five or ten per cent) rather than precise, because most problems that we have in software development can be seen and solved with first-order measures rather than six-decimal analyses derived from models that are invalid anyway.
  • Show the managers that the majority of problems that you find aren’t exposed by mindless repetition of test cases, but by actions and observations that the test cases don’t cover—that is, by your sapient investigation of the product.
  • Help managers and programmers alike to recognize that test automation is more, far more, than programming a machine to pound keys.
  • Help everyone to understand that automation extends some kinds of testing and greatly limits others.
  • Help to keep people aware of the difference between testing and checking.  Help them to recognize the value of each, and that excellent checking requires a great deal of testing skill.
  • Work to demonstrate that your business is skilled exploration of the product, and help managers to realize that that’s how we really find problems.
  • Help the team to avoid the trap of thinking of software development as a linear process rather than an organic one.
  • Help managers and programmers to avoid confusing the “testing phase” with what it really is: the fixing phase.
  • When asked to “sign off” on the product, politely offer to report on the testing you’ve done, but leave approval to those whose approval really matters: the product owners.

Want to earn the respect of your team?

  • Be a service to the project, not an obstacle. You’re a provider of information, not a process enforcer.
  • Stop trying to “own” quality, and take as your default assumption that everyone else on the project is at least as concerned about quality as you are.
  • Recognize that your role is to think critically—to help to prevent programmers and managers from being fooled—and that that starts with not being fooled yourself.
  • Diversify your skills, your team, and your tactics. As Karl Weick says, “If you want to understand something complicated, you have to complicate yourself.”
  • As James Bach says, invent testing for yourself. That is, don’t stop at accepting what other people say (including me); field-test it against your own experience, knowledge, and skills. Check in with your community, and see what they’re doing.
  • If there’s something that you can learn that will help to sharpen your knowledge or your senses or your capacity to test, learn it.
  • Study the skills of testing, particularly your critical thinking skills, but also work on systems thinking, scientific thinking, the social life of information, human-computer interaction, data representation, programming, math, measurement…
  • Practice the skills that you need and that you’ve read about. Sharpen your skills by joining the Weekend Testing movement.
  • Get a skilled mentor to help you. If you can’t find one locally, the Internet will provide. Ask for help!
  • Don’t bend to pressure to become a commodity. Certification means that you’re spending money to become indistinguishable from a hundred thousand other people. Make your own mark.
  • Be aware that process models are just that—models—and that all process models—particularly those involving human activities like software development—leave out volumes of detail on what’s really going on.
  • Focus on developing your individual skill set and mindset, so that you can be adaptable to any process model that comes along.
  • Share your copy of Lessons Learned in Software Testing and Perfect Software and Other Illusions About Testing.

Ultimately, if you’re a tester, get out of the quality assurance business. Be the best tester you can possibly be: a skilled investigator, rather than a programmer or project manager wannabe.

Note: every link above points to something that I’ve found to be valuable in developing the thinking and practice of testing. Some I wrote; most I didn’t.  Feast your mind!

Posted in Management, Quality, Tester Skill | 52 Comments »

« Older Entries
Newer Entries »