Blog: Testers: Get Out of the Quality Assurance Business

The other day, Cory Foy tweeted a challenge: “Having a QA department is a sign of incompetency in your Development department. Discuss.”

Here’s what I think: I’m a tester, and it’s time for our craft to grow up. Whatever the organizational structure of our development shops, it’s time for us testers to get out of the Quality Assurance business.

In the fall of 2008, I was at the Amplifying Your Effectiveness conference (AYE), assisting Fiona Charles and Jerry Weinberg with a session called “Testing Lies”. Jerry was sitting at the front of the room, and as people were milling in and getting settled, I heard him in the middle of a chat with a couple of people sitting close to him. “You’re in quality assurance?” he asked. Yes, came the answer. “So, are you allowed to change the source code for the programs you test?” No, definitely not. “That’s interesting. Then how can you assure quality?”

A good question, and one that immediately reminded me of a conversation more than ten years earlier. In the fall of 1996, Cem Kaner presented his Black Box Software Testing course at Quarterdeck. I was a program manager at the time, but the head of Quality Assurance (that is, testing) had invited me to attend the class. As part of it, Cem led a discussion as to whether the testing group should really be called “Quality Assurance”. His stance was that individuals—programmers and testers alike—could certainly assure the quality of their own work, but that testers could not assure the quality of the work of others, and shouldn’t try it. The quality assurance role in the company, Cem said, lay with the management and the CEO (the principal quality officer in the company), since it was they—and certainly not the testers—who had the authority to make decisions about quality. Over the years he has continued to develop this idea, principally in versions of his presentations and papers on The Ongoing Revolution in Software Testing, but the concept suffuses all of his work. The role for us is not quality assurance; we don’t have control over the schedule, the budget, programmer staffing, product scope, the development model, customer relationships, and so forth. But when we’re doing our best work, we’re providing valuable, timely information about the actual state of the product and the project. We don’t own quality; we’re helping the people who are responsible for quality and the things that influence it. “Quality assistance; that’s what we do.”

More recently, in an interview with Roy Osherove, James Bach also notes that we testers are not gatekeepers of quality. We don’t have responsibility for quality any more than anyone else; everyone on the development team has that responsibility. When he first became a test manager at Apple Computer way back when, James was energized by the idea that he was the quality gatekeeper. “I came to realize later that this was terribly insulting to everyone else on the team, because the subtle message to everyone else on the team is ‘You guys don’t really care, do you? You don’t care like I do. I’m a tester, that means I care.’ Except the developers are the people who create quality; they make the quality happen. Without the developers, nothing would be there; you’d have zero quality. So it’s quite insulting, and when you insult them like that, they don’t want to work with you.”

Last week, I attended the STAR East conference in Orlando. Many times, I was approached by testers and test managers who asked for my help. They wanted me to advise them on how they could get programmers to adopt “best practices”. They wanted to know how they could influence the managers to do a better job. They wanted to know how to impose one kind of process model or another on the development team. In one session, testers bemoaned the quality of the requirements that they were receiving from the business people (moreover, repeating a common mistake, the testers said “requirements” when they meant requirement documents). In response, one fellow declared that when he got back to work, the testers were going to take over the job of writing the requirements.

In answer to the request for advice, the most helpful reply I can give, by far, is this: these are not the businesses that skilled testers are in.

We are not the brains of the project. That is to say, we don’t control it. Our role is to provide ESP—not extra-sensory perception, but extra sensory perception. We’re extra eyes, ears, fingertips, noses, and taste buds for the programmers and the managers. We’re extensions of their senses. At our best, we’re like extremely sensitive and well-calibrated instruments—microscopes, telescopes, super-sensitive microphones, vernier calipers, mass spectrometers. Bomb-sniffing detectors. (The idea that we are the test instruments comes to me from Cem Kaner.) We help the programmers and the managers to see and hear and otherwise sense things that, in the limited time available to them, and in the mindset that they need to do their work, they might not be able to sense on their own.

Listen: if you really want to improve the quality of the code and think that you can, become a programmer. I’ve done that. I can assure you that if you do it too, you’ll quickly find out how challenging and humbling a job it is to be a truly excellent programmer—because like all tools, the computer extends your incompetence as quickly and powerfully as it extends your competence. If you want to manage the project, become a project manager. I’ve done that too, and I was pretty good at it. But try it, and you’ll soon discover that quality is value to some person or persons who matter, and that your own standards of quality are far less important than those who actually use the product and pay the bills. Become a project manager, and you’ll almost immediately realize that the decision to release a product is informed by technical issues but is ultimately a business decision, balancing the value of the product and bugs—threats to the value of the product—against the costs of not releasing it.

In neither case would I have found it helpful for someone who was neither a programmer nor a project manager to whine to me that I didn’t have respect for quality; worse still would have been instruction on how to do my job from people who had never done that kind of work. In both of those roles, what I wanted from testers was information. As a programmer, I wanted to know about problems the testers had found in my code, how they had found them, the ways in which the product might not work, and the steps and clues I needed to point me towards finding and fixing the problem myself. As a program manager, I wanted to know what information the testers had gathered about the product, and how they had configured, operated, observed, and evaluated the product to get that information. I wanted to know how our product worked differently from the ways in which it had always worked. I wanted to know about internal inconsistencies. I wanted to know how our product worked in comparison to other products on the market; how it was worse, how it was better. I wanted to know how the product stacked up against claims that we were making for it.

So:  you want to have an influence on quality, and on the team.  Want to know how to influence the programmers in a positive way?

  • Tell the programmers, as James suggests in the interview, that your principal goal is to help them look good—and then start believing it. Your job is not to shame, or to blame, or to be evil. I don’t think we should even joke about it, because it isn’t funny.
  • You’re often the bearer of bad news. Recognize that, and deliver it with compassion and humility.
  • You might be wrong too.  Be skeptical about your own conclusions.
  • Focus on exploring, discovering, investigating, and learning about the product, rather than on confirming things that we already know about it.
  • Report what you’ve learned about the product in a way that identifies its value and threats to that value.
  • Try to understand how the product works on all of the levels you can comprehend, from the highest to the lowest.  Appreciate that it’s complex, so that when you have some harebrained idea about how simple it is to fix a problem, or to find all the bugs in the code, you can take the opportunity to pause and reflect.
  • Express sincere interest in what programmers do, and learn to code if that suits you. At least, learn a little something about how code works and what it does.
  • Don’t ever tell programmers how they should be doing their work. If you actually believe that that’s your role, try a reality check: How do you like it when they do that to you?

Want to know how to influence the managers?

  • Provide them with the information they need to make informed decisions, and then let them make the decisions.
  • Remain fully aware that they’re making business decisions, not just technical ones.
  • Know that the product doesn’t necessarily have to hew to your standard of quality.
  • It’s not the development manager’s job, nor anyone else’s, to make you happy. It might be part of their job to help you to be more productive.  Help them understand how to do that.  Particularly draw attention to the fact that…
  • Issues that slow down testing are terribly important, because they allow bugs the opportunity to hide for longer and deeper. So report not only bugs in the product, but issues that slow down testing.
  • If you want to provide information to improve the development process, report on how you’re actually spending your time.
  • Note, as is so commonly the case, why testing is taking so long—how little time you’re spending on actually testing the product, and how much time you’re spending on bug investigation and reporting, setup, meetings, administrivia, and other interruptions to obtaining test coverage.
  • Focus on making these reports accurate (say to the nearest five or ten per cent) rather than precise, because most problems that we have in software development can be seen and solved with first-order measures rather than six-decimal analyses derived from models that are invalid anyway.
  • Show the managers that the majority of problems that you find aren’t exposed by mindless repetition of test cases, but by actions and observations that the test cases don’t cover—that is, by your sapient investigation of the product.
  • Help managers and programmers alike to recognize that test automation is more, far more, than programming a machine to pound keys.
  • Help everyone to understand that automation extends some kinds of testing and greatly limits others.
  • Help to keep people aware of the difference between testing and checking.  Help them to recognize the value of each, and that excellent checking requires a great deal of testing skill.
  • Work to demonstrate that your business is skilled exploration of the product, and help managers to realize that that’s how we really find problems.
  • Help the team to avoid the trap of thinking of software development as a linear process rather than an organic one.
  • Help managers and programmers to avoid confusing the “testing phase” with what it really is: the fixing phase.
  • When asked to “sign off” on the product, politely offer to report on the testing you’ve done, but leave approval to those whose approval really matters: the product owners.

Want to earn the respect of your team?

  • Be a service to the project, not an obstacle. You’re a provider of information, not a process enforcer.
  • Stop trying to “own” quality, and take as your default assumption that everyone else on the project is at least as concerned about quality as you are.
  • Recognize that your role is to think critically—to help to prevent programmers and managers from being fooled—and that that starts with not being fooled yourself.
  • Diversify your skills, your team, and your tactics. As Karl Weick says, “If you want to understand something complicated, you have to complicate yourself.”
  • As James Bach says, invent testing for yourself. That is, don’t stop at accepting what other people say (including me); field-test it against your own experience, knowledge, and skills. Check in with your community, and see what they’re doing.
  • If there’s something that you can learn that will help to sharpen your knowledge or your senses or your capacity to test, learn it.
  • Study the skills of testing, particularly your critical thinking skills, but also work on systems thinking, scientific thinking, the social life of information, human-computer interaction, data representation, programming, math, measurement
  • Practice the skills that you need and that you’ve read about. Sharpen your skills by joining the Weekend Testing movement.
  • Get a skilled mentor to help you. If you can’t find one locally, the Internet will provide. Ask for help!
  • Don’t bend to pressure to become a commodity. Certification means that you’re spending money to become indistinguishable from a hundred thousand other people. Make your own mark.
  • Be aware that process models are just that—models—and that all process models—particularly those involving human activities like software development—leave out volumes of detail on what’s really going on.
  • Focus on developing your individual skill set and mindset, so that you can be adaptable to any process model that comes along.
  • Share your copy of Lessons Learned in Software Testing and Perfect Software and Other Illusions About Testing.

Ultimately, if you’re a tester, get out of the quality assurance business. Be the best tester you can possibly be: a skilled investigator, rather than a programmer or project manager wannabe.

Note: every link above points to something that I’ve found to be valuable in developing the thinking and practice of testing. Some I wrote; most I didn’t.  Feast your mind!

68 Responses to “Testers: Get Out of the Quality Assurance Business”

  1. [...] for this team; they can sit together. Testers aren’t involved (yet) — I talked with the QA manager (who is trying to manage the day to day work of testers on two agile projects plus a few [...]

  2. [...] and a “Quality Assurance Manager.” We all know now, or should, that those roles have nothing to do with quality assurance. I didn’t write feature code as a QA analyst and I didn’t dictate to programmers how to [...]

  3. Miguel says:

    I think Quality Assurance may be a good name to a team in some cases, at my company the testing team have different responsibilities besides testing software such as document analysis, questioning development processes, software-related legal issues delegated to software, software-related political issues, etc… What name would you think for a department like this one?

    I understand you could think that the main problem here is the responsibilities taken by the testers but let’s face it, if it is what it is, and you only can switch the name, which would it be?

    Michael replies: Let’s make it clear that I can’t—and I shouldn’t—enforce orthodoxy, so this is an opinion, not a statement of dogma or best practice.

    You could call the testing group “quality assistance”, or “quality awareness”, if you want to keep the QA abbreviation. Consider that programmers do all of the things that you mention above, at least to some degree. Programmers do more than programming, but we still call them “programmers”. So it seems to me that “testers” could do all that stuff, and you could still call them testers. Why not call them “testers”?

    The point of the post is not really about the label; it’s about the idea. Whatever else testers might do, they don’t assure quality. They do provide information to the people who DO assure quality.

  4. [...] do you think? Is this topic important for testing? Should we treat is in the way Michael Bolton advises testers with respect to “Quality Assurance Business” (spoiler alert/hint: He advises to stay out of [...]

  5. [...] Read this post on [...]

  6. [...] ? ?????????, ????????? ?????? ?????? ??????? – “Testers: Get Out of the Quality Assurance Business”. ??? ?????? ?? ???????. ??????? ????? – [...]

  7. Kyo says:

    From my personal experience what developers tend to appreciate most of all it’s the fact that even when you’re a QA you can talk with them at the same level about the code, I use to be a former programmer a really good one, but because of a bad experience in a previous job (lack of creative ways or innovation paths to develop new features due to manager restrictions) I decided to switch to QA and what I have found it’s that most of the teams ussually start the project giving me the evil eye “grrrr that qa guy I don’t like him…” but when they start seeing me on the code reviews exposing my personal opinions on ***what are the best ways to implement something from a QA perspective***, they see the tools that I program to automate the several applications, or even libraries for commercial or open source automation/performance solutions and last but most importante they see a change on how I report the bugs that I found, I can’t stand out this enough but every QA should know where in the code is the defect that they just found, so you can report in an accurate way to the programmers and minimize the resolving time a great deal, it’s not the same when you say “login is failing when I type blah blah blah” instead of saying “the XXXX field on this function/method/class is throwing THIS exception when I give this input/s”.
    Then you have their respect and they’ll treat you as an equal that is helping them to develop a better software, and ofcourse you’ll have a direct impact in the project schedule and quality being a member that any team will want on their line up.
    Ofcourse is a must to be polite and always treat the code that those developers have spent hours of creative thinking to create with the respect that deserves as stated on this article.
    I disagree with the idea that if you know how to code and have opinions about the source of your current project you should move to the dev team, I’m more of the opinion that the days of the black box testing should be done and buried QAs need to know the code that they’re testing and the techonolgies that are being used to create it in order to provide even more information to the managers that will make the call to deliver the product or not.

    Michael replies: I agree that it’s a good idea for some testers, at least, to understand technology at the code level. And it’s a good idea for testers to understand the technologies that they’re testing. I’ll go further: often testers develop knowledge about the technologies that are being developed. But I don’t agree that “the days of black box testing should be dead and buried”. I fine with not seeing the code of things that I’m testing. It affords a different perspective that can be especially valuable when everyone else is focused on the code. If you believe that every tester must know software on a coding level, you’d have to agree that every tester should also understand design, usability, security, performance, diagramming, systems thinking, critical thinking, all domain knowledge, and should have a complete grasp of how to write grammatically correct English sentences. Otherwise you’d you’d have to explain why those things aren’t important. As alternative, perhaps you’d acknowledge that a really good development team consists of a number of people with diverse talents and skills working together, and that it’s okay for a tester not be a coder when the tester brings other skills and perspectives to the table.

  8. Baby Bunny says:

    Wow and thank you for your time to create this thread I am finding it invalueable. At present I’m seeking a career change and am a zygote within the IT arena, if its ok to label it that. What you have stated applies to all industries in a transferable skills/idea’s set way. That little box might be green this side however the opposite side might be red, oh wait… your colour blind and that box is actually big, now its not even a box… wow how can I see what in it if its not a box? hmm
    I guess sometimes we get so focused on the problems we forget to focus on their solutions. I’m going to look at what developers do within this industry to see if my skills can help there, thanks again for the reading tools.

  9. […] that I’m advocating to getting into Quality Assurance business as Micheal Bolton has very strong arguments against it. But that the ownership of Quality is shared amongst the team rather than just the […]

  10. […] online forum of some sort. Also, I would like to add that Michael Bolton has (as in most topics), covered this specific topic much more extensively than I, and is a great source of information in […]

  11. Pat Barnes says:

    I like a lot of what you said. Especially your recommendations to testers. But I think you have your “assure” and “ensure” mixed up. Quality Assurance doesn’t own, let alone “create” quality, but it provides confidence to the consumer of the product or results of a process that it is of a particular measure of quality (e.g., no sev 1 defects, or no salmonela poisoning).

    Michael replies: One of these days I’ll have to do a long post on the confidence issue. For now, here’s a short one (along with some longer comments).

    As testers, we cannot report that there are no Severity 1 defects in the product. At best, we can report that we have found no Severity 1 defects in the product, but we cannot achieve complete coverage, we are not omniscient, and we are not perfect. As with salmonella, you can’t observe the product (patient) and find evidence of no salmonella; you can only fail to find evidence of salmonella.

    Bugs don’t necessarily present immediately. Some do, but others remain latent for some time. Salmonella doesn’t present symptoms immediately, but over several hours, and the incubation period may be as long as a day. Observing your lunchtime customers doesn’t tell you in time for dinner that you’ve got clean prep surfaces. Even by lunchtime today, you don’t know that everything was perfectly clean yesterday; only that your customers have not developed an infection—but perhaps that’s because their immune response is very robust, and they were not exposed to enough contamination for illness to develop.

    The incubation period for discovering a bug may be even longer—years. Software is a domain that lives in the world of black and grey swans. We cannot present reasons why our clients can be confident, but we can show (or fail to show) reasons why they could doubt their confidence. Excellent testing and test reporting depends on understanding the extent and limits of what we know and what we can’t know.

  12. Pat Barnes says:

    I hear what you are saying, but let me try another approach. I buy stuff. Mostly electronic stuff and cars, and my wife buys everything else. I rely heavily on Consumer Reports. I like the testing that they do and the fact that they are independent of the folks who are trying to sell me stuff. They provide me with “quality assurance.” They do nothing to actually “ensure” the product has quaility, but the fact that they are looking at it in some detail, and measuring it against some benchmarks, e.g., reported defects by readers in a survey, provides me with more assurance that a product will be reliable–or more reliable than some competing product–than if I just pick the pretty one. What they do is testing (and data gathering). What they provide to me the consumer or stakeholder is QA. It is not a lot of other things. It is not a garnatee that there are no defects. That’s not what I said. That’s not QA anyway, and your thesis isn’t no defects, it’s that testing is not QA. Consumer Reports can’t tell me that my new Honda has no defects. But they can provide me with some assurance that If I buy the Honda, I’m likely to get less defects than if I buy the Yugo. This is what Testers do for the customer. Confidence is also not binary. So in fact when CR tells me about their extensive testing process, and why the recommend the Honda, they do in deed provide reasons why I can be confident. Which is why I just bought my wife a new Honda.

  13. Mike is very interest article. However in my view you cover the article from the “Role” point of view. What about the QA as a skill? A tester must have QA skills that will hep him to build better Test plans , Test cases etc,, So I find it a little dangerous as novice users may misread that they do not have to worry about QA as a skill.

    Michael replies: To me, quality assurance is not a skill. Skills do enable you to assure the quality of your own work to some degree, though.

  14. […] as a service to project stakeholders, which is echoed by Michael Bolton in his blog article titled Testers: Get Out of the Quality Assurance Business. In the article Bolton implores testers to recognise that they don’t solely “own” […]

  15. shrirang says:

    Very interesting blog. Please add me to your mailing list. Thanks a lot for sharing.


  16. Mr. Test says:

    I’ve been doing “QA” for well over 20 years. If I did not test the software, there would be little quality in many cases. I have to urge programmers to make changes to the software, not only because of bugs, but because of usability issues. If I did not URGE programmers who often have massive egos about the quality of their work, to fix or change the way that something does or does not work, most of it would not get done. So, I am assuring quality even though I am not the initial creator of the software. I’ve found that most programmers have little idea about what end users experience and are often detached from that reality. Without quality assurance testing and analysis, there is little quality. Quality Assistance? …maybe, but not just a tester.

    Michael replies: I notice that you call yourself “Mr. Test”. I wonder what “Mr. Develop” and “Mr. Manage” might say about the idea that they are detached from reality, or about the idea that you and you alone are “assuring” quality. I’m also curious about what you think being “just” a tester means.

  17. Jackie says:

    I moved from dev to qa about 4 years ago.
    My experience:
    QA people substitute the knowledge, the competence with the best practices etc.
    QA people think, they are working independently.
    They are not and they should not.
    QA people hate anyone who can actually do the job. I can read and write software. They sense it on the day one.
    I am well hated from the day one. Have been in this situation, at least 3 times.

    Michael replies: I’d like to offer this question, gently: Are you sure that differences in technical skill represent the only explanation for what sounds like a more complex social problem? The one and only explanation? “Hate” is a pretty strong word.

Leave a Reply