Interview on Rapid Software Testing

March 27th, 2018

Not too long ago, I had a conversation with Joe Colantonio. Joe asks questions that prompt answers about the nature of testing, accountability, and telling the testing story.

Enjoy!

Very Short Blog Posts (33): Insufficient Information and Insufficient Time

March 19th, 2018

Here’s a question I get from testers quite a lot:

“What do I do when the developers give me something to test with insufficient information and time to test it?”

Here’s my quick answer: test it.

Here’s my practical answer: test it with whatever time and information you have available. (Testing is evaluating a product by learning about it through exploration and experimentation.) When your time is up, provide a report on what you have learned about the product, with particular focus on any problems you have found.

Identify the important risks and product factors of which you are aware, and which you have covered. (A product factor, or product element, is something that can be examined during a test, or that could influence the outcome of a test.) Identify important risks and product factors that you’re aware of and that you haven’t covered. Note the time and sources of information that you had available to you.

If part of the product or feature is obscure to you because you perceive that you have had insufficient information or time or testabilty to learn about it, include that in your report.

(I’ll provide a deep answer to the question eventually, too.)

Related posts:

How Is the Testing Going?
Testability
Testing Problems Are Test Results

Four (and More) Questions for Testers to Ask

March 11th, 2018

Testers investigate problems and risk. Other people manage the project, design the product, and write the code. As testers, we participate in that process, but in a special way and from a special perspective: it’s our primary job to anticipate, seek, and discover problems.

We testers don’t prevent problems; we don’t design or build or fix the product. We may help to prevent existing problems from going any farther, by discovering bugs, misunderstandings, issues, and risks and bringing them to light. With our help, the people who build and manage the project can address the problems we have revealed, and prevent worse problems down the line.

Over the last while, I’ve been working with clients that are “shifting left”, “going Agile”, “doing DevOps”, or “getting testers involved early”. Typically this takes the form of having a tester present for design discussions, planning meetings, grooming sessions, and the like.

This is usually a pretty good idea; if there is no one in the testing role, people tend not to think very deeply about testing—or about problems or risk. That’s why, even if you don’t have someone called “tester” on the team, it’s an awfully good idea to have someone in the testing role and the testing mindset. Here, I’ll call that person “tester”.

Alas, I’ve sometimes observed that, once invited to the meetings, testers are sometimes uncertain about what they’re doing there.

A while back, I proposed at least four things for testers to do in planning meetings: learning; advocating for testability; challenging what we’re hearing; and establishing our roles as testers. These activities help to enable sensemaking and critical thinking about the product and the project. How can testers do these things successfully? Here’s a set of targeted questions.

What are we building? Part of our role as testers is to come to a clear understanding of the system, product, feature, function, component, or service that we’re being asked to test. (I’ll say “product” from here on, but remember I could be referring to anything in the list.) We could be talking about the product itself or a representation of it. We could be looking at a diagram of it;reviewing a document or description of it; evaluating a workflow; playing with a prototype. Asking for any these can help if we don’t have them already. A beneficial side effect is helping to refine everyone’s understanding of the product—and how we’d achieve successful completion of the project or task.

So we might also ask: What will be there when we’ve built it? What are the bits and pieces? (Can we see a diagram?) What are the functions that the product offers; what should the product do? What gets input, processed, and output? (Do we have a data dictionary?) What does the product depend upon? What depends on the product? (Has someone prepared a list of dependencies? A list of what’s supported and what isn’t?)

For whom are we building it? If we’re building a product, we’re ultimately building it for people to use. Sometimes we make the mistake of over-focusing on a particular kind of user: the person who is immediately encountering the product, with eyes on screen and fingers on keyboard, mouse, or glass. Often, however, that person is an agent for someone else—for a bank teller’s application, think of the bank teller, but also think of the customer on the other side of the counter; the bank’s foreign exchange traders; the bank teller’s manager. Beyond using the product, there are other stakeholders: those who support it, connect to its APIs, test it, document it, profit from it, or defend it in court.

So we might also ask: Who else is affected by this product? Who do they work for, or with? What matters to them? (These questions are targeted towards operations value-related testability.) Who will support the product? Maintain it? Test it? Document it?

What could go wrong? The most important questions for testers to raise are questions about problems and risks. Developers, designers, business people, or others might discuss features or functions, but people who are focused on building a product are not always focused on how things could go badly. Switching from a builder’s mindset to a tester’s mindset is difficult for builders. For testers, it’s our job.

So we might also ask: What Bad Things could happen? What Good Things could fail to happen? Under what conditions might they happen or not happen? What might be missing? What might be there when it shouldn’t be there? And for whom are we not building this product—like hackers or thieves?

When something goes wrong, how would we know? Once again, this is a question about testability, and also a question about oracles. As James Bach has said, “software testing is the infinite art of comparing the invisible to the ambiguous to prevent the unthinkable from happening to the anonymous”. For any non-trivial program, there’s a huge test space to cover, and bugs and failures don’t always announce themselves. Part of our job is to think of the unthinkable and to help those invisible things to become visible so that we can find problems—ideally in the lab before we ship. Some problems might escape the lab (or our continuous deployment checks, if we’re doing that).

So we might also ask: How might we miss something going wrong? What do we need for intrinsic testability—at the very least, log files, scriptable interfaces, and code that has been reviewed, tested, and fixed as it’s being built. And what about subjective testability? Do we have the domain knowledge to recognize problems? What help might we need to obtain that? Do we have the specialist skills—in (for example) security, performance, or tooling—on the team? Do we need help there? If we’re working in a DevOps context,
doing live site testing or testing in production, how would we detect problems rapidly?

In sprint planning meetings, or design discussions, or feature grooming sessions, questions like these are important. Questions focused on problems don’t come naturally to many people, but asking such questions should be routine for testers. While everyone else is envisioning success, it’s our job to make sure that we’re anticipating failure. When everyone else is focused on how to build the product, it’s important for us to keep an eye on how the entire team can study and test it. When everyone else is creatively optimistic, it’s important for us to be pragmatically pessimistic.

None of the activities in planning and review replace testing of the product that is being built. But when we participate in raising problems and risks early on, we can help the team to prevent those problems—including problems that make testing harder or slower, allowing more bugs to survive undetected. Critical thinking now helps to enable faster and easier testing and development later.

Now a word from our sponsor: I help testers, developers, managers, and teams through consulting and training in Rapid Software Testing (RST). RST is a skill set and a mindset of testing focused on sharpening critical thinking, eliminating waste, and identifying problems that threaten the value of the product or the project, and the principles can be adapted to any development approach. If you need help with testing, please feel free to get in touch.

Signing Off

March 1st, 2018

Testers ask: “I’m often given a product to test, but not enough time to test it. How am I supposed to sign off on the release when I haven’t tested enough?”

My reply goes like this:

If you’re a tester, it seems profoundly odd to me that you are responsible for signing off on a release. The decision to release a product is a business decision, not a technical one. The decision is, of course, informed by technical considerations, so it’s entirely reasonable for you to provide a report on those considerations. It would be foolish for for someone in a business role to ignore your report. But it would be just as foolish for a someone in a business role to abdicate the business decision to technical people. We serve the business; we don’t run it, and technical people often aren’t privy to things in the business’ domain.

The idea that testers can either promote or prevent a release can be tested easily. Try refusing to sign off until you’re entirely happy with what you know about the product, and with the extent of what you know. You’ll get a fairly quick result.

Perhaps managers will go along with you. You’ll test the product until you are satisfied that you have covered everything important, that you have discovered all the significant problems, that those problems have been fixed, and that the discovery of any more problems that matter would be a big surprise to you. Your client (that is, the person to whom you report) will give you all the time and all the resources you need until the product meets your standards. Yes, that seems unlikely to me, too.

More likely, at some point you will be overruled. Management will decide that your concerns are not important enough to block the release. Then you will be able to say, “So, the decision to approve or prevent the release is not really my decision? Cool.” Then, perhaps silently, “I’m glad that’s your job after all. I’m happy not being a fake product owner.”

Without a product owner’s authority (and title, and salary), I’m not willing to sign—or even make—a go/no-go decision. That decision is not my responsibility, but the responsibility of people with the authority to make it. If they ask for my recommendation, I’ll provide a list of problems that I know about, and any reasons to believe that there might be more problems lurking. Then I will politely recommend that they weigh these against the needs and objectives of everyone else involved—development, operations, support, sales, marketing, finance, and so forth.

So what if someone asks you to sign something? I am willing to sign an account of what I know about the status of the product, and what I’ve done to learn about it. That account can be fairly concise, but in expanded form, it will look like this:

I typically start with “I’ve tested the product, and I’ve found these problems in it.” I then provide a list of a few problems that I believe to be most important to inform management’s decisions. For the rest, I’ll provide a summary, including patterns or classes of general problems, or pointers to problem reports. My special focus is on the problems; as the newspaper reporters will tell you, “if it bleeds it leads”. I’ll welcome requests for more information. If there’s any doubt about the matter, I emphasize that the decision to ship or not rests with the person responsible for making the release decision.

(Some people claim “the whole team decides when to release and when not to”. That’s true when the whole team agrees, or when disagreements are tractable. When they’re not, in every business situation I’ve been in, there is a single person who is ultimately responsible for the release decision.)

If I haven’t found any problems—which is rare—I won’t sign anything claiming that there are no problems in the product. I’m willing to assert that I’m not aware of any problems. I cannot responsibly say that there are no problems, or that I’m capable of finding all problems. To say that there are no problems is only an inference; to say that I’m not aware of any problems is a fact.

Whether I’ve found problems or not, I’m willing to sign a statement like this: “I have covered these conditions to this degree, and I have not covered these other conditions.” The conditions include specific product factors and quality criteria, like those found in the Heuristic Test Strategy model, or others that are specific to the project’s context. This gives warrant to my statement that there are problems (or that I’m not aware of them), and identifies why management should be appropriately trusting and appropriately skeptical about my evaluation. For an informed release decision, management needs to know about things I haven’t covered, and my perception of the risk associated with not covering them.

Happy news about the product might be worth mentioning, but it takes second place to reporting the problems and risks. I want to make sure that any concerns I have are prominent and not buried in my report.

I’m also willing to sign a statement saying “Here are some of the things that helped me, and here are some of the things that didn’t help; things that slowed my testing down, made it more difficult, reduced the breadth and depth of coverage I was able to obtain.” Whether I sign such a statement or not, I want to make sure I’ve been heard. I also want to offer some ideas that address the obstacles, and note that with management help, maybe we can reduce or remove some of them so that I can provide more timely, more valuable coverage of the product. When I can do that, I can find deeper, or more subtle, or more intermittent, and possibly more dangerous bugs.

Of course, I don’t run the project. There may be business considerations that prevent management from helping me to address the obstacles. If I’ve been heard, I’ll play the hand I’ve been dealt; I’ll do my best to address any problems I’ve got, using any resources I can bring to bear. It’s my job to make management aware of any risks associated with not dealing with the obstacles—on paper or in a quick email, if I’m worried about accountability. After that, decisions on how to manage the project belong with management.

In other words: I’m prepared to sign off on a three-part testing story. As a tester, I’m prepared to accept responsibility for my story about the quality of the product, but the product does not have to adhere to my quality standards. I’ll sign off on a report, but not on a decision. The business doesn’t need my permission to ship.

Finding the Happy Path

February 28th, 2018

In response to yesterday’s post on The Happy Path colleague and friend Albert Gareev raises an important issue:

Until we sufficiently learned about the users, the product, and the environment, we have no idea what usage pattern is a “happy path” and what would be the “edge cases”.

I agree with Albert. (See more of what he has to say here.) This points to a kind of paradox in testing and development: some say that we can’t test the product unless we know what the requirements are—yet we don’t know what many of the requirements are until we’ve tested! Testing helps to reduce ambiguity, uncertainty, and confusion about the product and about its requirements—and yet we don’t know how to test until we’ve tried to test!

Here’s how I might address Albert’s point:

To test a whole product or system means more than demonstrating that it can work, based on the most common or optimistic patterns of its use. We might start testing the whole system there, but if we wanted to develop a comprehensive understand of it, we wouldn’t stop at that.

On the other hand, the whole system consists of lots of sub-systems, elements, components, and interactions with other things. Each of those can be seen as a system in itself, and studying those systems contributes to our understanding of the larger system.

We build systems, and we build ideas on how to test them. At each step, considering only the most capable, attentive, well-trained users; preparing only the most common or favourable environments; imagining only the most optimistic scenarios; performing only happy-path testing on each part of the product as we build it; all of these present the risk of misunderstanding not only the product but also the happy paths and edge cases for the greater system. If we want to do excellent testing, all of these things—and our understanding of them—must not only be demonstrated, but must be tested as well. This means we must do more than creating a bunch of high-level, automated, confirmatory checks at the beginning of the sprint, and then declaring victory when they all “pass”.

Quality above depends on quality below; excellent testing above depends on excellent testing below. It’s testing all the way down—and all the way up, too.

Very Short Blog Posts (32): The Happy Path

February 26th, 2018

“Happy path testing” isn’t really testing at all. Following the “happy path” is a demonstration.

Here’s the role demonstration plays in testing: it’s nice to know that your product can achieve the happy path before you start to test it. To the degree a demonstration is a test, it’s a very shallow test.

If you’re building something new and non-trivial that matters to people, or that could harm people, there’s a risk that you might not entirely understand it or what it affects. To develop your understanding, you’ll probably want to test it; to learn about it; to investigate it; to interact with it directly; to probe it with tools; to stress it out. You’ll probably want to explore it and experiment with it; to evaluate it. That’s testing.

If you can’t even achieve the happy path, you’re not ready for testing.

Related posts:
Finding the Happy Path
Testing and Checking Refined
Why Checking is Not Enough
Acceptance Tests: Let’s Change the Title, Too
More of What Testers Find
Why We Do Scenario Testing

How is the testing going?

February 8th, 2018

Last week on Twitter, I posted this:

“The testing is going well.” Does this mean the product is in good shape, or that we’re obtaining good coverage, or finding lots of bugs? “The testing is going badly.” The product is in good shape? Testing is blocked? We’re noting lots of bugs erroneously?

People replied offering their interpretations. That wasn’t too surprising. Their interpretations differed; that wasn’t too surprising either. I was more surprised at how many people seemed to believe that there was a single basis on which we could say “the testing is going well” or “the testing is going badly”—along with the implicit assumption that people would automatically understand the answer.

To test is—among many other things—to construct, edit, narrate, and justify a story. Like any really good story, a testing story involves more than a single subject. A excellent, expert testing story has at least three significant elements, three plot lines that weave around each other like a braid. Miss one of those elements, and the chance of misinterpretation skyrockets. I’ve talked about this before, but it seems it’s time for a reminder.

In Rapid Software Testing, we emphasize the importance of a testing story with three strands, each of which is its own story.

We must tell a story about the product and its status. As we have tested, we have learned things about the product: what it is, what it does, how it works, how it doesn’t work, and how it might not work in ways that matter to our various clients. The overarching reason that most clients hire testers is to learn about problems that threaten the value of the product, so bugs—actual, manifest problems—tend to lead in the product story.

Risks—unobserved but potential problems—figure prominently in the product story too. From a social perspective, good news about the product is easier to deliver, and it does figure in a well-rounded report about the product’s state. But it’s the bad news—and the potential for more of it—that requires management attention.

We must tell a story about the testing. If we want management to trust us, our product story needs a warrant. Our product story becomes justified and is more trustworthy when we can describe how we configured, operated, observed, and evaluated the product. Part of this second strand of the testing story involves describing the ways in which we recognized problems; our oracles. Another part of this strand involves where we looked for problems; our coverage.

It’s important to talk about what we’ve covered with our testing. It may be far more important to talk about what we haven’t covered yet, or won’t cover at all unless something changes. Uncovered areas of the test space may conceal bugs and risks worse than any we’ve encountered so far.

Since we have limited time and resources, we must make choices about what to test. It’s our responsibility to make sure that our clients are aware of those choices, how we’re making them, and why we’re making them. We must highlight potentially important testing that hasn’t been done. When we do that, our clients can make informed decisions about the risks of leaving areas of the product untested—or provide the direction and resources to make sure that they do get tested.

We must tell a story about how good the testing is. If the second strand of the testing story supports the first, this third strand supports the second. Here it’s our job to describe why our testing is the most fabulous testing we could possibly do—or to the degree that it isn’t, why it isn’t, and what we need or recommend to make it better.

In particular, we must describe the issues that present obstacles to the fastest, least expensive, most powerful testing we can do. In the Rapid Software Testing namespace, a bug is a problem that threatens the value of the product; an issue is a problem that threatens the value of the testing. (Some people say “issue” for what we mean by “bug”, and “concern” for what we mean by “issue”. The labels don’t matter much, as long people recognize that there may be problems that get in the way of testing, and bring them to management’s attention.)

A key element in this third strand of the testing story is testability. Anything that makes testing harder, slower, or weaker gives bugs more time and more opportunity to survive undetected. Managers need to know about problems that impede testing, and must make management decisions to address them. As testers, we’re obliged to help managers make informed decisions.

On an unhappy day, some time in the future, when a manager asks “Why didn’t you find that bug?”, I want to be able to provide a reasonable response. For one thing, it’s not only that I didn’t notice the bug; no one on the team noticed the bug. For another, I want to be able to remind the manager that, during development, we all did our best and that we collaboratively decided where to direct our attention in testing and testability. Without talking about testing-related issues during development, those decisions will be poorly informed. And if we missed bugs, I want to make sure that we learn from whatever mistakes we’ve made. Allowing issues to remain hidden might be one of those mistakes.

In my experience, testers tend to recognize the importance of the first strand—reporting on the status of the product. It’s not often that I see testers who are good at the second strand—modeling and describing their coverage. Worse, I almost never encounter test reports in which testers describe what hasn’t been covered yet or will not be covered at all; important testing not done. As for the third strand, it seems to me that testers are pretty good at reporting problems that threaten the value of the testing work to each other. They’re not so good, alas, at reporting those problems to managers. Testers also aren’t necessarily so good at connecting problems with the testing to the risk that we’ll miss important problems in the product.

Managers: when you want a report from a tester and don’t want to be misled, ask about all three parts of the story. “How’s the product doing?” “How do you know? What have you covered, and what important testing hasn’t been done yet?” “Why should we be happy with the testing work? Why should we be concerned? What’s getting in the way of your doing the best testing you possibly could? How can we make the testing go faster, more easily, more comprehensively?”

Testers: when people ask “How is the testing going?”, they may be asking about any of the three strands in the testing story. When we don’t specify what we’re talking about, and reply with vague answers like “the testing is going well”, “the testing is going badly”, the person asking may apply the answer to the status of the product, the test coverage, or the quality of the testing work. The report that they hear may not be the report that we intended to deliver. To be safe, even when you answer briefly, make sure to provide a reply that touches on all three strands of the testing story.

Testing and Management Efficiency

February 5th, 2018

To a naïve manager who doesn’t understand testing very well, the visible manifestation of testing is a tester, sitting in front of a computer, banging on keys to operate a product, comparing output with a predicted result, and marking “pass” or “fail” on a spreadsheet.

Therefore, thinks the manager: one way to make testing dramatically more efficient and effective is to automate the testing. Substitute the human tester for a program on the same computer. Have the program bang on virtual keys to manipulate products and tools. Then algorithmically compare program output to a predicted result; and then mark “pass” or “fail” on an algorithmically generated report. (It doesn’t really help that many testers, managed by such managers, may believe in the same idea.)

To a naïve tester who doesn’t understand management very well, the visible manifestation of management is a manager, sitting in front of a computer, receiving messages from employees, banging on keys to scroll through documents and write a memo, checking the memo for spelling and grammar, attaching it to a message in Microsoft Outlook, and pressing “Send”.

Therefore, thinks the tester: one way to make management dramatically more efficient and effective is to automate the management. Install optical character recognition and voice recognition software onto the manager’s computer. Have a program feed problems, requests, and supporting documents into a spreadsheet full of business rules. Next, apply those rules, and feed the output into a memo-generating template that includes the words “approved” or “rejected”. Then have the program run the memo through a spelling and grammar checker, attach the memo to an Outlook message, and press Send. (In reality, even the most naive tester doesn’t believe that would lead to good management.)

Moral: some physical, visible, mechanical behaviours are included in each job, but is not the important part of either one. Both jobs require studying a situation, wrestling with uncertainty, comprehending business value (and threats to it), observing processes, analyzing risk, making decisions, performing actions, and interpreting outcomes.

Tools can help enormously with all of those activities, but for testing and management alike, fixation on visible behaviours misses the point. Obsession with mechanizing the visible things risks displacing the real work.

How To Get What You Want From Testing (for Managers): The Q & A

November 22nd, 2017

On November 21, 2017, I delivered a webinar as part of the regular Technobility Webinar Series presented by my friend and colleague Peter de Jager. The webinar was called “How To Get What You Want from Testing (for Managers)”, and you can find it here.

Alas, we had only a one-hour slot, and there were plenty of questions afterwards. Each of the questions I received is potentially worthy of a blog post on its own. Here, though, I’ll try to summarize. I’ll give brief replies, suggesting links to where I might have already provided some answers, and offering an IOU for future blog posts if there are further questions or comments.

If the CEO doesn’t appreciate testing and QA enough in order to develop application of high quality how QA Manager can get battle?

First, in my view, testers should think seriously about whether they’re in the quality assurance business at all. We don’t run the business, we don’t manage the product, and we can’t assure quality. It’s our job to shine light on the product we’ve got, so that people who do run the business can decide whether it’s the product they want.

Second, the CEO is also the CQAO—the Chief Quality Assurance Officer. If there’s a disconnect between what you and the CEO believe to be important, at least one of two things is probably true: the CEO knows or values something you don’t, or you know or value something the CEO doesn’t. The knowledge problem may be resolved through communication—primarily conversation. If that doesn’t solve the disconnect, it’s more likely a question of a different set of values. If you don’t share those, your options are to adopt the CEO’s values, or to accept them, or to find a CEO whose values you do share.

How to change managers to focus on the whole (e.g. lead time) instead of testing metrics (when vendor X is coding and vendor Y is testing?

As a tester, it’s not my job to change a manager, as such. I’m not the manager’s manager, and I don’t change people. I might try to steer the manager’s focus to some degree, which I can do by pointing to problems and to risks.

Is there a risk associated with focusing on testing metrics instead of on the whole? My answer, based on experience, is Yes; many managers have trouble observing software and software development. If you also believe that there’s risk associated with metrics fixation, have you made those risks clear to the manager? If your answer to that question is “I’ve tried, but I haven’t been successful,” get in touch with me and perhaps I can be of help.

One way I might be able to help right away is to recommend some reading: “Software Engineering Metrics: What Do They Measure and How Do We Know” (Kaner and Bond); by Measuring and Managing Performance in Organizations, by Robert Austin; and by Jerry Weinberg’s Quality Management series, especially Quality Software Management, Vol. 2: First Order Measurement (also available as two e-books, “How to Observe Software Systems” and “Responding to Significant Software Events“). Those works outline the problems, and suggest some solutions.

An important suggestion related to this is to offer, to agree upon, and to live up to a set of commitments, along with an understanding of what roles involve.

How do you ensure testers put under the microscope the important bugs and ignore the trivial stuff?

My answers here include: training, continuous feedback, and continuous learning; continuous development, review, and refinement of risk models; providing testers with knowledge of and access to stakeholders that matter. Good models of quality criteria, product elements, and test techniques (the Heuristic Test Strategy Model is an example) help a lot, too.

Suggestions for the scalability of regression testing, specifically when developers say “this touches everything.”

When the developer says “this touches everything”, one interpretation is “I’m not sure what this touches” (since “this touches everything” is, at best, rarely true). “I’m not sure what this touches,” in turn, really means “I don’t actually understand what I’m modifying”. That interpretation (like others revealed in the “Testing without Machinery” chapter in Perfect Software and Other Illusions About Testing) points to a Severity 0 product and project risk.

So this is not simply a question about the scalability of regression testing. This is a question about the scalability of architecture, design, development, and programming. These are issues for the whole team—including managers—to address. The management suggestion is to steer things towards getting the developer’s model of the code more clear, refactoring the code and the design until it’s comprehensible.

I’ve done some presentations on regression testing before, and some blog posts here and here (although, ten years later, we no longer talk about “manual tests” and “automated tests”; we do talk about testing and checking.)

How can we avoid a lot of back and forth, e.g. submitting issues piecemeal vs. submitting in a larger set.

One way would be to practice telling a three-part testing story and delivering the news. That said, I’m not sure I have enough context to help you out. Please feel free to leave a comment or otherwise get in touch.

How much of what you teach can expand to testing in other contexts?

Plenty of it, I’d say. In Rapid Software Testing, we take an expansive view of testing; evaluating a product by learning about it through exploration and experimentation, which includes to some degree: questioning, study, modeling, observation, inference, and plenty of other stuff. That’s applicable to lots of contexts. We also teach applied critical thinking—that is, thinking about thinking with the intention of avoiding being fooled. That’s extensible to lots of domains.

I’m curious about the other contexts you might have in mind, and why you ask. If I can be of assistance, please let me know.

I agree 100% with everything you say, and I feel daily meetings are not working, because everybody says what they did and what they will do and there is no talk about the state of the product.

That’s a significant test result—another example of the sort of thing that Jerry Weinberg is referring to in the “Testing without Machinery” chapter in Perfect Software and Other Illusions About Testing. I’m not sure what your role is. As a manager, I would mandate reports on the status of the product as part of the daily conversation. As a tester, I would provide that information to rest of the team, since that’s my job. What I found about the product, what I did, and what happened when I did are part of that three-part testing story.

Perhaps test cases are used as they are quantifiable and helps organise the testing of parts of the product.

Sure. Email messages are also quantifiable and they also help to organise the testing of parts of the product. Yet we don’t evaluate a business by the number of email messages it produces. Let’s agree to think all the way through this.

“Writing” is not “factory work” either, but as you say, many, many tools help check spelling, grammar, awkward phrases, etc.

Of course. As I said in a recent blog post, “A good editor uses the spelling checker, while carefully monitoring and systematically distrusting it.” We do not dispute the power and usefulness of tools. We do remain skeptical about what will happen when tools are put in the hands of those who don’t have the skill or wisdom to apply them appropriately.

I appreciate the “testing/checking” distinction, but wish you had applied the labels in the other way, given the rise of ‘the checklist” as a crucial tool in both piloting and surgery—application of the checklist is NOT algorithmic, when done as recommended.

Indeed. I’m confident, though, that smart people can keep track of the differences between “checking” or “a check” and a “checklist”, just as they can keep track of the differences between “testing” or “a test” and “testimony”.

I would include “breaking” the product (to see if it fails gracefully and to find its limits).

I prefer to say “stressing” the product, and I agree absolutely with the goal to discover its limits, whether it fails gracefully, and what happens when it doesn’t.

“Breaking” is a common metaphor, to be sure. It worries me to some degree because of the public relations issue: ““The software was fine until the testers broke it.” “We could ship our wonderful product on time if only the testers would stop breaking it.” “Normal customers wouldn’t have problems our wonderful product; it’s just that the testers break it.” “There are no systemic management or development problems that have been leading to problems in the product. Nuh-uh. No way. The testers broke it.

What about problems with the customer/user? Uninformed use or, even more problematical, unexpected use (especially if many users do the unexpected — using Lotus 123 as a word processor, for instance). I would argue you need to “model the human (including his or her context of machine, attempted use, level of understanding, et al.).

And I would not argue with you. I’d agree, especially with respect to the tendency of users to do surprising things.

I think stories about the product should also include items about what the product did right (or less bad than expected), especially items that were NOT the focus of the design (ie, how well does Lotus 123 work as a word processor?).

They could certainly include that to some degree. After all, the first items in our list of what to report in the product story are what it is and what it does, presumably successfully. Those things are important, and it’s worth reporting good news when there’s some available. My observation and experience suggests that reporting the good news is less urgent than noting what the product doesn’t do, or doesn’t do well, relative to what people want it to do, or hope that it does. It’s important, to me, that the good news doesn’t displace the important bad news. We’re not in the quality reassurance business.

So far, you appear to be missing much of the user side– what will they know, what context will they be in, what will they be doing with the product? (A friend was a hero for getting over-sized keyboards for football players to deal with all the typing problems that appeared in the software but were the consequence of huge fingers trying to hit normal-sized keytops.

That’s a great story.

I am really missing any mention of users, especially in a age of sensitivity to disabilities like lack of sight, hearing, manual dexterity, et al. It wasn”t until 11:45 or so that I heard about “use” and “abuse.” Again, all good stuff, but missing a huge part of wht I think is the source of risk.

I was mostly talking to project managers and their relationships with testers in this talk (and at that, I went over time). In other talks (and especially when talking to testers), I would underscore the importance of modeling the end user’s desires and actions.

Please in the blog, talk about risk from unexpected use (abuse and unusual uses).

The short blog post that I mentioned above talks about that a bit. Here is a more detailed one. Lately, I’ve been enjoying Gojko Adzic’s book Humans vs. Computers, and can recommend it.

The End of Manual Testing

November 8th, 2017

Testers: when we speak of “manual testing”, we help to damage the craft.

That’s a strong statement, but it comes from years of experience in observing people thinking and speaking carelessly about testing. Damage arises when some people who don’t specialize in testing (and even some who do) become confused by the idea that some testing is “manual” and some testing is “automated”. They don’t realize that software development and the testing within it are design studio work, not factory work. Those people are dazzled by the speed and reliability of automation in manufacturing. Very quickly, they begin to fixate on the idea that testing can be automated. Manual bad; automated good.

Soon thereafter, testers who have strong critical thinking skills and who are good at finding problems that matter have a hard time finding jobs. Testers with only modest programming skills and questionable analytical skills get hired instead, and spend months writing programs that get the machine to press its own buttons. The goal becomes making the automated checks run smoothly, rather than finding problems that matter to people. Difficulties in getting the machine to operate the product take time away from interaction with and observation of the product. As a result, we get products that may or may not be thoroughly checked, but that have problems that diminish or even destroy value.

(Don’t believe me? Here’s an account of testing from the LinkedIn engineering blog, titled “How We Make Our UI Tests Stable“. It’s wonderful that LinkedIn’s UI tests (checks, really) are stable. Has anyone inside LinkedIn noticed that LinkedIn’s user interface is a hot, confusing, frustrating, unusable mess? That LinkedIn Groups have lately become well-nigh impossible to find? That LinkedIn rudely pops up a distracting screen after each time you’ve accepted a new invitation to connect, interrupting your flow, rather than waiting until you’ve finished accepting or rejecting invitations? That these problems dramatically reduce the desire of people to engage with LinkedIn and see the ads on it?)

Listen: there is no “manual testing”; there is testing. There are no “manual testers”; there are testers. Checking—an element of testing, a tactic of testing—can be automated, just as spell checking can be automated. A good editor uses the spelling checker, while carefully monitoring and systematically distrusting it. We do not call spell checking “automated editing”, nor do we speak of “manual editors” and “automated editors”. Editors, just “editors”, use tools.

All doctors use tools. Some specialists use or design very sophisticated tools. No one refers to those who don’t as “manual doctors”. No one speaks of “manual researchers”, “manual newspaper reporters”, “manual designers”, “manual programmers”, “manual managers”. They all do brain- and human-centred work, and they all use tools.

Here are seven kinds of testers. The developer tests as part of coding the product, and the good ones build testability into the product, too. The technical tester builds tools for herself or for others, uses tools, and in general thinks of her testing in terms of code and technology. The adminstrative tester focuses on tasks, agreements, communication, and getting the job done. The analytical tester develops models, considers statistics, creates diagrams, uses math, and applies these approaches to guide her exploration of the product. The social tester enlists the aid of other people (including developers) and helps organize them to cover the product with testing. The empathic tester immerses himself in the world of the product and the way people use it. The user expert comes at testing from the outside, typically as a supporting tester aiding responsible testers.

Every tester interacts with the product by various means, perhaps directly and indirectly, maybe at high levels or low levels, possibly naturalistically or artificially. Some testers are, justifiably, very enthusiastic about using tools. Some testers who specialize in applying and developing specialized tools could afford to develop more critical thinking and analytical skill. Correspondingly, some testers who focus on analysis or user experience or domain knowledge seem to be intimidated by technology. It might help everyone if they could become more familiar and more comfortable with tools.

Nonetheless, referring to any of the testing skill sets, mindsets, and approaches as “manual” spectacularly misses the mark, and suggests that we’re confused about the key body part for testing: it’s the brain, rather than the hands. Yet testers commonly refer to “manual testing” without so much as a funny look from anyone. Would a truly professional community play along, or would it do something to stop that?

On top of all this, the “manual tester” trope leads to banal, trivial, clickbait articles about whether “manual testing” has a future. I can tell you: “manual testing” has no future. It doesn’t have a past or a present, either. That’s because there is no manual testing. There is testing.

Instead of focusing on the skills that excellent testing requires, those silly articles provide shallow advice like “learn to program” or “learn Selenium”. (I wonder: are these articles being written by manual writers or automated writers?) Learning to program is a good thing, generally. Learning Selenium might be a good thing too, in context. Thank you for the suggestions. Let’s move on. How about we study how to model and analyze risk? More focus on systems thinking? How about more talk about describing more kinds of coverage than code coverage? What about other clever uses for tools, besides for automated checks?

(Some might reply “Well, wait a second. I use the term ‘manual testing’ in my context, and everybody in my group knows what I mean. I don’t have a problem with saying ‘manual testing’.” If it’s not a problem for you, I’m glad. I’m not addressing you, or your context. Note, however, that your reply is equivalent to saying “it works on my machine.”)

Our most important goal as testers, typically, is to learn about problems that threaten value in our products, so that our clients can deal with those problems effectively. Neither our testing clients nor people who use software divide the problems they experience into “manual bugs” and “automated bugs”. So let’s recognize and admire technical testers, testing toolsmiths and the other specialists in our craft. Let us not dismiss them as “manual testers”. Let’s put an end to “manual testing”.