Blog Posts from December, 2018

I Represent the User! And We All Do

Saturday, December 15th, 2018

As a tester, I try to represent the interests of users. Saying the user, in the singular, feels like a trap to me. There are usually lots of users, and they tend to have diverse and sometimes competing interests. I’d like to represent and highlight the interests of users that might have been forgotten or overlooked.

There’s another trap, though. As Cem Kaner has pointed out, it’s worth remembering that practically everyone else on the team represents the interests of end users in some sense. “End users want this product in a timely way at a reasonable price, so let’s get things happening on schedule and on budget,” say the project managers. “End users like lots of features,” say the marketers. “End users want this specific feature right away,” say the sales people. “End users want this feature optimized like I’m making it now,” say the programmers. I’d be careful about claiming that I represent the end user—and especially insinuating that I’m the only one who does—when lots of other people can credibly make that claim.

Meanwhile, I aspire to test and find problems that threaten the value of the product for anyone who matters. That includes anyone who might have an interest in the success of the product, like managers and developers, of course. It also includes anyone whose interests might have been forgotten or neglected. Technical support people, customer service representatives, and documentors spring to mind as examples. There are others. Can you think of them? People who live in other countries or speak other languages, whether they’re end users or business partners or colleagues in remote offices, are often overlooked or ignored.

All of the people in our organization play a role in assuring quality. I can assure the quality of my own work, but not of the product overall. For that reason, it seems inappropriate to dub myself and my testing colleagues as “quality assurance”. The “quality assurance” moniker causes no end of confusion and angst. Alas, not much has changed over the last 35 years or so: no one, including the most of the testing community, seems willing to call testers what they are: testers.

That’s a title I believe we should wear proudly and humbly. Proudly, because we cheerfully and diligently investigate the product, learning deeply about it where most others merely prod it a little. Humbly, because we don’t create the product, design it, code it, or fix it if it has problems. Let’s honour those who do that, and not make the over-reaching claim that we assure the quality of their work.

Rapid Software Testing classes are coming to Brisbane, Sydney, Utrecht, Reykjavik, and Copenhagen—and elsewhere. Join us!

Who Needs the Testers?

Thursday, December 6th, 2018

This post is a lightly-edited transcript from a LinkedIn article, which was itself adapted and extended from a recent thread on Twitter.

Another day, another story that goes like this. A colleague tells me that he’s working with an organization by training developers in how to do testing. That sounds like a pretty good idea at first, although most developers are already pretty good at the kind of testing that developers need to do, especially when they collaborate and review each other’s work.

But this is not the kind of testing that the training is focused on. It turns out that developers are being trained in higher-level integration and system testing—the kind of deep testing that requires significant domain knowledge and a substantial shift from the developer mindset. Why are the developers getting this training? Because the organization got rid of the testers who were formerly doing this work, and is hoping that the developers will do it.

Why were the testers canned? The testers were unsuccessful at a mandate handed down from management: “We have to do more automated testing! Automate all the testing! Automate everything!” This means that the testers were being asked to do programming. And, unsurprisingly, most of the testers weren’t great at that.

Part of the problem was that, at best, the testers got shallow training in programming. Another part is that nobody really knew (or knows) what “automate everything” means. (The managers were probably thinking about the visible aspects of testing: pressing keys, moving mice, and comparing output to specified, desired results. That’s something we call checking. Testing can’t be automated, although checking can.) And another part is that programming turns out to be time-consuming and tricky. Who knew?!

The result was a bunch of automated checking that was shallow, brittle, didn’t find many bugs, and that didn’t keep pace with development. Testers had lots of questions for the programmers, and investigating the problems detected by automated checks took time and effort. Many of those problems turned out to be non-problems because of errors in the check code. Programmers and managers perceived, not unreasonably, that all this was interrupting the programmers’ work.

The short story motivating the decision to get rid of the testers was: testing was slowing down development. The solution: get rid of the testers. Get the programmers to do the “testing” (that is, programmed checking), since the programmers are already good at programming.

It’s usually a very good idea for programmers to do checking, especially at the unit level. Low-level checking provides the programmers with very fast feedback, alerting them to coding errors and problems that might otherwise get buried. It helps with the discipline of building the product cleanly and simply, such that it can be built and changed safely. But checking is not all there is to testing.

In the Rapid Software Testing namespace, testing is evaluating a product by learning about it through exploration and experimentation. So let’s expand the short story above: evaluating the product by learning about it through exploration and experimentation was slowing down development.

Now, let’s make that a little more concise: learning about what they were building was slowing down development. Another way to put that is that development was going too fast for people to learn about what they were building. As a former program manager, I find that ominous.

One big error, apparently, was in believing that programmed checking is all there is to testing, which led to this interpretation: getting non-programmers to write programs as a means of learning about the product was slowing down development.

It seems to me that while writing programs might be helpful for some learning purposes, it’s a highly imperfect and incomplete way to learn about a product. Why was this not recognized?

It’s often the case, alas, that testers don’t have the skills to compose, edit, narrate, and justify the story of their testing work. It’s also common for people—even testers—to believe that testing is all about developing and following scripted procedures and checking output, rather than investigating risk.

When testing is reduced to demonstrating, by rote, that the product can work, it’s easy to believe that testing is simply a programming task. Create programs to do really fast typing and really fast comparison of desired and actual results. Simple! No testers required!

Mind you, by that line of reasoning, all programmers do is type instructions from business people into a computer, right? So, here’s a modest proposal for those managers: teach business people to program, and then we won’t need programmers OR testers. Simple! No technologists required! Yet managers rarely suggest this.

Why not? Perhaps it’s because managers are aware the programming is hard. Yet those same managers often seem unaware that investigating complex systems for deeply hidden, subtle, rare, emergent problems is also hard. Testers must take responsibility for making that clear.

That requires two things: doing excellent testing, and describing excellent testing. That won’t happen when we reduce testing to test cases; when we talk of “manual” testing and “automated” testing; when we forget that we’re here to find problems that matter to people.

Doing excellent testing requires testers to understand the context of the product, where and how it will be used, and the project and processes being used to develop it. Excellent testing requires testers to have some degree of technical skill, but also to have the social skills, the communication skills, and the domain knowledge to test effectively.

Doing excellent testing requires testers to be rapid learners. It requires testers to develop rich, detailed models of risk and of the product so that they can cover it with testing. It requires good oracles—ways to recognize problems when we encounter them—far more than comparing the product to an specified, desired, “expected” result or to some line in a requirements document. It requires testers to be aware that a product is not simply units of code; the product and its users represent a system of people and things in meaningful relationship. Such systems have interactions and properties that are emergent, not obvious from simply checking the parts.

Describing excellent testing requires testers to tell the testing story, including what isn’t being tested—and what’s slowing testing down, or making testing harder, or reducing the value of the work. It requires testers to be able to articulate all of the dimensions of excellent testing: context, risk, coverage, oracles, systems thinking—and why testing a product is so much more than writing code to check its output.

================================================

Irrespective of your role, do you want to learn more about deeper testing? Rapid Software Testing 4.0 is coming to Australia (Sydney and Brisbane) in February 2019. Come to the Sydney Testers’ Meetup on December 13, 2018 for a preview: Rapid Software Testing in Agile Contexts. A public class of RST will be presented in Iceland in March. See all upcoming events, including those with James Bach, here.

If We Do Sanity Testing Before Release, Do We Have To Do Regression Testing?

Monday, December 3rd, 2018

Here is an edition of the reply I offered to a question that someone asked on Quora. Bear in mind that it might be a good idea to follow the links for context.

If we do sanity testing before release, do we have to do regression testing?

What if I told you Yes? What if I told you No?

Some questions shouldn’t be answered. That is: some questions shouldn’t be answered with a Yes or a No without addressing the context first. No one can give you a good answer to your question unless they know you, your product, and your project’s context.

Even after that problem is addressed, people outside your context may not know what you mean by regression testing or sanity testing, and you can’t be sure of knowing what they mean. That applies to other terms in the conversation, too; maybe they’ll talk about “manual testing”; I don’t believe there’s such a thing as “manual testing”. Maybe you agree with them now; maybe you’ll agree with me after you’ve read the linked post. Or maybe after you read this one.

Some people will suggest that regression testing and sanity testing are fundamentally different somehow; I’d contend that a sanity test may be a shallow form of regression testing, when the sanity test is what I’ve talked about here, and when regression testing is testing focused on regression- or change-related risk. In order to sort that out, you’d have to talk it through to make sure that you’re not in shallow agreement.

Nonetheless, some people will try to answer your question. To prepare you for some of those answers: it’s probably not very helpful to think about needing to do one kind of testing or the other. It’s probably more helpful to think in terms of what you and your organization want to do, and choosing what to do based on what (you believe) you know about your product, and what (you believe) you want to know about it, given the situation.

While this is not an exhaustive list, here are a few factors to consider:

  • Do you and the developers already have a lot of experience with your product?
  • Is your product being developed in a careful, disciplined way?
  • Are the developers making small, simple, incremental changes that they comprehend the risks well?
  • Is the product relatively well insulated from dependencies on platforms (hardware, operating systems, middleware, browers…) that vary a lot?
  • Are there already plenty of unit-level checks in place, such that the developers are likely to be aware of low-level coding errors early and easily?
  • Is it unusual to do a shallow pass through the features of the product and find bugs that are sticking out like a sore thumb?
  • Do you and the developers feel like they’re working at a sustainable pace?

If the answer to all of those questions is Yes, then maybe your regression testing can afford to be more focused on deep, rare, hidden, subtle, emergent problems, which are unlikely to be revealed by a sanity test. Or maybe your product (or a given feature, or a given change, or whatever you’re focused on) entails relatively low risk, so deep regression testing isn’t necessary and a sanity test will do. Or maybe your product is poorly-understood and has changed a lot, so both sanity checking and deep regression testing could be important to you.

I can offer things for you to think about, but I don’t think it’s appropriate for me or anyone else to answer your question for you. The good news is that if you study testing seriously, practice testing, and learn to test, you’ll be able to make this determination in collaboration with your team, and answer the question for yourself.

James Bach and I teach Rapid Software Testing to help people to become smart, powerful, helpful, independent testers, with agency over their work. If you want help with learning about Rapid Software Testing for yourself or for your team, find out how you can attend a public class, live or on line, or request one in-house.