Blog Posts from September, 2012

Premises of Rapid Software Testing, Part 3

Thursday, September 27th, 2012

Over the last two days, I’ve published the premises of the Rapid Software Testing classes and methodology, as developed by James Bach and me. The first set addresses the nature of Rapid Testing’s engagement with software development—an ambitious activity, performed by fallible humans for other fallible humans, under conditions of uncertainty and time pressure. The second set addresses the nature of testing as an investigative activity focused on understanding the product and discovering problems that threaten its value. Today I present the last three premises, which deal with our relationship to our clients and to quality.

6. We commit to performing credible, cost-effective testing, and we will inform our clients of anything that threatens that commitment. Rapid Testing seeks the fastest, least expensive testing that completely fulfills the mission of testing. We should not suggest million dollar testing when ten dollar testing will do the job.

It’s not enough that we test well; we must test well given the limitations of the project. Furthermore, when we are under constraints that may prevent us from doing a good job, testers must work with the client to resolve those problems. Whatever we do, we must be ready to justify and explain it.

7. We will not knowingly or negligently mislead our clients and colleagues. This ethical premise drives a lot of the structure of Rapid Software Testing. Testers are frequently the target of well-meaning but unreasonable or ignorant requests by their clients. We may be asked to suppress bad news, to create test documentation that we have no intention of using, or to produce invalid metrics to measure progress.

We must politely but firmly resist such requests unless, in our judgment, they serve the better interests of our clients. At minimum we must advise our clients of the impact of any task or mode of working that prevents us from testing, or creates a false impression of the testing.

8. Testers accept responsibility for the quality of their work, although they cannot control the quality of the product. Testing requires many interlocking skills. Testing is an engineering activity requiring considerable design work to conceive and perform. Like many other highly cognitive jobs, such as investigative reporting, piloting an airplane, or programming, it is difficult for anyone not actually doing the work to supervise it effectively. Therefore, testers must not abdicate responsibility for the quality of their own work.

By the same token, we cannot accept responsibility for the quality of the product itself, since it is not within our span of control. Only programmers and their management control that.

Sometimes testing is called “QA.” If so, we choose to think of it as quality assistance (an idea due to Cem Kaner) or quality awareness, rather than quality assurance.

Posted in Context, Rapid Software Testing | 8 Comments »

Premises of Rapid Software Testing, Part 2

Wednesday, September 26th, 2012

Yesterday I published the first three premises that underlie the Rapid Software Testing methodology developed and taught by James Bach and me. Today’s two are on the nature of “test” as an activity—a verb, rather than a noun—and the purpose of testing as we see it: understanding the product and imparting that understanding to our clients, with emphasis on problems that threaten the product’s value.

4. A test is an activity; it is a performance, not an artifact. Most testers will casually say that they “write tests” or that they “create test cases.” That’s fine, as far as it goes. That means they have conceived of ideas, data, procedures, and perhaps programs that automate some task or another; and they may have represented those ideas in writing or in program code.

Trouble occurs when any of those things is confused with the ideas they represent, and when the representations become confused with actually testing the product. This is a fallacy called reification, the error of treating abstractions as though they were things. Until some tester engages with the product, observes it and interprets those observations, no testing has occurred.

Even if you write a completely automatic checking process, the results of that process must be reviewed and interpreted by a responsible person.

5. Testing’s purpose is to discover the status of the product and any threats to its value, so that our clients can make informed decisions about it. There are people that have other purposes in mind when they use the word “test.” For some, testing may be a ritual of checking that basic functions appear to work. This is not our view.

We are on the hunt for important problems. We seek a comprehensive understanding of the product. We do this in support of the needs of our clients, whoever they are.

The level of testing necessary to serve our clients will vary. In some cases the testing will be more formal and simple, in other cases, informal and elaborate.

In all cases, testers are suppliers of vital information about the product to those who must make decisions about it. Testers light the way.

I’ll continue with the last three premises of Rapid Software Testing tomorrow.

Posted in Context, Rapid Software Testing, Uncategorized | 14 Comments »

Premises of Rapid Software Testing, Part 1

Tuesday, September 25th, 2012

In February of 2012, James Bach and I got together for a week of work one-on-one, face-to-face—something that happens all too rarely. We worked on a number of things, but the principal outcome was a statement of the premises on which Rapid Software Testing—our classes and our methodology—are based. In deference to Twitter-sized attention spans like mine, I’ll post the premises over the next few days. Here’s the preamble and the first three points:

These are the premises of the Rapid Software Testing methodology. Everything in the methodology derives in some way from this foundation. These premises derive from our experience, study, and discussions over a period of decades. They have been shaped by the influence of two thinkers above all: Cem Kaner and Jerry Weinberg, both of whom have worked as programmers, managers, social scientists, authors, teachers, and of course, testers.

(We do not claim that Cem or Jerry will always agree with James or me, or with each other. Sometimes they will disagree. We are not claiming their endorsement here, but instead we are gratefully acknowledging their positive impact on our thinking and on our work. We urge thinking testers everywhere to study the writings and ideas of these two men.)

1. Software projects and products are relationships between people, who are creatures both of emotion and rational thought. Yes, there are technical, physical, and logical elements as well, and those elements are very substantial. But software development is dominated by human aspects: politics, emotions, psychology, perception, and cognition.

A project manager may declare that any given technical problem is not a problem at all for the business. Users may demand features they will never use. Your fabulous work may be rejected because the programmer doesn’t like you. Sufficiently fast performance for a novice user may be unacceptable to an experienced user.

Quality is always value to some person who matters. Product quality is a relationship between a product and people, never an attribute that can be isolated from a human context.

2. Each project occurs under conditions of uncertainty and time pressure. Some degree of confusion, complexity, volatility, and urgency besets each project. The confusion may be crippling, the complexity overwhelming, the volatility shocking, and the urgency desperate.

There are simple reasons for this: novelty, ambition, and economy. Every software project is an attempt to produce something new, in order to solve a problem.

People in software development are eager to solve these problems. At the same time, they often try to do a whole lot more than they can comfortably do with the resources they have. This is not any kind of moral fault of humans. Rather, it’s a consequence of the so-called “Red Queen” effect from evolutionary theory (the name for which comes from Through the Looking Glass): you must run as fast as you can just to stay in the same place. If your organization doesn’t run with the risk, your competitors will—and eventually you will be working for them, or not working at all.

3. Despite our best hopes and intentions, some degree of inexperience, carelessness, and incompetence is normal. This premise is easy to verify. Start by taking an honest look at yourself. Do you have all of the knowledge and experience you need to work in an unfamiliar domain, or with an unfamiliar product? Have you ever made a spelling mistake that you didn’t catch? Which testing textbooks have you read carefully? How many academic papers have you pored over? Are you up to speed on set theory, graph theory, and combinatorics? Are you fluent in at least one programming language? Could you sit down right now and use a de Bruijn sequence to optimize your test data? Would you know when to avoid using it? Are you thoroughly familiar with all the technologies being used in the product you are testing? Probably not—and that’s okay.

It is the nature of innovative software development work to stretch the limits of even the most competent people. Other testing and development methodologies seem to assume that everyone can and will do the right thing at the right time. We find that incredible. Any methodology that ignores human fallibility is a fantasy.

By saying that human fallibility is normal, we’re not trying to defend it or apologize for it, but we are pointing out that we must expect to encounter it in ourselves and in others, to deal with it compassionately, and make the most of our opportunities to learn our craft and build our skills.

I’ll continue with more Rapid Software Testing premises tomorrow.

Posted in Context, Rapid Software Testing, Uncategorized | 17 Comments »

The Customer Wants To Speak With You. Why Cover Your Ears?

Tuesday, September 4th, 2012

Speaking of oracles—the ways in which we recognize problems…

I’m on the mailing list for a company whose product I purchased a while ago. The other day, I received a mailing signed by the product marketing manager for that company. The topic of the mailing is a potential use for the product, but the product doesn’t support that purpose very well at all. In fact, I’ve often wanted to use the product for that purpose, but I’ve been stymied every time. The most recent upgrade of the product doesn’t support my task any better than the last version did. I’ve been so frustrated that I’ve been thinking of writing an email to the company, so this mailing ought to provide me with an easy opportunity to reply. Alas, the reply-to address is “no_reply@[that_company].com”. There are some links in the email, but they’re to blog posts and to the company’s main Web page. From experience, I know that I’ll have to go through extra effort to make contact with an actual person. In this mail signed by the product marketing manager, there’s no “contact me” or even “contact us” link or address by which I could send an email anywhere in the text. The company is jammed on Transmit, with no time for Receive. Actually, there is one email link: the one that allows you to unsubscribe from the mailing list. Now I’m tempted.

A lesson here for marketers: your customers are a primary source of information about your product and your company. Make it easy for them to send you that information, as Mark Federman suggests here. One of Mark’s important points is that if you make it easy for them to talk to you, your customers might complain—but they’ll also tell you about who’s doing a great job for you, and about what your company is doing right. Never make it hard for customers to send feedback to you, unless perhaps you want Seth Godin to make fun of you on his blog.

A lesson here for testers: try subscribing to your company’s mailing lists. If you do, there’s a good chance you’ll find out all kinds of stuff:

  • Bugs or issues in your company’s mailing, like the ones I’ve identified here.
  • If those bugs or issues gets fixed, fantastic information about bugs in the product.
  • Rich sources of claims about your product–claims that can be tested, or used as oracles.
  • Information on novel uses that customers have found for your product.
  • Information about how marketers think your product might be used–and what they’ve missed.

I’ve mentioned The Fundamental Regulator Paradox in these pages before (here and here). The basic idea is that stable systems have a kind of immune system that rejects incursions or even information from the outside, in order to prevent the system from being destabilized. But, as the paradox goes, things from the outside are the primary source of information on the surrounding environment and on things that might destabilize the system. Thus the better the system defends itself against information from the outside, the less it knows about what’s going on around it, the less adaptable it becomes, and the more vulnerable it is to damage when things change.

The product manager for this company is probably a very busy person, and it’s a big enough company that he might not want his main inbox flooded with messages from customers. That’s fine: set up another inbox, and have people from the marketing and support teams monitor it. And respond. At very least, provide help for the customers who are stuck, and thank others for the suggestions they will inevitably provide. Instead of thinking of customers as simply purchasers or users, think of them as important sources of ideas for your design and development teams; partners.

Also, for testers and marketers alike: treat social media as another source of outside information. Pradeep Soundararajan and his team at Moolya regularly astonish prospective clients by reporting on serious problems in their products long before Moolya has been hired. “How do you know about that?” say the execs. “WE didn’t know about that!” Pradeep’s answer: Moolya’s people spend some time reading Twitter. And Facebook. And the company’s own user forums. In such cases, some of the company’s internal employees probably know about those problems, too, but the managers and executives don’t keep the channels open, so messages from below go nowhere. How many times have you tried to give feedback to a customer service rep or field technician, only to be told, “You should contact head office yourself; they never listen to us.”

Marketing, management, and software development aren’t only about advertisements and analysis and algorithms; they’re about relationships between people. Make it easy for people to talk to you, listen, and you’ll hear some important things.

Posted in Uncategorized | 4 Comments »