Some people say they “don’t have enough requirements to start testing,” or that the requirements are unclear or incomplete or contradictory or out of date. First, those people usually mean requirements documents; don’t confuse that with requirements, which may be explicit or tacit. There are plenty of sources of requirements-related information, and it’s a tester’s job to discover them and make inferences about them. Second, insufficient clarity about requirements is both a test result and a project risk, and awareness of them may be crucially important project information. (If the testers aren’t clear on the requirements, are the programmers? And if they’re not, how is it that they’re building the product?) Finally, if there is uncertainty about requirements, one great way around that problem is to start testing and reporting on what you find. “Insufficient requirements” may be a problem for testing—but it’s also precisely a problem that testing can help to solve.
Many people seem certain about what happened to cause the healthcare.gov fiasco. Stories are starting to trickle out, and eventually they’ll be an ocean of them. To anyone familiar with software development, especially in large organizations, these stories include familiar elements of character and plot. From those, it’s easy to extrapolate and fill in the details based on imagination and experience. We all know what happened.
Well, we don’t. In a project of that size, no one knows what happened. No one can know what happened. Imagine Rashomon scaled up to hundreds of people, each making his own observations and decisions along the way.
As time goes by, I anticipate some people saying that the project will represent a turning point in software development and project management. “Surely,” they will say, “after a project failure of this size and scope, people will finally learn.” Alas, I’m less optimistic. As the first three premises of rapid software testing describe it, software development is a human activity that is surrounded by 1) confusion, 2) complexity, 3) volatility, 4) urgency and… 5) ambition. Increasing ambition causes increases in the other four items too. In our societies, we could help to defend ourselves against future fiascos by restraining our ambitions, but I fear that people will put blindfolds on each other, pass around the keys, and scramble to get back into the driver’s seat of the school bus. How will they do this?
One form of the blindfold is to say “That not going to be a problem here because…”
failure is not an option.
we have our best people on it.
we can’t disappoint the client.
it doesn’t have to be perfect. (thanks, Joe Miller, @lilshieste)
we’ll fix it in production.
no user would ever do that.
the users will figure it out.
the users will never notice that.
THAT bug is in someone else’s code.
we don’t have to fix that; that’s a new feature request.
it’s working exactly as designed.
if there’s no test case for it, it’s not a bug.
the clients will come to their senses before the ship date.
we have thousands of automated tests that we run on every build
this time it will be different.
we have budget to fix that before we deploy.
at least the back end is working right.
if there are performance problems, we’ll just add another few servers.
we’ve done lots of projects just like this one.
foreign-language support is something we could cut.
that list there says that this is a level three threat, not a level one threat.
the support people can handle whatever problems come up.
this graph shows that the load will never get that high.
now is too soon; we’ll tell the clients about the problems after we’ve fixed them.
we’re thinking positively&mdashthat can-do spirit will see us through.
we still have plenty of time left to fix that.
the spec didn’t say anything about having to handle special characters. How are single quotes a big deal?
the client should have thought of that before.
seriously, that’s just a cosmetic problem.
it’s important not to complicate things.
everybody WILL put in some overtime and we WILL get this thing done.
well… at least the front end looks good, and people will be happy with that.
everyone here is committed to making sure this ships on time.
we’ll just shorten the test cycle.
if there’s a problem, the other/upstream/downstream team will let us know.
they can take care of that in training.
we’ve planned to make sure that nothing unexpected happens.
we’ve got this fantastic new framework that’ll make things go faster.
we’ll pull a bunch of people off other projects to work on this one.
I wonder whether these things were said, at one time or another, during the healthcare.gov project. I don’t know if they were. I don’t know what happened. I didn’t work on it. But I’ve heard these things on projects before, I know that I can listen for them, and I know that they’re a sign of trouble ahead. Are they being said on your project?
Too often testing is focused on getting the right answers, rather than asking worthwhile questions and helping to get them answered. There are at least two overarching questions that a tester must ask. While looking directly at the product, at its customers, at the project, at the business, and at the relationships between them, the tester’s first question is “Is there a problem here?” When the tester observes anything looks like it might be a problem, the tester’s next question is for the testing client and for the rest of the project community: “Are we okay with this?”
Imagine a software project. Imagine the things that you want to accomplish, the problems you might encounter, the workarounds you could apply, the accidents (both happy and sad) that might happen, the missteps you may take, the steps you can take to prevent them; all of the actions you can perform to manage the project. Now, make a detailed plan that takes all of your expectations into account.
The more detailed your plan, the more likely it will differ from reality in important respects. Unexpected things will happen, some positive, some negative, and many of them out of your control. You can’t predict future events reliably, but you can prepare to respond to them. Therefore: you might want to relax your effort on specific plans somewhat, and emphasize developing skills and resources that will help you to deal capably with surprises.
Some may say that the purpose of testing is to validate assumptions made by business analysts, designers, or developers. To me, that is at best a potential side effect of testing—but not the goal. If you want testing to reveal important problems in the product, do not focus on validating assumptions (to do so would be more like checking; testing may include some checking). To foster discovery, excellent testing—like excellent science—seeks to invalidate assumptions.
To put it another way, it’s easy to check to show that something can work. As testers, we must probe, challenge and test the assumptions that might cause people to believe mistakenly that it will work. Those assumptions are where the risks live.
People often suggest that “understanding the requirements” is an essential step before you can begin testing. This may be true for checking or formal testing—examining a product in a specific way, or to check specific facts. But understanding the requirements is not a necessary precursor to testing, which is learning about a product through experimentation (a larger activity which might include checking) and creating the conditions to make that activity possible. Indeed, you may need to test in order to develop an understanding of the requirements, which in turn triggers more and better testing, yielding even better understanding of the requirements—and so on.
More generally, when thinking about testing, think more about loops, and less about lines.
In response to my post from a couple of days ago, Gus kindly provides a comment, and I think a discussion of it is worth a blog post on its own.
Michael, I appreciate what you are trying to say but the simile doesn’t really work 100% for me, let me try to explain.
The simile has prompted you to think and to question, so in that sense, it works 100% for me. Triggering thought is, after all, is why people use similes. (See also Surfaces and Essences: Analogy as the Fuel and Fire of Thinking.)
I would apply lean principles and cut some waste from your interview process. I will fail the candidate as soon as she gives me the first wrong answer.
I have 5 questions and all have to be answered correctly to hire the person for a junior position (release 1).
Interview candidate A:
Ask question 1 OK
Ask question 2 FAIL
Send candidate A home
Second interview to candidate A:
Ask question 1 OK
Ask question 2 OK
Ask question 3 FAIL
Send candidate A home
Third interview to candidate A:
Ask question 1 OK
Ask question 2 OK
Ask question 3 OK
Ask question 4 OK
Ask question 5 OK
Hire candidate A
All right. You seem to have left out something important in your process here, which I would apply after each step—indeed, after each question and answer: make a conscious decision about your next step. To me, that requires continuous review of your list of questions for relevance, significance, sufficiency, and information value. Interviewing is an exploratory process. A skilled interviewer will respond, in the moment, to what the candidate has said. A skilled interviewer will think less in terms of “pass or fail”, and more in terms of “What am I learning about this candidate? What does the last answer suggest I should ask next? What other information, exclusive of the answer, might I apply to my decision-making process? What else should I be looking for?” When the candidate gets the answer wrong, the skilled interviewer will ask “Was it really wrong? Maybe there are multiple right answers to the same question. Maybe she didn’t understand the question because I asked in in an ambiguous way, and she gave a right answer to an alternative interpretation. Maybe her answer was a question for me, intended to clarify my question.”
I can’t emphasize this enough: like interviewing, testing is about far more than pass or fail. Testing is about exploration, discovery, investigation, and learning, with the goal of imparting what we’ve learned to people who matter. Testing is about trying to understand the product that we’ve got, with the goal of revealing information that helps our clients decide if it’s the product they want. Testing is usually (but not always) focused on finding evident problems, apparent problems and potential problems, not only in our products, but in our ideas about our products. Testing is also about finding problems in our testing, and every one of the “fail” moments above is a point at which I would want to consider a problem with the test. (The “pass” moments are like that too, if I really want to do a great job.)
At this point when candidate A will want to be promoted to a senior position (translate with next release of the software) I will prepare other 5 different questions probing against the new skills and responsibilities and as I have automated the first 5 questions I can send her a link to a web site where she will have to prove that she hasn’t forgotten about the first 5 before she can be even considered for the new position.
I’d do things slightly differently.
First I would ask “What would prompt me to ask the same questions again? Are those still the most important questions I could ask as she’s heading for her new role? What reason do I have to believe that she might have lost some capability she previously had? Are there other questions related to her old role—not necessarily to her new one—that I should ask that might be more revealing or more significant?” Note that there might be entirely legitimate reasons to believe that she might have backslid somehow—but at that point, I’d also want to ask “What are the conditions that would have allowed her to backslide without me noticing it—and what could I do to minimize those kinds of conditions?”
Then there would be another question I’d ask: “What if she has learned to answer a specific question properly, but is not adaptable to the general case? Should I be asking the same question in a different way, to see if she gives the same answer? Should I be asking a similar question that has a different answer, and see if she notices and handles the difference?”
Now: it might be costly to vary my questions, so I might simply shrug and decide just to go with the ones I’d asked before. But the point of evaluating my process is to ask, “How might I be fooling myself in the belief that I still know this person well?”
Assumes she answers correctly the 5 automated questions, at this point I will do the interview for the senior role.
Interview candidate A for senior role:
Ask question 6 OK
Ask question 7 FAIL
Send candidate A home
and so on.
I don’t see a problem with this process as long as I am allowed using everything I learn from the feedback with the candidate up to question “N” to adapt and change all the questions greater than “N”
Up until this point, you haven’t mentioned that, and your description of your process doesn’t make that at all clear. You’ve only mentioned the “pass” and “fail” parts of “everything I learn from the feedback”. Now, you might be taking that into account in your head, but notice how your description, your process model, doesn’t reflect that—so it becomes easy to misinterpret what you actually do. In addition, you’ve focused on adapting and changing all the questions greater than N—but I’d be interested in the possibility of adapting and changing all the questions less than or equal to N, too.
More importantly: qualifying someone for an important job is not about making sure that they can get the right answers on a canned test, just as testing a product is not about making sure that the functions produce expected result to some number of test cases. The specific answers might have some significance, but if I’m serious about hiring the right people for the job, I don’t want to make my decisions solely by putting them in front of a terminal, having them fill out an online form, and checking their answers. I want to evaluate them from a number of criteria: do they respond quickly, in a polite and friendly way? Do they work well with others? Are they appropriately discrete? Are they adaptable? Can they deal with heavy workloads? Do they learn quickly? In order to learn those things, I need to do more than ask pass-or-fail questions. I need to have unscripted, spontaneous, and free-flowing conversation with them; interview and interaction, and not just interrogation. You see?
Imagine that you are working for me, and that I want your help in qualifying and hiring new staff. I start by giving you my idea of how to interview a candidate for a job.
“Prepare a set of questions with specific, predetermined answers. Asking questions is expensive, so make sure to come up with as few questions as you can. Ask the candidate those questions, and only those questions. (It’s okay if someone else does the asking; anybody should be able to do that.) Check the candidate’s answers against what you expected. If he gives the answers that you expected, you can tell me that he’s good enough to hire. If he doesn’t, send him away. When he comes back, ask him the original questions. Keep asking those questions over and over, and when he starts giving the right answers consistently, then we’ll hire him.”
Now, a few questions for you.
1) Would you think me a capable manager? Why or why not?
2) What might you advise me about the assumptions and risks in my approach towards interviewing and qualifying a candidate?
3) What happens in your mind when you replace “interviewing a candidate” with “testing a product or service”, “questions” with “test cases”, “asking” with “testing”, “answers” with “results”, “hire” with “release”? Having done that, what problems do you see in the scenario above?
4) How do you do testing in your organization?
Having trouble understanding why James Bach and I think it’s important to distinguish between checking and testing? Consider this: a pile of leaves is not a tree. Leaves are important parts of trees, but there’s a lot more to a tree than just its leaves. The leaves owe their existence to being part of a larger system of the tree. Nature makes sure that leaves drop off and are replaced periodically, especially in environments that undergo significant changes from time to time. And if you asked someone to describe his tree, you’d probably—and properly—think him strange if he pointed to a pile of leaves and said, “this is my tree”.
Some testers have got into the habit of saying that “we break the software”. That leads to psychological and political problems: “The product was fine until the testers broke it.” The software is what it is, either broken or not, when we get it. So, try saying “We look for problems that could threaten the value of the software.” As James Bach says, the only things we break are illusions.