Blog Posts from September, 2013

Interview and Interrogation

Friday, September 27th, 2013

In response to my post from a couple of days ago, Gus kindly provides a comment, and I think a discussion of it is worth a blog post on its own.

Michael, I appreciate what you are trying to say but the simile doesn’t really work 100% for me, let me try to explain.

The simile has prompted you to think and to question, so in that sense, it works 100% for me. Triggering thought is, after all, is why people use similes. (See also Surfaces and Essences: Analogy as the Fuel and Fire of Thinking.)

I would apply lean principles and cut some waste from your interview process. I will fail the candidate as soon as she gives me the first wrong answer.

I have 5 questions and all have to be answered correctly to hire the person for a junior position (release 1).

Interview candidate A:
Ask question 1 OK
Ask question 2 FAIL
Send candidate A home

Second interview to candidate A:
Ask question 1 OK
Ask question 2 OK
Ask question 3 FAIL
Send candidate A home

Third interview to candidate A:
Ask question 1 OK
Ask question 2 OK
Ask question 3 OK
Ask question 4 OK
Ask question 5 OK

Hire candidate A

All right. You seem to have left out something important in your process here, which I would apply after each step—indeed, after each question and answer: make a conscious decision about your next step. To me, that requires continuous review of your list of questions for relevance, significance, sufficiency, and information value. Interviewing is an exploratory process. A skilled interviewer will respond, in the moment, to what the candidate has said. A skilled interviewer will think less in terms of “pass or fail”, and more in terms of “What am I learning about this candidate? What does the last answer suggest I should ask next? What other information, exclusive of the answer, might I apply to my decision-making process? What else should I be looking for?” When the candidate gets the answer wrong, the skilled interviewer will ask “Was it really wrong? Maybe there are multiple right answers to the same question. Maybe she didn’t understand the question because I asked in in an ambiguous way, and she gave a right answer to an alternative interpretation. Maybe her answer was a question for me, intended to clarify my question.”

I can’t emphasize this enough: like interviewing, testing is about far more than pass or fail. Testing is about exploration, discovery, investigation, and learning, with the goal of imparting what we’ve learned to people who matter. Testing is about trying to understand the product that we’ve got, with the goal of revealing information that helps our clients decide if it’s the product they want. Testing is usually (but not always) focused on finding evident problems, apparent problems and potential problems, not only in our products, but in our ideas about our products. Testing is also about finding problems in our testing, and every one of the “fail” moments above is a point at which I would want to consider a problem with the test. (The “pass” moments are like that too, if I really want to do a great job.)

At this point when candidate A will want to be promoted to a senior position (translate with next release of the software) I will prepare other 5 different questions probing against the new skills and responsibilities and as I have automated the first 5 questions I can send her a link to a web site where she will have to prove that she hasn’t forgotten about the first 5 before she can be even considered for the new position.

I’d do things slightly differently.

First I would ask “What would prompt me to ask the same questions again? Are those still the most important questions I could ask as she’s heading for her new role? What reason do I have to believe that she might have lost some capability she previously had? Are there other questions related to her old role—not necessarily to her new one—that I should ask that might be more revealing or more significant?” Note that there might be entirely legitimate reasons to believe that she might have backslid somehow—but at that point, I’d also want to ask “What are the conditions that would have allowed her to backslide without me noticing it—and what could I do to minimize those kinds of conditions?”

Then there would be another question I’d ask: “What if she has learned to answer a specific question properly, but is not adaptable to the general case? Should I be asking the same question in a different way, to see if she gives the same answer? Should I be asking a similar question that has a different answer, and see if she notices and handles the difference?”

Now: it might be costly to vary my questions, so I might simply shrug and decide just to go with the ones I’d asked before. But the point of evaluating my process is to ask, “How might I be fooling myself in the belief that I still know this person well?”

Assumes she answers correctly the 5 automated questions, at this point I will do the interview for the senior role.

Interview candidate A for senior role:
Ask question 6 OK
Ask question 7 FAIL
Send candidate A home

and so on.

I don’t see a problem with this process as long as I am allowed using everything I learn from the feedback with the candidate up to question “N” to adapt and change all the questions greater than “N”

Up until this point, you haven’t mentioned that, and your description of your process doesn’t make that at all clear. You’ve only mentioned the “pass” and “fail” parts of “everything I learn from the feedback”. Now, you might be taking that into account in your head, but notice how your description, your process model, doesn’t reflect that—so it becomes easy to misinterpret what you actually do. In addition, you’ve focused on adapting and changing all the questions greater than N—but I’d be interested in the possibility of adapting and changing all the questions less than or equal to N, too.

More importantly: qualifying someone for an important job is not about making sure that they can get the right answers on a canned test, just as testing a product is not about making sure that the functions produce expected results for some number of test cases. The specific answers might have some significance, but if I’m serious about hiring the right people for the job, I don’t want to make my decisions solely by putting them in front of a terminal, having them fill out an online form, and checking their answers. I want to evaluate them from a number of criteria: do they respond quickly, in a polite and friendly way? Do they work well with others? Are they appropriately discrete? Are they adaptable? Can they deal with heavy workloads? Do they learn quickly? In order to learn those things, I need to do more than ask pass-or-fail questions. I need to have unscripted, spontaneous, and free-flowing conversation with them; interview and interaction, and not just interrogation. You see?

Interview Questions

Wednesday, September 25th, 2013

Imagine that you are working for me, and that I want your help in qualifying and hiring new staff. I start by giving you my idea of how to interview a candidate for a job.

“Prepare a set of questions with specific, predetermined answers. Asking questions is expensive, so make sure to come up with as few questions as you can. Ask the candidate those questions, and only those questions. (It’s okay if someone else does the asking; anybody should be able to do that.) Check the candidate’s answers against what you expected. If he gives the answers that you expected, you can tell me that he’s good enough to hire. If he doesn’t, send him away. When he comes back, ask him the original questions. Keep asking those questions over and over, and when he starts giving the right answers consistently, then we’ll hire him.”

Now, a few questions for you.

1) Would you think me a capable manager? Why or why not?
2) What might you advise me about the assumptions and risks in my approach towards interviewing and qualifying a candidate?
3) What happens in your mind when you replace “interviewing a candidate” with “testing a product or service”, “questions” with “test cases”, “asking” with “testing”, “answers” with “results”, “hire” with “release”? Having done that, what problems do you see in the scenario above?
4) How do you do testing in your organization?

Very Short Blog Posts (4): Leaves and Trees

Tuesday, September 24th, 2013

Having trouble understanding why James Bach and I think it’s important to distinguish between checking and testing? Consider this: a pile of leaves is not a tree. Leaves are important parts of trees, but there’s a lot more to a tree than just its leaves. The leaves owe their existence to being part of a larger system of the tree. Nature makes sure that leaves drop off and are replaced periodically, especially in environments that undergo significant changes from time to time. And if you asked someone to describe his tree, you’d probably—and properly—think him strange if he pointed to a pile of leaves and said, “this is my tree”.

Very Short Blog Posts (3): The Software Is Already Broken

Monday, September 16th, 2013

Some testers have got into the habit of saying that “we break the software”. That leads to psychological and political problems: “The product was fine until the testers broke it.” The software is what it is, either broken or not, when we get it. So, try saying “We look for problems that could threaten the value of the software.” As James Bach says, the only things we break are illusions.

Very Short Blog Posts (2): Confidence

Thursday, September 12th, 2013

It is not the job of testing to build confidence in the product. Confidence is a relationship between the product and some stakeholder. It is much more the job of testing to identify problems in the product—and in people’s perceptions of the product—that are based on or that would lead to unwarranted confidence.

Very Short Blog Posts (1): “We Don’t Have Time for Testing”

Sunday, September 8th, 2013

When someone says “We don’t have time for testing”, try translating that into “We don’t have time to think critically about the product, to experiment with it, and to learn about ways in which it might fail.” Then ask if people feel okay about that.