Thursday, January 25, 2007
Test Project Estimation, The Rapid Way
(I'm suspicious of Blogger, suddenly--I sure don't remember changing the status of this post from "published" to "draft", but one way or another, it happened.. Sorry if you're seeing it again.)
Erik Petersen (with whom I've shared one of the more memorable meals in my life) says, in the Software Testing Yahoo! group,
I know when I train testers, nearly all of them complain about not enough time to test, or things being hard to test. The lack of time is typically being forced into a completely unrealistic time frame to test against.
I used to have that problem. I don't have that problem any more, because I've reframed it (thanks to Cem Kaner, Jerry Weinberg, and particularly James Bach for helping me to get this). It's not my lack of time, because the time I've got is a given. Here's a little skit for you.
I'm sitting in my office. Someone, a Pointy-headed Boss (Ph.B.), barges in and says, "We're releasing on March 15th. How long to you need to test this product?"
Me: (pause) Um... Let's see. June 22.
Ph.B.: WHAT?! That can't be!
Me: You had some other date in mind?
Ph.B.: Well, something a little earlier than that.
Me: Okay... How about February 19?
Ph.B.: WHAT!?! We want to release it on March 15th!
Me: Oh. So... how about I test until about, say, March 14.
Ph.B.: Well that's... better...
Me: (pause) ...but I won't tell you that it's ready to ship.
Ph.B.: How do you know already that it won't be ready to ship?
Me: I don't know that. That's not what I mean; I'm sorry, I didn't make myself clear. I mean that I won't tell you whether it's ready to ship.
Ph.B.: What? You won't? Why not?!
Me: It's not my decision whether to ship. The product has to be good enough for you, not for me. I don't have the business knowledge you have. I don't know if the stock price depends on quarterly results, and I definitely don't know if there are bonuses tied to this release. There are bunches of factors that determine the business decision. I can't tell you about most of those. But I can tell you things that I think are important about the product. In particular, I can tell you about important problems.
Ph.B.: But when will you know when I can ship?
Me: Only you can know that. I can't make your decision, but I can give you information that helps you to make it. Every day, I'll learn more and more about the product and our understanding of it, and I'll pass that on to you. I'll focus on finding important problems quickly. If you want to know something specific about the product, I'll run tests to find it out, and I'll tell you about what I find. Any time you want to ask me to report my status, I'll do that. If at any time you decide to change the ship date, I'll abide by that; you can release before or after or on the 15th--whenever you decide that you don't have any more important questions about the product, and that you're happy with the answers you’ve got.
Ph.B.: So when will you have run all the tests?
Me: All the tests that I can think of? I can always think of more questions that I could ask and answer about the product--and I'll let you know what those are. At some point, you'll decide that you don't need those questions answered--the questions or answers aren't interesting enough to prevent you from releasing the product. So I'll keep testing until I'm done.
Ph.B.: When will you be done?
Me: You're my client; I'll test as long as you want me to. I'll be done when you ask me to stop testing--or when you ship.
Rapid testers are a service to the project, not an obstacle. We keep providing service until the client is satisfied. That means, for me, that there's never "not enough time to test"; any amount of time is enough for me. The question isn't whether the tester has enough time; the question is whether the client has enough information--and the client gets to decide that.
Erik Petersen (with whom I've shared one of the more memorable meals in my life) says, in the Software Testing Yahoo! group,
I know when I train testers, nearly all of them complain about not enough time to test, or things being hard to test. The lack of time is typically being forced into a completely unrealistic time frame to test against.
I used to have that problem. I don't have that problem any more, because I've reframed it (thanks to Cem Kaner, Jerry Weinberg, and particularly James Bach for helping me to get this). It's not my lack of time, because the time I've got is a given. Here's a little skit for you.
I'm sitting in my office. Someone, a Pointy-headed Boss (Ph.B.), barges in and says, "We're releasing on March 15th. How long to you need to test this product?"
Me: (pause) Um... Let's see. June 22.
Ph.B.: WHAT?! That can't be!
Me: You had some other date in mind?
Ph.B.: Well, something a little earlier than that.
Me: Okay... How about February 19?
Ph.B.: WHAT!?! We want to release it on March 15th!
Me: Oh. So... how about I test until about, say, March 14.
Ph.B.: Well that's... better...
Me: (pause) ...but I won't tell you that it's ready to ship.
Ph.B.: How do you know already that it won't be ready to ship?
Me: I don't know that. That's not what I mean; I'm sorry, I didn't make myself clear. I mean that I won't tell you whether it's ready to ship.
Ph.B.: What? You won't? Why not?!
Me: It's not my decision whether to ship. The product has to be good enough for you, not for me. I don't have the business knowledge you have. I don't know if the stock price depends on quarterly results, and I definitely don't know if there are bonuses tied to this release. There are bunches of factors that determine the business decision. I can't tell you about most of those. But I can tell you things that I think are important about the product. In particular, I can tell you about important problems.
Ph.B.: But when will you know when I can ship?
Me: Only you can know that. I can't make your decision, but I can give you information that helps you to make it. Every day, I'll learn more and more about the product and our understanding of it, and I'll pass that on to you. I'll focus on finding important problems quickly. If you want to know something specific about the product, I'll run tests to find it out, and I'll tell you about what I find. Any time you want to ask me to report my status, I'll do that. If at any time you decide to change the ship date, I'll abide by that; you can release before or after or on the 15th--whenever you decide that you don't have any more important questions about the product, and that you're happy with the answers you’ve got.
Ph.B.: So when will you have run all the tests?
Me: All the tests that I can think of? I can always think of more questions that I could ask and answer about the product--and I'll let you know what those are. At some point, you'll decide that you don't need those questions answered--the questions or answers aren't interesting enough to prevent you from releasing the product. So I'll keep testing until I'm done.
Ph.B.: When will you be done?
Me: You're my client; I'll test as long as you want me to. I'll be done when you ask me to stop testing--or when you ship.
Rapid testers are a service to the project, not an obstacle. We keep providing service until the client is satisfied. That means, for me, that there's never "not enough time to test"; any amount of time is enough for me. The question isn't whether the tester has enough time; the question is whether the client has enough information--and the client gets to decide that.
# posted by Michael @ 1:07 AM 
Comments:
<< Home
Bingo! Thanks for the great example discussion! That's exactly what I believe us testers should be doing: providing information to those that make the decisions.
The trick is to get that through to those buried in a process that makes "QA" (really testers) the gatekeeper, the bottleneck, and the scapegoat.
The trick is to get that through to those buried in a process that makes "QA" (really testers) the gatekeeper, the bottleneck, and the scapegoat.
Thanks for presenting a new and novel approach to handling such open ended questions like "When Testing will be completed". In my opinion, the challenge driving this approach and emphasizing it to management - lies in effectively articulating thoughts like "I will test as long as you wish" and "Will not make the mistake of deciding when to ship"
With due respect all *managers* out there - most of the time, managers want a number - such a number that suits them (this you have very well
articulated) and expect the tester to give that *number*. Few managers conveniently shift the responsibility of making decisions about the ship date to the testers.
There are two extreme variations of this "shifting" behavior
In one case - No release until QA (aka testers in some communities) says it is ready.
Other one is - Even when the tester says there are issues that *matter* they ship the product (while blaming testing for delaying the ship date
in some way).
Our industry really needs a paradigm shift in pushing Testing as service (open ended quest for problems) and constantly staying away from assuming the role of the Project manager
Post a Comment
With due respect all *managers* out there - most of the time, managers want a number - such a number that suits them (this you have very well
articulated) and expect the tester to give that *number*. Few managers conveniently shift the responsibility of making decisions about the ship date to the testers.
There are two extreme variations of this "shifting" behavior
In one case - No release until QA (aka testers in some communities) says it is ready.
Other one is - Even when the tester says there are issues that *matter* they ship the product (while blaming testing for delaying the ship date
in some way).
Our industry really needs a paradigm shift in pushing Testing as service (open ended quest for problems) and constantly staying away from assuming the role of the Project manager
<< Home
