DevelopsenseLogo

How do we know when we’ve done enough exploratory testing?

This was written in reply to a suggestion on comp.software.testing that we can’t decide when we’ve done enough ad hoc or exploratory testing.

The original poster asked:

Can you predict how much time such testing stage will take? What are grounding your estimates on? When do you have a chance to adjust your estimates? Is not it too late? And what are you doing in case it is? How do you estimates depend on the level of experience of your testers? Let’s imagine you have a newbie on your team. What estimates you will be using for that person?

My reply:

Can you predict how long daily exercise will take?

I can predict how long an exploratory testing stage will take: ten minutes. Or ninety minutes. Or many sessions that add up to eight hours. Or many sessions that add up to three weeks. I’m grounding my estimates on the time that I allocate to the task–and what’s more, unlike other forms of estimation, I can make the estimate totally predictable: I’ll stop when the alloted time is up.

As soon as I find some interesting information (the product appears to be in great shape, the product is in terrible shape, there is an area about which we know far too little, etc., etc.), I can adjust my estimate. It’s not too late if the product hasn’t shipped. If I have a newbie on the team, I have choices: I can monitor and mentor the newbie closely; I can give the newbie more highly structured processes to follow; or I can simply recognize that in the same amount of time, I might not get the same quantity or quality of information that I can get from a more experienced person.

How do we demonstrate that we’ve done enough exploratory testing? We could do this by telling a story about test coverage. If we’ve modeled the product in lots of different ways, and then tested in accordance with those models, we can say that we’ve done enough testing and stop. If we’re satisfied that we have addressed the list of compelling questions that we set out to answer (along with the other questions that we realized are important along the way), then we can say that we’ve done enough testing and stop. If the product is so horribly broken that it has to be send in for major rework, then we can say that we’ve done enough testing and stop. If management decides that it has sufficient confidence to ship, and ships, then we can say that we’ve done enough testing and stop. If management decides that it must ship the product, even though confidence in its quality is less than what we’d like it to be, we can say that we’ve done enough testing and stop. If we’re testing on behalf of someone who is trying to decide whether the software is acceptable, and they say it is, we can say that we’ve done enough testing and stop. If we’re testing on behalf of someone who is trying to decide whether the software is acceptable, and they say it isn’t, until we get another version, we can say that we’ve done enough testing and stop.

Do you notice that this is exactly the same set of stopping heuristics that we use for other forms of testing?

Leave a Comment