Saturday, November 24, 2007
Pairwise Testing
I wrote a paper on pairwise testing in 2004 (and earlier), and now, in 2007, it's time for an update. This post is an edited version of an appendix that I've recently added to that paper.
First, there appears to be great confusion in the world between orthogonal arrays and pairwise testing. People use the terms interchangeably, but there is a clear and significant difference. I'm fairly proud of the fact that I note that difference in my article albeit in some painful and not-very-interesting-to-most-people detail, and I think I get it right. If we're going to talk about these things we might as well get them right, so if I'm wrong, I urge you to disabuse me.
Second, I'm no longer convinced of the virtues of either orthogonal arrays or pairwise testing, at least not in the pat and surfacey way that I talked about them in the article.
An on-the-job experience provided a tremor. The project was already a year behind schedule (for an 18-month project), and in dire shape. Pretty much everyone knew it, so the goal became plausible deniability--or, less formally, ass-covering. One of the senior project manager looked over my carefully constructed pairwise table, and said "Hey... this looks really good--this will impress the auditors." He didn't have other questions, and he seemed not to be concerned about the state of the project. Impressing the auditors was all that mattered.
This gave me pause, because it suddenly felt as though my work was being used to help fool people. I wondered if I was fooling myself too. Until that moment, I had taken some pride in the work that I had been doing. Figuring out the variables to be tested had taken a long time, and preparing the tables had taken quite a while too. Was the value of my work matching the cost? I suddenly realized that I hadn't interacted with the product at all. When I finally got around to it, I discovered that the number, scope, and severity of problems in the product were such that the pairwise tests were, for that time, not only unnecessary but a serious waste of time. The product simply wasn't stable enough to use them. Perhaps much later, after those problems had been fixed, and after I had learned a lot more about the product, I could have done a far better job of creating pairwise tables--but by then I might have found that pairwise tables wouldn't have shed light on the things that mattered to the project. At that point I should have been operating and observing the product, rather than planning to test a product that desperately needed testing.
My test manager, for whom I have great respect, disappeared from that project due to differences with the project managers, and I was encouraged to disappear a week or two later. The project had been scheduled to deploy about six weeks after that. It didn't. It eventually got released four months later, was pulled from production, and then re-released about six months after that.
A year or so later, there was an earthquake in the form of this paper by James Bach and Pat Schroeder. If you want to understand a much more nuanced and coherent story about pairwise testing than the one that I prepared in 2004, look there.
Pairwise testing is very seductive. It provides us with a plausible story to tell about one form of test coverage, it's dressed up in fancy mathematical clothing, and it looks like it might reduce our workload. Does it provide the kind of coverage the kind that's most important to the project? Is reducing the number of tests we run a goal, or is it a form of goal displacement? Might we be fooling ourselves? Maybe, or maybe not, but I think we should ask. I should have.
First, there appears to be great confusion in the world between orthogonal arrays and pairwise testing. People use the terms interchangeably, but there is a clear and significant difference. I'm fairly proud of the fact that I note that difference in my article albeit in some painful and not-very-interesting-to-most-people detail, and I think I get it right. If we're going to talk about these things we might as well get them right, so if I'm wrong, I urge you to disabuse me.
Second, I'm no longer convinced of the virtues of either orthogonal arrays or pairwise testing, at least not in the pat and surfacey way that I talked about them in the article.
An on-the-job experience provided a tremor. The project was already a year behind schedule (for an 18-month project), and in dire shape. Pretty much everyone knew it, so the goal became plausible deniability--or, less formally, ass-covering. One of the senior project manager looked over my carefully constructed pairwise table, and said "Hey... this looks really good--this will impress the auditors." He didn't have other questions, and he seemed not to be concerned about the state of the project. Impressing the auditors was all that mattered.
This gave me pause, because it suddenly felt as though my work was being used to help fool people. I wondered if I was fooling myself too. Until that moment, I had taken some pride in the work that I had been doing. Figuring out the variables to be tested had taken a long time, and preparing the tables had taken quite a while too. Was the value of my work matching the cost? I suddenly realized that I hadn't interacted with the product at all. When I finally got around to it, I discovered that the number, scope, and severity of problems in the product were such that the pairwise tests were, for that time, not only unnecessary but a serious waste of time. The product simply wasn't stable enough to use them. Perhaps much later, after those problems had been fixed, and after I had learned a lot more about the product, I could have done a far better job of creating pairwise tables--but by then I might have found that pairwise tables wouldn't have shed light on the things that mattered to the project. At that point I should have been operating and observing the product, rather than planning to test a product that desperately needed testing.
My test manager, for whom I have great respect, disappeared from that project due to differences with the project managers, and I was encouraged to disappear a week or two later. The project had been scheduled to deploy about six weeks after that. It didn't. It eventually got released four months later, was pulled from production, and then re-released about six months after that.
A year or so later, there was an earthquake in the form of this paper by James Bach and Pat Schroeder. If you want to understand a much more nuanced and coherent story about pairwise testing than the one that I prepared in 2004, look there.
Pairwise testing is very seductive. It provides us with a plausible story to tell about one form of test coverage, it's dressed up in fancy mathematical clothing, and it looks like it might reduce our workload. Does it provide the kind of coverage the kind that's most important to the project? Is reducing the number of tests we run a goal, or is it a form of goal displacement? Might we be fooling ourselves? Maybe, or maybe not, but I think we should ask. I should have.
Technorati
