Over and over, I’ve said that testing is better on many levels when it’s investigative, rather than merely confirmatory. Today I was browsing the Web. A question that someone had asked me about SwissAir led me to Wikipedia; from there I jumped to the article on Air Canada. I’m also a fan of Stan Rogers’ music, and so I looked into the incident in which he died—a fire that started in mid-air, and led to the deaths of 23 people on the ground in Cincinnati. Wikipedia claimed that the fire was triggered by an electrical short-circuit in the lavatory. I hadn’t heard that explanation; my understanding had been that the fire was caused by a burning cigarette in the lavatory garbage can. (My belief had been strengthened by the fact that onboard smoking bans came into force not long after the investigation into the fire.) Further research seems to confirm that an electrical problem was indeed the trigger—the direct cause of the fire. Other factors made it worse: “the delay before the crew detected and responded to the fire, the ineffective use of the fire extinguishers, the toxicity of the seat covers, and the difficulties encountered in evacuation.”
But then I read something that I found even more interesting: “Accident investigators tend to take an expansive approach when determining the ’cause’ of an accident. Aware that regulations are influenced by accident reports, investigators often seek to effect the greatest possible change. ‘It’s better if you don’t find the exact cause because then only one thing will get fixed,’ according to an NTSB investigator. Instead, for every serious accident the NTSB recommends a laundry list of changes in FAA regulations.” (That’s at http://content.cdlib.org/xtf/view?docId=ft8f59p27j&chunk.id=d0e2155&toc.depth=1&toc.id=d0e2100&brand=eschol ).
So the idea that our tests should pinpoint the problem is better seen as a heuristic—an approach that is often the right one, but that might be the wrong one for some purpose. It might be better not to pinpoint only one problem if it means that we fix a few of them instead.