Over and over, I’ve said that testing is better on many levels when it’s investigative, rather than merely confirmatory. Today I was browsing the Web. A question that someone had asked me about SwissAir led me to Wikipedia; from there I jumped to the article on Air Canada. I’m also a fan of Stan Rogers’ music, and so I looked into the incident in which he died–a fire that started in mid-air, and led to the deaths of 23 people on the ground in Cincinnati. Wikipedia claimed that the fire was triggered by an electrical short-circuit in the lavatory. I hadn’t heard that explanation; my understanding was that the fire was caused by a burning cigarette in the lavatory garbage can. (My belief was strengthened by the fact that onboard smoking bans came into force not long after the investigation into the fire.) Further research seems to confirm that an electrical problem was indeed the trigger–the direct cause of the fire. Other factors made it worse: “the delay before the crew detected and responded to the fire, the ineffective use of the fire extinguishers, the toxicity of the seat covers, and the difficulties encountered in evacuation.”

But then I read something that I found even more interesting: “Accident investigators tend to take an expansive approach when determining the ’cause’ of an accident. Aware that regulations are influenced by accident reports, investigators often seek to effect the greatest possible change. “It’s better if you don’t find the exact cause because then only one thing will get fixed,” according to an NTSB investigator. Instead, for every serious accident the NTSB recommends a laundry list of changes in FAA regulations.” (That’s at http://content.cdlib.org/xtf/view?docId=ft8f59p27j&chunk.id=d0e2155&toc.depth=1&toc.id=d0e2100&brand=eschol ).

So the idea that our tests should pinpoint the problem is better seen as a heuristic–an approach that is often the right one, but that might be the wrong one for some purpose. It might be better not to pinpoint one problem if it means that we fix a few of them instead.

A correspondent on the Agile Testing mailing list asked recently

Shall automated acceptance tests use the GUI the app provides?

My reply sat in my drafts folder for a while, and I just found it. Too late for the conversation, really, so I’ll post it here.

My thought, as usual, is that automated acceptance tests should or should not use the GUI depending on

- the questions you want to ask and answer;
- the business domain that you’re working in, and therefore the kind and magnitude of the risk you want to address;
- the skill of your toolsmith (or the extent to which she can leverage the skills of others);
- the capabilities of your automation tools;
- the extent to which you wish to model real users and observe what they observe;
- the extent to which you want to deliver yourself automatically to a point where you can begin human-driven testing;
- the extent to which tests have already been created for lower levels;
- the extent to which you’re willing to value speed, precision, and volume over human observation and cognition;
- the extent to which you can vary the data that you’re using–either systematically, pseudo-randomly, or randomly–in a productive way;
- the extent to which you can create or use oracles that will allow the tool to help you recognize problems;
- the extent to which you want to act in a confirmatory, rather than an investigative mode; (and therefore)
- the extent to which you’re willing to be vulnerable to confirmation bias; and
- the extent to which you’re can defend against automation bias.

In my observation and experience, the cost of automating a test increases and the value decreases–generally–the closer to the GUI you get. Automated tests at the unit level tend to be easily to comprehend, simple to automate, subject to falsifiable assertions, and immediately responsive to developers. Automated tests at the GUI level tend to be inadequate for recognizing problems that humans will spot easily, complex to program, much less clearly decidable, and harder to troubleshoot and debug, thus less immediately responsive to developers. There are bound to be exceptions.

You might get different answers, making better use of your tools and the people who use and program them, if you think of test automation in James Bach’s definition as “any use of tools to support testing”.