I feel that you might be mixing “test design technique” with “testing” – a common thing observed as in “Keyword driven testing”, “model based testing”, “action based testing” etc.

We need to be careful about claims like “xxx technique will systematically reduce the number of test cases while providing full test coverate”.

No technique, can reduce the number of test cases, it is tester’s model/assertion that some test cases are not required to be executed as they would provide the same information as the rest of other cases.

/Rikard

Set aside questions of whether it’s good for impressing the auditors. I think it’s good for at least three other purposes:

(1) Sometimes, there is a genuine (or perceived) risk of interaction among variables. An all-pairs tool gives you a simply-structured vehicle for designing tests that explore that risk. My primary concern about using all-pairs in this case (any form of combination testing that isn’t carefully thought out) is that people often go through the motions of entering the variables’ values into the input fields but they don’t then continue with the program to see how it uses those values, checking the decisions or calculations that might actually be affected by the combination.(Similarly for combination tests used for configuration testing, you have to use the configured system enough to find out if there is a problem.) I think all-pairs is at least as good a start for the design as any other combination heuristic. Note that I am suggesting that all-pairs is a vehicle for EXPLORING the risk. Once you have a better idea of what variables interact and how, other tests become more interesting.

(2) Sometimes, there is no reason to know in advance whether there are interactions, but you have to do some budget negotiations. The all-pairs criterion is a way of stating a scope of preliminary combination testing. If you find bugs with these tests, you continue with more tests that are better tailored to the risks you now understand. On the other hand, if you don’t find bugs with these tests, you stop at the agreed stopping point, having spent the expected cost.

(3) All-pairs provides a structure for considering what variables are worth combining and what values of those variables are worth combining. Not everyone needs that structure, and this structure doesn’t work for every test designer, but it’s a tool in the belt.

One way to think about this is to distinguish between three types of combination test design: (a) mechanical combination test design (all-pairs is an example, random combinations put together algorithmically with a random number generator is another) (b) scenario-based design, in which you consider how the product is likely to be used or installed and (c) risk-based design in which you consider specifically how these combinations might create a failure. Mechanically-oriented designs are not optimized for real-life emulation or risk. They are just mechanical.

A second distinction is between using combination testing in an exploratory way or a scripted-regression way. I’ve described using all-pairs in an exploratory way at the system level.

At the subsystem level, I’d be more likely to use all-pairs in a regression suite. Consider protocol testing — testing communications between two applications (probably two that are authored and maintained separately). The protocol specifies how they communicate. Over time, one program might be revised in accordance with an upgraded protocol (or just be revised incorrectly), causing bad interoperation with the other application. A regression suite of test messages, where the risk over time is very general: a well-formed message (which combines values of many variables) will be misunderstood or a badly-formed-under-this-protocol message will be accepted. As with automated unit tests, I would expect these types of tests to be cheap to create and run and useful over time as refactoring aids, more than as system test aids.

Certainly not wanting to clog up your blog ( hey that rhymes! ).

You may recall earlier this year that I fired off the e-flare under the heading “Dealing with Tester Regret”. I believe it was during this time you sent me the link to your presentation for “Emotions in Testing”. You were utilizing them as a heuristic, an oracle, etc. (Pardon the dismissive tone of that last sentence – it is a good work a worthy of reflection.)

Your circumstance sounds painfully familiar to the problem I was having. While my catalyst was internally motivated – as opposed to externally/managerially – something south of my emotions were engaged. It was my spirit or what some would call ‘conscious’.

You were used with others to help me navigate those murky waters which had until then involved much penitence and self-mutilation. I responded to this tumult with post #3934 on the Context Driven Software Testing Forum. I was hoping to return the favor.

As far as the “the” is concerned: consider “the” in this context as being used by an INFP. It is a soft “the” meant to soothe and support; not blame or diagnose.

Zach…

—Michael B.

It reminds me of my days as a session musician. I’d cut a track for the artist and never see them again. They could have edited all my parts, re-cut them with another drummer, or released the album with stickers proclaiming my awesome work (never happened). You can only be so responsible for how people decide to spin your stuff.

It appears that an epiphany occurred that caused you to re-analyze the whole shebang. It struck a chord with your internal value system literally. Remember your post on “Emotions in Testing”? Maybe you should add this experience to your presentation. It would be interesting (and beneficial) for us to know how one deals with circumstances where we feel culpable even if we’re not.