Blog Posts from May, 2011

Framing Test Framing

Tuesday, May 17th, 2011

Rikard Edgren is a testing philosopher in Sweden. He independently develops his own ideas on testing (an example here), collaborates with his colleagues on The Test Eye, and critiques and builds on the work of other people in the community. He offered a comment to my recent post on test framing, and the comment deserves a post on its own. He said,

I don’t think explicit framing is needed very often in reality, often it is apparent that a test is needed. But I think it is an essential skill, that when being trained, will bring along other testing skills as well. And sometimes you definitely need it, e.g. when presenting a problem that might not be adressed otherwise.

Yes, and that’s an important point. We will rarely be asked to frame a test, and we will rarely do it explicitly, or in writing. However, I argue that we should always and at any time be able to frame a test; that is, to tie the elements of the testing story together with logical connections.

However, framing would fit my testing world better with a few changes:

1) add “description” (or similar) so framing is about how you describe and present a test or a problem.
This also aligns better with how “framing” is used in other fields, e.g. you can frame something in different ways, so it suits the audience, and the most important aspects.

Well, maybe. To me, the description dominates the framing; the framing, the logical structure, is a subset of and a basis for the description. When you frame something in different ways—that is, when you reframe it—you’re presenting a new perspective, for sure. But to me the key element is that the new description is based on different premises, and a different set of propositions and logical connections.

2) Use “a” instead of “the”. With current definition it is easy to think that there is one “true” frame, but in reality there can be many.

Doh! You’ve caught me in violation of the “a vs. the” heuristic. The “the” in this case implies that you’ve got a particular frame in mind, and that it’s the one you’re going use to deliver your interpretation. Nonetheless, you’re right: there may be multiple ways of framing a given test. Lucky for both of us that the “a vs. the” heuristic is a heuristic, not a rule.

3) Remove “logical”, so framing can take advantage of people’s ability to connect things in other ways than logic, so it isn’t necessary to give all the details for all framings. The logic part is one important bit, but there should be focus on other rationale/feelings as well.

There is every opportunity for other rationales and feelings to appear in the framing of a test. Framing doesn’t deny emotions; on the contrary. I think you might be conflating logical oracles and logical argumentation. There’s a difference.

There are certain aspects of a program that are informed by logic, such that a logical principle or mechanism points to a problem. That is, the oracle is based on a logical decision rule. To use the trivial example of an adding program,

IF the program is intended to add numbers, AND
IF we are using the rules of everyday mathematics, AND
IF we add two numbers
AND the addition of those two numbers produces a result that is not equal to their sum,
THEN the result is mathematically false AND
IF we see a result that’s mathematically false
THEN we assume that there’s a problem with the program.

In this case, logical and mathematical truth or falsehood act as our oracle—while we’re also framing the test by establishing logical connections between proposition.

Yet we can use logical connections to frame tests based other oracles as well.

IF the program is intended to provide us with the service of adding numbers conveniently AND
IF we perform a test in which we tab from one control to the next AND
IF the controls are laid out such that tabbing from the first input field value to the second involves a tab stop on another control AND
IF that wastes the customer’s time or distracts her needlessly
THEN we can justify the claim that there’s a bug here.

Note that the basis for the bug is qualitative, but the line of reasoning used to frame the test is logical, based on propositions and logical connectors.

This gives us something like: Test framing is a description of connections that structure, inform, and motivate a test.

I’d see the “description” you have here as implicit in the way James and I define framing: the set of logical connections that stucture and inform a test.

And here’s a made-up problem report example:

Main content of web site should be loaded first
When the web site is launched it starts to display sidebars and various frames before the main content of the page.
This is normally not a problem, but with slow connection, e.g. a mobile phone outside town, it gives a not so strong impression (see video available at …)

First impressions are important to our site, and my testing colleagues agree with me that a better display sequence would be:
company logo -> main content -> the rest

Here we have holes, subjective opinions, but still something that is easy to understand and make a decision from.
OK as framing or not?

I’m glad you’ve given me an example, because it clarifies the misunderstanding. In answer to your question, I would suggest No, not okay as framing. There are propositions—statements—here, but they’re not linked by logical connectives. Let me propose a counter-example:

IF we colleagues are in agreement on principles that govern the order in which elements are displayed AND
IF we believe that our company’s logo should be the most prominent element in the Web site AND
IF we believe that the main content is the most important element of a web page to the users, AND
THEREFORE we further believe that the logo and the main content should be displayed before the other elements
UNLESS the connection is so fast that users cannot perceive a difference; WHEREAS
IF, when the web site is launched, it starts to display sidebars and various frames before the main content of the page, AND
IF there the order of loading is perceptible AND
IF the that order is not only perceptible but prominent on a slow connection AND
IF slow connections are common on mobile phones outside of town AND
IF we presume that a significant proportion of our users will be using such connections AND
IF we believe that those users will suffer annoyance AND
IF that annoyance is also the first impression that the user receives
THEREFORE we believe that this delay represents a threat to our image AND
IF we view threats to our image as a bug
THEREFORE we have reason to characterize our observation of the loading order as a bug or at least an issue.

Now, this is a pretty elaborate construction, but it can happen pretty much at the speed of thought. The point is that we can trace from premise to conclusion through logically connected statements and propositions. I think if I asked you to do that with your example above, you could do it easily, using informal yet rigorous logic. Yet, as my earlier post suggested, testers aren’t used to doing that. I think the craft could use practice in the art of logical argumentation, even when it’s about qualitative issues that aren’t themselves governed by logic.

Exploratory Testing is All Around You

Monday, May 16th, 2011

I regularly converse with people who say they want to introduce exploratory testing in their organization. They say that up until now, they’ve only used a scripted approach.

I reply that exploratory testing is already going on all the time at your organization.  It’s just that no one notices, perhaps because they call it

  • “review”, or
  • “designing scripts”, or
  • “getting ready to test”, or
  • “investigating a bug”, or
  • “working around a problem in the script”, or
  • “retesting around the bug fix”, or
  • “going off the script, just for a moment”, or
  • “realizing the significance of what a programmer said in the hallway, and trying it out on the system”, or
  • “pausing for a second to look something up”, or
  • “test-driven development”, or
  • “Hey, watch this!”, or
  • “I’m learning how to use the product”, or
  • “I’m shaking out it a bit”, or
  • “Wait, let’s do this test first instead of that test”, or
  • “Hey, I wonder what would happen if…”, or
  • “Is that really the right phone number?”, or
  • “Bag it, let’s just play around for a while”, or
  • “How come what the script says and what the programmer says and what the spec says are all different from each other?”, or
  • “Geez, this feature is too broken to make further testing worthwhile; I’m going to go to talk to the programmer”, or
  • “I’m training that new tester in how to use this product”, or
  • “You know, we could automate that; let’s try to write a quickie Perl script right now”, or
  • “Sure, I can test that…just gimme a sec”, or
  • “Wow… that looks like it could be a problem; I think I’ll write a quick note about that to remind me to talk to my test lead”, or
  • “Jimmy, I’m confused… could you help me interpret what’s going on on this screen?”, or
  • “Why are we always using ‘tester’ as the login account? Let’s try ‘tester2’ today”, or
  • “Hey, I could cancel this dialog and bring it up again and cancel it again and bring it up again”, or
  • “Cool! The return value for each call in this library is the round-trip transaction time—and look at these four transactions that took thirty times longer than average!”, or
  • “Holy frijoles! It blew up! I wonder if I can make it blow up even worse!”, or
  • “Let’s install this and see how it works”, or
  • “Weird… that’s not what the Help file says”, or
  • “That could be a cool tool; I’m going to try it when I get home”, or
  • “I’m sitting with a new tester, helping her to learn the product”, or (and this is the big one)
  • “I’m preparing a test script.”

Now it’s possible that none of that stuff ever happens in your organization. Or maybe people aren’t paying attention or don’t know how to observe testing. Or both.

Then, just before I posted this blog entry, James Bach offered me two more sure-fire clues that people are doing exploratory testing: they say, “I am in no way doing exploratory testing”, or “we’re doing only highly rigorous formal testing”. In both cases, the emphatic nature of the claim guarantees that the claimant is not sufficiently observant about testing to realize that exploratory testing is happening all around them.

Update, October 12, 2015: In fact, in the Rapid Software Testing namespace, we now maintain it’s redundant to say “exploratory testing”, in the same way it’s redundant to say “carbon-based human” or “vegetarian potato”. It is formal scripting—not exploration‐that is the interloper on testing’s territory. We explain that here.

I’ve Been Framed!

Monday, May 16th, 2011

Last week in the Netherlands, I presented a workshop on test framing, a skill that James Bach and I like to talk about. Test framing is the set of logical connections that structure, inform, and motivate a test.

Testing is a process of composing, editing, narrating, and justifying a story with three parallel threads. There is a product element—the part of the story that describes product, how it works, how it fails, and how it might fail in ways that matter to our clients. That story is backed by the activity element—the part of the story that describes what we did and how we did it. Here we describe what we did, how we designed our tests, and how we configured, operated, observed and evaluated the product. The narrative of our activities gives warrant to the product story, making it credible and accountable.

In turn, the testing story needs its own warrant, so there’s a justification element to supply that. The activity element is about what we did, how we did, when we did it, who was involved. The justification element is about why. In the justification part of the story, we talk about the quality of our work, and the obstacles and issues that might threaten it. We explain why we’ve done what we’ve done, why we haven’t done other things yet, or why we won’t do certain things at all unless things change. We relate the activities to the testing mission and the product story, and explain why the testing work we’ve done is good enough, and why it’s reasonable to have postponed or written off the testing work we haven’t done.

Framing is the network of logical statements and connectives that link the three story elements and their details.  To frame a test means to create an unbroken line of logic from the mission—the information objective within some context—through the product story and our activities, down to the observed outcome of the test and our interpretation of it. (I’ve written about that here, too.) The logic consists of propositions—philospher-talk for “regular statements”—linked by logical connectives.  In some kinds of formal logic, only specific operators are permitted, like “if”, “and”, “not”, “or”, “if and only if”. Test framing uses a somewhat less formal and more permissive form of logic, but the intention is to provide a means of reasoning through a test idea, its execution, and its interpretation.  There are a couple of worked-out examples in this document, which was first presented at EuroSTAR in November, 2010, and which has been undergoing revision from time to time.

James and I have observed that many of the people in our classes and those who approach us for coaching really struggle with framing a test. Some of these testers are relatively inexperienced, of course. Yet we also observe that very experienced and skilled testers often struggle almost as hard. It seems that people aren’t used to the practice. It’s not that their testing is unmotivated, exactly; more that they’re not used to articulating and justifying their testing. That creates a feedback loop, which I’ll describe with a few gross generalizations:

a) Since testers don’t tell the story of their testing, clients don’t understand what testers are up to.

b) In order to evaluate the quality of testing, clients use a bunch of surrogate measures such as test case counts and defect escape ratios.

c) Testers get out of the habit of telling the testing story, and learn to emphasize the numbers (to please the client).

d) Go to (a).

So if you can work as a tester for years without framing your tests, and people ask you to frame tests rarely or not at all, is an inability to frame tests a problem? We think so.

  • Test framing is an important element in bug advocacy—the skill of presenting a problem report to a testing client in a way that accurately conveys the meaning, significance, and risk associated with our actions, observations and evaluations. Without the capacity to trace the logical reasoning behind test activity, our bug advocacy will be considerably weaker.
  • Decisions about quality are inherently subjective. As Jerry Weinberg points out, they’re political and emotional, yet the people who make them want to appear rational. Good test framing can help to provide some rationally based grounds for justifying quality decisions—and for providing reasonable alternatives.
  • Test framing provides a way for more experienced testers to describe important parts of testing to novices or trainees.
  • Usually people talk about traceability to requirements, when really they mean traceability to explicit requirements in requirements documents. Good test framing can provide traceability to implicit or as-yet-unstated requirements. That is, test framing can help to reveal and develop a richer and more representative requirements story.
  • Test framing logically ties the elements of testing story together—the product element, the testing element, and the justification element. Test framing is like the sinew that joins the bones of the testing story.
  • Practicing test framing can help us to frame other aspects of the project, including requirements, specifications, bugs, and management decisions. Is someone saying something that doesn’t make sense to you? Try framing it. Framing an observation, inference, or decision can help you to understand the elements that have gone into it, and the logic that links them—or the logical gaps that make it unreasonable, unjustified, or risky.
  • Finally, and perhaps most importantly, framing is an important aspect of self-management and exploratory approaches. A tester that can connect the mission of testing to the test result and everything in between can also connect the logical and emotional aspects of testing. Moreover, a tester that can frame his or her work can show a rational basis for it, which is key to developing trust. A tester that can’t frame testing work into a logical structure is more likely to be directed and controlled by managers and clients.  Whether true or not, managers will percieve that the testers don’t understand what they are doing.  Even if the testers do understand, they won’t get credit for it.

After the aforementioned workshop, I posted a couple of documents on the Web and notified the people who attended. Shortly thereafter, I got this reply that I can quote with Francien’s kind permisison:

From: Francien Ramakers [mailto:xxxxxxxx@xxx.com]
Sent: 12 May 2011 05:36
To: Michael Bolton
Subject: RE: Test Framing at TestNet

Hi Michael,

The first link does not work :(.

When I click on it I get a page with “You know, I could have sworn that page was here a minute ago.” from your blog.

Framing of this test:

– If the purpose of the link was to provide the readers of your email with a document and if the idea was or this to be possible simply by clicking on the link within the email and if clicking on it does not result in display of a document, and if the other links do work fine (indicating that that the issue is unlikely to be caused by another problem, such as being offline) then something is wrong with the first link.

– If the purpose of the first link in your email was to test how many people download the material after you mail them and if all people that try to download any of it click on the first link and if all people that click on the link have got the same result as me and if they all mail you when they do then this is not a bug but a feature.

In both cases, I would still really like to receive the document it refers to. Could you please send it to me?

Hmmm… and now I am not sure whether I framed the test or the bug…

Kind regards,

Francien.

Note that Francien has supplied not only a logical basis for the theory that she’s witnessing a bug—but also that she’s providing another, equally logical basis for an alternative interpretation of her observations. That’s important, because a tester is someone who knows that things can be different. Locking on to a single interpretation is a risky business for a tester, since a single proposition somewhere in the frame can flip the interpretation of the test from bug to feature, and vice-versa.

So I replied.

From: Michael Bolton [mailto:michael@developsense.com]
Sent: donderdag 12 mei 2011 12:56
To: Francien Ramakers
Subject: RE: Test Framing at TestNet

Hi, Francien…

Well done!

I would like to suggest that you framed the test and the bug. I can frame the reason for the bug,

if you like.

IF I give a presentation AND

IF a part of my mission in delivering the presentation is to supply people potentially with useful notes and information

THEN I should prepare an email for them which has the correct links to that information in it

AND IF someone complains that one of those links doesn’t work

THEN I suspect that I’ve made an error in the links

THEREFORE I test the link

AND IF I can reproduce the problem

THEN I suspect the link is incorrect

UNLESS I didn’t upload the file in the first place (which I now realize I didn’t)

THEREFORE I really shouldn’t do all this stuff too late at night.

🙂

—Michael B.

Francien replied,

Hi Michael,

Thanks! 🙂

I think it’s cool that you framed the answer!

Mailing about it made me realize framing can be applied to other things then just tests.

Cheers,

Francien.

Francien, of course, is right. Framing can be about many other things: not only testing, but also about bugs, requirements, specifications, and so forth.

For extra points: in my framing, can you identify the proposition in my framing that is not logically connected to the others?