Blog: Ideas Around Bug Clusters

My colleague and friend Erik Petersen talks about bug clusters a lot. (I’ll get to discussing Black Swans in this space later, but I’m obliged to point out here that I disagree with Erik; I would argue that neither most bugs nor exploratory testing are black swans as he suggests.) I’ve been wanting to respond to the discussion for a while, and I think enough time has passed that I’m finally able to articulate my skepticism about the concept.

Note that, as James Bach says in his forthcoming book, skepticism is not the rejection of belief; it’s the rejection of certainty. My uncertainty with respect to bug clusters revolves around oversimplification of the notion. I think the idea as Erik explains it, while potentially powerful, begs important questions around modeling and factoring, and around cost versus value. My goal is to question the heuristic, and James and Pat Schroeder did with pairwise programming, not to dismiss it out of hand, but to arrive at a deeper understanding of how it might be useful and where it might be dangerous. In particular, I want to deprecate any notion that bug clusters might be a silver bullet, as Erik suggests—and I hope that when I’m done, he’ll appreciate the critique and respond to it. (That’s how things work in our community: Friends disagree. It’s important and okay to disagree. We try to work it out, and it’s nice if we do but okay if we don’t.)

First, modeling. Myers suggested that certain parts of the program tended to be far more problematic than others. The “parts” in question were, to him, areas of the program’s source code. To many people, the program is the source code. But consider Kaner‘s definition of a computer program, from his talk Software Testing as a Social Science. He says that a computer program is “A communication among several humans and computers who are distributed over space and time that contains instructions that can be run on a computer.” By this definition, the source code isn’t the program; it’s contained within the program. The program is the communication, and part of that communication contains source code.

What else is in the communication? What are the factors of a computer program, when we use Kaner’s definition as a point of departure?

A communication among separated agencies requires interfaces between them. There are several interfaces between the several people in the communication. There is a user interface for the end user, application programming interfaces for the programmers, testing interfaces for the testers. Typically there is documentation for each of these interfaces, too.

The communication between people is mediated by communication among computers and related systems. That mediated communication involves hardware (which may include computers, switching devices, processors that we may or may not think of as computers), protocols (which circumscribe the extents and limits of aspects of the communication), software (that may process the communication in some way, enhancing it or encapsulating it or compressing it or distorting it), and firmware (which itself may involve interfaces, data, programs, and protocols).

The product that enables the overall communication may be modeled in terms of structure (including computers, programs, modules, objects, classes, data objects within the classes, functions within the classes, and, yes, lines of code that enable those functions); in terms of functions that the product performs; in terms of data on which the program interacts; in terms of platforms upon which the program depends; in terms of operations that describe the way in which people use the program; and in terms of time and its interaction with the program.

This is just one set of ways of modeling or mapping the product. For the purposes of testing—questioning the product in order to evaluate it—we could conceive of many others. We could come up with a complexity map, or a map of Which Programmer Wrote Which Stuff, or Which Groups Specified Which Requirements. We could list our ideas about the program in terms of “things that involve localization and things that do not”; “things that involve currency and things that don’t”; “database interactions and not-database-interactions”; “dialog layout”; “workflows”; ad infinitum. We could come up with lists of risks, subdivided into various categories. All of these models and maps provide us with ideas about relationships, about what might constitute “near” or “far away“—and consequently, potentially useful but potentially overwhelming numbers of ideas about the points around which things could be said or imagined to cluster.

I have no problem believing in the notion of bug clusters. Certainly people tend to be good at some things, but not so good at others. Our conceptions of something might be very clear in some areas, more vague in others. It’s reasonable to see an error in one place and infer that may be errors close to it. But “close” using what map? Proximity is multivariate.

Second, cost versus value. When I find a bug, it’s wonderful to be able to use that information to find another one, so I often look for other bugs “clustered” “near” this one, based on whatever model I’ve been using or whatever factors I choose—consciously, subconsciously, or unconsciously—to apply to my understanding of what I perceive to be the bug. If I’m really on my game as a tester, I must also be ready to suspend certainty on whether my client will even agree that this is a bug. The factors that I perceive might be very important, and those models might be very powerful—or they might be total red herrings. So at the same time, I also have to consider the the opportunity cost of this activity, and the risk of not finding other bugs which might be more serious, or other information that might be more important in other, “far away” areas of my current model, or in different models altogether. It might be a better use of time to report the first problem that I find, noting my models and my suspicions, and let others (the designers, the business analysts, the programmers, the product managers) seek “related” problems using my models or their own.

The Bug Cluster Heuristic says to me “If you see a problem, suspect similar problems near it.” This heuristic can useful to me, but it depends on my models and my notion of “near”, and it depends on my considering carefully the cost and the value of further investigation, versus the value of obtaining broader or deeper coverage. A momentary pause for reflection on those questions could be time well spent.

Want to know more? Learn about upcoming Rapid Software Testing classes here.

3 responses to “Ideas Around Bug Clusters”

  1. JonathanKohl says:

    My experience: sometimes bugs appear to cluster, and other times they don’t.

    When I started programming more, I figured out why my programmer colleagues got so irritated when I would say: “bugs tend to cluster…” and they’d retort: “Sometimes! And sometimes not!”

    When I was in university, the programs I wrote for class in Pascal were very simple, and had no dependencies on 3rd party libraries (as far as I could tell). If there was an area with a bug, there was a good chance there were more bugs due to my misunderstanding of a requirement, or my skill limitations with the language. So in those programs, bugs appeared to cluster.

    Moving to C++, we started using 3rd party libraries with the STL and things changed. Present day, I dabble a bit in Ruby, Java and C#, and even the simplest programs can contain many, many 3rd party libraries. Often, what /appears/ to be a bug cluster from a black box 😉 testing perspective is due to a single configuration error, or me using code in a way that it wasn’t intended. The fix is one of those standard “one liners” that takes care of all of those bugs that link to a single cause. Is that a bug cluster? I don’t think so. All of those bugs were a variation on one problem.

    Here’s an example. I have a small testing program written in C#. I removed some error checking code (it comes free with the language) on an input field, which can result in all sorts of fun errors for testers to find. They usually find a lot of them, but it’s a simple (almost one line) code change to fix, and a common error to occur when a programmer is in a hurry, or refactors and forgets about something.

    In the areas where I control the logic completely, there are a couple of errors that are obvious to spot, but take more effort to find. Typically, a tester will find one bug there, and find several on the input field, and then tell me that the input field is where the bug cluster is.

    I’m no great shakes as a programmer, but I wonder if more modern toolsets have an effect on the so-called bug clustering heuristic. It’s easy to misinterpret what is going on from a tester’s perspective, and miss the logic errors in an area of code that the programmer wrote themselves completely, from scratch.

    Even then though, you can have the same effects – a simple coding error causing problems that can appear to be different at the UI, but when you view stack traces or error logs, they lead to the same problem. Other times, an incredibly unstable area of the software can be due to a simple configuration problem due to 3rd party integration.

    I don’t ignore the bug clustering heuristic, and when I do bug fix verification I try to analyze around the defect area and look for other problems that might not be addressed. Sometimes, a fix brings on unintended consequences adjacent to the bug fix, and you repeat the bug find/fix cycle for a while. Other times, a simple fix solves the problem. So it remains “heuristic” to me. Consider it, test it out, and go with the results. 🙂

    -Jonathan

  2. Michael says:

    Thanks for the detailed reply, Jonathan. You triggered another recognition for me: bugs might also appear to cluster around causes, and they might appear to cluster around effects. We might observe what looks like dozens of problems, all over the program, that are related to a single line of code; or we might see from the outside a single problem that reflects a pattern of problems in the code, such that instances of the pattern are strewn all over the place within the source.

  3. bharani says:

    I am Bharani Tester from India, First of all i let u know how i came across the word
    “Bug Cluster” . Since, i am non-vice tester I was viewing the testheuristic pdf of Ms. Elisabeth Hendrickson. Suddenly, i was observed the word :Bug Cluster; It was new to me and i was immediately used searchengine to know about this. But i am not statisfied with the definition. Then, Immediately I was asking test expert Mr. Shiva Mathivanan from India, about the ideas of bug cluster, He finally, said me to runthrough your article. Really, i got good idea about the bug cluster, I observed that some bugs may be clear or some may be vague. Nice sentence “”if you see a problem, Suspect similar problems near it” which i understood clearly about this Bug cluster & also started to follow ur Oracle and heuristics (Testing without Mind Map) are really good. Thanks for sharing this article……. 🙂

    Michael replies: You’re welcome.

Leave a Reply

Your email address will not be published. Required fields are marked *