Monday, January 19, 2009
Meaningful Metrics
Over the years, I can remember working with exactly one organization that used my idea of an excellent approach to software engineering metrics. Their approach was based on several points:
- They treated metrics as first-order approximations, and recognized that they were fundamentally limited and fallible.
- They used the metrics for estimating, rather than for predicting. When their estimations didn't work out, they didn't use the discrepancy to punish people. They used it to try to understand what they hadn't understood about the task in the first place.
- They used inquiry metrics, rather than control metrics. used the metrics to prompt questions about their assumptions, rather than to provide answers or drive their work.
- They used a large number of observational modes to manage their business and to evaluate (and solve) their problems. Most importantly, the managers observed people and what they did, rather than watching printed reports. They used close personal supervision, collaboration, and conversation as their primary approach to learning about what was happening on the project. They watched the game, rather than the box scores.
- They didn't collect any metrics on things that weren't interesting and important to them.
- They didn't waste time collecting or managing the metrics.
- They had no interest in making the metrics look good. They were interested in optimizing the quality of the work, not in the appearance afforded by the metrics.
- They took a social sciences approach to measurement, as Cem Kaner describes the social sciences here (in particular on page 3 of the slides). Rather than assuming that metrics gave them complete and accurate answers, they assumed that the metrics were giving them partial answers that might be useful.
In summary, they viewed metrics in the same kind of way as excellent testers view testing: with skepticism (that is, not rejecting belief but rejecting certainty), with open-mindedness, and with awareness of the capacity to be fooled. Their metrics were (are) heuristics, which they used in combination with dozens of other heuristics to help in observing and managing their projects.
The software development and testing business seems to have a very poor understanding of measurement theory and metrics-related pitfalls, so conversations about metrics are often frustrating for me. People assume that I don't like measurement of any kind. Not true; the issue is that I don't like bogus measurement, and there's an overwhelming amount of it out there.
So, to move the conversation along, I'll suggest that anyone who wants to have a reasonable discussion with me on metrics should read and reflect deeply uponSoftware Engineering Metrics: What Do They Measure and How Do We Know (Kaner and Bond)
and then explain how their metrics don't run afoul of the problems very clearly identified in the paper. It's not a long paper. It's written by academics but, mirabile dictu, it's as clear and readable as a newspaper article (for example, it doesn't use pompous Latin expressions like mirabile dictu).
Here are some more important references:
- The Dark Side of Software Metrics (.pdf, Hoffman)
- Meaningful Metrics (.pdf, Allison)
- How to Lie With Statistics (book, Huff)
- Measuring and Managing Performance in Organizations (book, Austin)
- Quality Software Management, Vol. 2: First Order Metrics (book, Weinberg)
- Why Does Software Cost So Much? (book, deMarco)
Show me metrics that have been thoughtfully conceived, reliably obtained, carefully and critically reviewed, and that avoid the problems identified in these works, and I'll buy into the metrics. Otherwise I'll point out the risks, or recommend that they be trashed. As James Bach says, "Helping to mislead our clients is not a service that we offer."
While I agree with the main position here, I disagree about one perspective. There are no bogus measurements, only bogus interpretation of those measurements. We humans have a tendency to force meaning on everything, including measurements, where they do not exist. I do agree if a metric does not reveal interesting information it should be set aside. I also agree that we should not spend too much time gathering statistics. Measurements should be a way to point toward areas of interest and allow us to form ideas about our area of study.
My employer recently started a Quality program here. I will be sharing this article with many of my colleagues.
However, properly applied metrics (as you said, for inquiry) can be useful. The most important rule of using metrics is that the benefit gained be greater than the effort to obtain them. And that doesn't mean elevating the perceived importance of the metrics, as is so often the case, especially if they have pretty charts and animated powerpoint, that besides the presentation time, takes an hour away from each worker.
Useful metrics cannot intrude, or else there's also the chance of an uncertainty prinicpal that the observation influences the outcome, whether it be through lost productivity, morale, or gaming the system.
<< Home
Post a Comment