DevelopsenseLogo

Measurement & Metrics

Calculating ROI

In preparation for our automation and management classes, James Bach and I are currently analysing ROI calculators (“Here’s what you can save over N years using our product!”) and we’re going deep on reverse-engineering one from a prominent tool vendor. Opaque formulas; undefined and unexplained terminology; unnamed and inexplicable constants; weird, inexplicable coefficients in the calculation; poor testability; hilarious bugs sitting right there on the surface; impossible conclusions due to … Read more

Talking About Coverage

A while back I wrote a post on coverage. In fact, I’ve written a few posts about coverage, but I’m talking about this one. A question came up recently on LinkedIn that helped me to realize I had left out something important. In that post, referring to coverage as “the proportion of the product that has been tested”, I said A software product is not a static, tangible thing; it’s a … Read more

Very Short Blog Posts (30): Checking and Measuring Quality

This is an expansion of some recent tweets. Do automated tests (in the RST namespace, checks) measure the quality of your product, as people sometimes suggest? First, the check is automated; the test is not. You are performing a test, and you use a check—or many checks—inside the test. The machinery may press the buttons and return a bit, but that’s not the test. For it to be a test, … Read more

Very Short Blog Posts (29): Defective Detection Effectiveness

Managers are responsible for hiring testers, for training them, and for removing any obstacles that make testing harder or slower. Managers are also responsible for hiring developers and designers, and providing appropriate training when it’s needed. If there are problems in development, managers are responsible for helping the developers to address them. Managers are also responsible for the scope of the product, the budget, the staffing, and the schedule. As … Read more

Facts and Figures in Software Engineering Research (Part 2)

On July 23, 2002, Capers Jones, Chief Scientist Emeritus of a company called Software Productivity Research gave a presentation called “SOFTWARE QUALITY IN 2002: A SURVEY OF THE STATE OF THE ART”. In this presentation, he shows data on a slide titled “U.S. Averages for Software Quality”. (Source: http://bit.ly/1rj19Ol, accessed September 5, 2014) It is not clear what “defect potentials” means. A slide preceding this one says defect potentials are … Read more

Facts and Figures in Software Engineering Research

On July 23, 2002, Capers Jones, Chief Scientist Emeritus of a company called Software Productivity Research, gave a presentation called “SOFTWARE QUALITY IN 2002: A SURVEY OF THE STATE OF THE ART”. In this presentation, he provided the sources for his data on the second slide: SPR clients from 1984 through 2002 • About 600 companies (150 clients in Fortune 500 set) • About 30 government/military groups • About 12,000 … Read more

Weighing the Evidence

I’m going to tell you a true story. Recently, in response to a few observations, I began to make a few changes in my diet and my habits. Perhaps you’ll be impressed. I cut down radically on my consumption of sugar. I cut down significantly on carbohydrates. (Very painful; I LOVE rice. I LOVE noodles.) I started drinking less alcohol. (See above.) I increased my intake of tea and water. … Read more

Construct Validity

A construct, in science, is (informally) a pattern or a means of categorizing something you’re talking about, especially when the thing you’re talking about is abstract. Constructs are really important in both qualitative and quantitative research, because they allow us to differentiate between “one of these” and “not one of these”, which is one of the first steps in measurement and analysis. If you want to describe something or count … Read more

Very Short Blog Posts (19): Testing By Percentages

Every now and then, in some forum or another, someone says something like “75% of the testing done on an Agile project is done by automation”. Whatever else might be wrong with that statement, it’s a very strange way to describe a complex, cognitive process of learning about a product through experimentation, and seeking to find problems that threaten the value of the product, the project, or the business. Perhaps … Read more

Counting the Wagons

A member of Linked In asks if “a test case can have multiple scenarios”. The question and the comments (now unreachable via the original link) reinforce, for me, just how unhelpful the notion of the “test case” is. Since I was a tiny kid, I’ve watched trains go by—waiting at level crossings, dashing to the window of my Grade Three classroom, or being dragged by my mother’s grandchildren to the … Read more