But if you’re interested in having a conversation, what’s your concept of a heuristic? To me, a heuristic is a fallible method for solving a problem or making a decision. Heuristics (as pointed out in Billy Vaughan Koen’s foundational book, Discussion of the Method) are at the core of engineering work; indeed, Koen makes a compelling case that all knowledge and all methods are heuristic.

What’s your concept of “scientific”? Wikipedia suggests that “To be termed scientific, a method of inquiry must be based on gathering empirical and measurable evidence subject to specific principles of reasoning.” This is not antithetical to the use of heuristics. Indeed, science itself is entirely based in heuristics. The principle behind the scientific method is that all single experiments are fallible and open to alternative interpretations, and that any matter of scientific fact is a provisional conclusion, based on reasoning to the best explanation so far. (Since challenges to the infallibility of the scientific method are relatively new—dating back only three hundred and fifty years or so—they may have escaped the attention of dedicated neo-Platonists.)

Or is your concern that heuristically-based approaches are intrinsically unscientific? If so, have you looked at Paul Feyerabend’s Against Method, in which he shows that the progress of science itself has been decidedly unscientific?

Is there something non-empirical or unmeasurable about the stopping heuristics that I’ve provided? Is your objection that most of these heuristics don’t use the kind of third-order measurement that physicists use? If so, you might like to consider that most of the important aspects of software development and of software’s value are rooted in self-aware and social systems. In such domains, third-order measurement is not only inaccurate and inappropriate (look here) but also leads to distortion and dysfunction (look here).

Is your objection that heuristics are unreliable or invalid? On the one hand, that’s a reasonable objection because heuristics are by definition not perfectly reliable. On the other hand, it’s not like more algorithmic methods are by their nature more trustworthy simply because they’re algorithmic. After all, an algorithm can be applied in an inappropriate context, or can be based on an invalid model. The Weibull distribution is a classic example. Cem Kaner and Walter (“Pat”) Bond give a detailed refutation of its applicability to software bug-finding metrics here.

Most of all I’m curious about how the list of stopping heuristics I’ve provided is any less scientific than this list of heuristics that I found on the page pointed to by the link that you kindly provided:

We can release when:
Coverage is OK

Non-functional features are OK
Code turmoil is OK
Feature stability is OK

In general, I’m inclined to agree with the items in this list and the evaluation criterion of “OK”, since both appear to be heuristically based. I do have some specific concerns though.

All the code has been exercised. It’s possible to exercise every line of code in a product without observing a single bug, even when there are terrible problems in the product, just as it’s possible for a parking enforcement cop to walk along every street in a town without observing a single expired meter or issuing a single ticket.

Every feature has been shown to work. That’s a nice idea too, but it’s not terribly hard, even for a profoundly broken program, to show that features can work. Excellent testing isn’t about that, though; it’s about a diligent search for possible failures. You see the difference, I hope: one approach is focused on confirmation and verification; the other is focused on exploration, discovery, investigation, and learning. The former is a very weak kind of testing; the latter much stronger.

Every use case scenario has been exercised. That’s certainly one kind of (or heuristic for) test coverage. Thinking about it for a moment raises the possibility that there are ways of using the product that aren’t covered by the scenarios in the use cases, especially since use cases are typically crafted at the beginning of the project or development cycle, when we know less than we will ever know about the product.

All the tests have been run. That a heuristic for test coverage too, but it poses some questions. What about the quality of the tests? What about the quality of the oracles that inform the tests? What about the skill of the tester? Do the tests cover the product sufficiently to address the most important risks and to inform the ship or no-ship decision?

The number of bugs found is almost the number expected. Almost? The number expected? What’s the basis for your expectation? Is that expectation valid? What might threaten the validity of that expectation? What if there are more bugs than you expected? What if there are fewer bugs than you expected? What does “almost” mean? Everything but the last two? or three? or four bugs that on their own would demolish the value of the product?

The unfixed bugs are few enough. I have no particular concern with the number of bugs in the product, since the number has little relevance. What does have relevance is the significance of the bugs, irrespective of their number. (As examples, Microsoft Windows 2000, a very successful product, shipped with over 60,000 open issues; in the 1960s, an Algol compiler from IBM shipped with only one bug: it wouldn’t load. People still obtained value from Windows 2000, which may be unsurprising. What might be a little more surprising is that some people obtained value from the Algol compiler too; the availability of an Algol compiler, even one that didn’t work, was sufficient to address a sales problem that IBM needed to address.)

So, considering that you’ve presented a list of heuristics that seem to be rooted in un- or quasi-scientific principles at best, colour me confused about your objections to mine.