A couple of people, both SDETs at Microsoft, have responded either to Joel Spolsky’s post on SDET culture at Microsoft or to my excerpt of it in a recent blog post of usefulness in software. Those people deserve a response.
The first issue is that they’ve taken Joel to task for promulgating rumours. But these are rumours in the same sense that the cell theory and the theory of natural selection are theories; usefully complete, good-enough summaries of the issue for the purpose of furthering the discussion–in the same sense that we spoke of Conrad Black’s alleged embezzlement. I’ve heard about the issue of SDET hiring focus from at least a dozen people, and most of them work at Microsoft.
I can’t speak for Joel, but I can speak for my own observations: I’m not blaming the SDETs for the problems in Vista. The decision to ship a product, irrespective of the shape that it’s in, is always a business decision. Testers aren’t to blame for the flaws in the product. We don’t put ‘em in, and if we find them, we are to some degree lucky.
Wait–why lucky? Doesn’t skill matter? What about experience? Don’t tools help? Sure they do; all those things can help a lot. But… products and the systems that they run on are hideously complex. The set of tests that we could run is potentially infinite. Problems are unexpected, subjective, numerous, and many of them can’t be identified by a non-sapient machine. Most of all, the problems in software are usually hidden. If we’re searching for problems an intractably large space in very limited time, luck will play a part.
Testers don’t (and shouldn’t) direct product development. Program managers, project managers, product owners, senior managers–whatever they might be called–do that. Those people have authority over the budget; the schedule; the scope of the problem that it’s intended to solve; staffing, hiring and firing; business relationships; contractual obligations; annual bonuses; and the decision to ship the product. They also have control over its design, how much it’s going to pester the user, and whose software is going to ask (persistently; very persistently in Vista’s case) for you to install it. For a price.
So the SDETs aren’t responsible for the problems in Vista. What is responsible, I think, is the culture. That culture leads what I perceive to be an overly heavy automation strategy and a testing department dominated by things that programmers value, rather than things that customers value. That’s a problem, but it’s only a symptom of a wider problem. That problem results in a Vista that is so annoying that I spent a substantial amount of time and effort to make sure that it didn’t get installed on machines that I use regularly. Vista isn’t valuable to me (and, I believe, to millions of others), even though it might be functionally correct. It does perfectly all kinds of things that I don’t want it to do; and it fails to do at all things that I want it to do. That was my point, and I think it was one of Joel’s, too.