A couple of people, both SDETs at Microsoft, have responded either to Joel Spolsky’s post on SDET culture at Microsoft or to my excerpt of it in a recent blog post of usefulness in software. Those people deserve a response.

The first issue is that they’ve taken Joel to task for promulgating rumours. But these are rumours in the same sense that the cell theory and the theory of natural selection are theories; usefully complete, good-enough summaries of the issue for the purpose of furthering the discussion–in the same sense that we spoke of Conrad Black’s alleged embezzlement. I’ve heard about the issue of SDET hiring focus from at least a dozen people, and most of them work at Microsoft.

I can’t speak for Joel, but I can speak for my own observations: I’m not blaming the SDETs for the problems in Vista. The decision to ship a product, irrespective of the shape that it’s in, is always a business decision. Testers aren’t to blame for the flaws in the product. We don’t put ’em in, and if we find them, we are to some degree lucky.

Wait–why lucky? Doesn’t skill matter? What about experience? Don’t tools help? Sure they do; all those things can help a lot. But… products and the systems that they run on are hideously complex. The set of tests that we could run is potentially infinite. Problems are unexpected, subjective, numerous, and many of them can’t be identified by a non-sapient machine. Most of all, the problems in software are usually hidden. If we’re searching for problems an intractably large space in very limited time, luck will play a part.

Testers don’t (and shouldn’t) direct product development. Program managers, project managers, product owners, senior managers–whatever they might be called–do that. Those people have authority over the budget; the schedule; the scope of the problem that it’s intended to solve; staffing, hiring and firing; business relationships; contractual obligations; annual bonuses; and the decision to ship the product. They also have control over its design, how much it’s going to pester the user, and whose software is going to ask (persistently; very persistently in Vista’s case) for you to install it. For a price.

So the SDETs aren’t responsible for the problems in Vista. What is responsible, I think, is the culture. That culture leads what I perceive to be an overly heavy automation strategy and a testing department dominated by things that programmers value, rather than things that customers value. That’s a problem, but it’s only a symptom of a wider problem. That problem results in a Vista that is so annoying that I spent a substantial amount of time and effort to make sure that it didn’t get installed on machines that I use regularly. Vista isn’t valuable to me (and, I believe, to millions of others), even though it might be functionally correct. It does perfectly all kinds of things that I don’t want it to do; and it fails to do at all things that I want it to do. That was my point, and I think it was one of Joel’s, too.

I’ve just had to buy a new computer, and am now becoming intimate with Windows Vista. “Becoming intimate” is a euphemism that people sometimes use when they mean “being screwed”. The biggest problem I’ve encountered so far is with Vista’s User Interface Feature Concealment module, which took the few remaining things that were reasonably intuitive and accessible in Windows XP and hid them.

Joel Spolsky, he of Joel on Software, is someone that I’ve admired for a long time. In this blog post—the first part of a talk that he gave at Yale on November 28, 2007, along with part two and part three—he identifies a key concept about quality that few people identify: the absence of bugs doth not a quality product make. Technical correctness is a wonderful thing, but there has to be value in that correctness. Quoth Joel (from part one):

The Windows Vista team at Microsoft is a case in point. Apparently—and this is all based on blog rumors and innuendo—Microsoft has had a long term policy of eliminating all software testers who don’t know how to write code, replacing them with what they call SDETs, Software Development Engineers in Test, programmers who write automated testing scripts.

The old testers at Microsoft checked lots of things: they checked if fonts were consistent and legible, they checked that the location of controls on dialog boxes was reasonable and neatly aligned, they checked whether the screen flickered when you did things, they looked at how the UI flowed, they considered how easy the software was to use, how consistent the wording was, they worried about performance, they checked the spelling and grammar of all the error messages, and they spent a lot of time making sure that the user interface was consistent from one part of the product to another, because a consistent user interface is easier to use than an inconsistent one.

None of those things could be checked by automated scripts. And so one result of the new emphasis on automated testing was that the Vista release of Windows was extremely inconsistent and unpolished. Lots of obvious problems got through in the final product… none of which was a “bug” by the definition of the automated scripts, but every one of which contributed to the general feeling that Vista was a downgrade from XP. The geeky definition of quality won out over the suit’s definition; I’m sure the automated scripts for Windows Vista are running at 100% success right now at Microsoft, but it doesn’t help when just about every tech reviewer is advising people to stick with XP for as long as humanly possible. It turns out that nobody wrote the automated test to check if Vista provided users with a compelling reason to upgrade from XP.

Jerry Weinberg, who has a habit of Getting There First, is one of the people who does talk about this. Here’s an example from Quality Software Management, Vol. 1: Systems Thinking.

Though copious errors guarantee worthlessness, having zero errors guarantees nothing at all about the value of software. Let’s take one example. Would you offer me $100 for a zero defect program to compute the horoscope of Philip Amberly Warblemaxon, who died in 1927 after a 37-year career as a filing clerk in a hat factory in Akron? I doubt it, because to have value, software must be more than perfect. It must be useful to someone.”

Dave Platt also gets it right in the talk that goes with his wonderful book Why Software Sucks: “As surprising as it might be to you, your customers don’t want to use your software. They want to have used your software.”

I have it from a fairly reliable source that Microsoft, as of a couple of months ago, had 650 open positions for teste{backspace}{backspace}{backspace}{backspace}{backspace}SDETs. After my first looks at Vista (over the last couple of weeks), and of Office 2007, I’ll say this: don’t bother filling those open reqs, guys. Instead of finding even more SDETs to write even more scripts to show that the program still works the way some programmer coded it, try hiring more people who are interested in looking at the whole program, not just the API. There are SDETs like Michael Hunter (as of way before 2019-01-15, no longer at Microsoft), but they’re very few and very far between. You have more than enough people with his coding skills; you need to hire more people, way more people, with his thinking skills.

Much of what I’ve reported about EuroSTAR has been on the topic that had the most energy around it–the very gratifying response to my keynote presentation. I’d be remiss in not mentioning some other highlights.

A tip of the hat to Tracy O’Connell and the friendly, helpful, and capable staff of EuroSTAR 2007. The entire conference went off splendidly.

Henrik Andersson, a member of the context-driven community and a real driver of investigative testing at Sogeti, was a very kind and able host. Henrik is a kind of agent provocateur (in all of the good ways, and none of the bad) in the Swedish testing community. If you’re able to hear him speak or to get into a conversation with him, do it. You’ll be rewarded for your time.

Sogeti held a wonderful dinner in downtown Stockholm one of the nights of the conference, and I was very grateful to be invited. Many thanks.

At that dinner, I met Mattias Göransson. We played three testing games, exercises from Rapid Software Testing. He did very well on two of them. One, called The Mysterious Sphere, he destroyed. With one question (which I can’t reveal here, because to do so would compromise the exercise for anyone reading the blog), he undermined the foundation of the problem posed by the exercise, such that the problem entirely collapsed around itself. I was dead impressed.

The night earlier, Björn Hagstrom was also game enough to try one of the testing puzzles. This one uses the dice to model a certain class of testing problem. It’s not an easy problem to start with, and there’s an element of randomness to it that can lead to quick insight or great difficulty in solving it. Bjorn had absolutely the worst luck I’ve seen in almost three years of presenting this exercise. Nonetheless, he hung in there, and used his wiles and his resources (including some observations from a tester sitting next to him, a French fellow named Yannick–I’m sorry I didn’t get his last name, because I’d like to keep in touch with him). Admirable stamina in the face of terrible luc

Torbjörn Ryber also worked on a special puzzle, a work in progress that will eventually become an RST exercise. He provided a bunch of valuable observations that will make the exercise stronger. Better yet, he brought (and gave away) 6oo copies of his just-published-in-English book, Essential Software Test Design. I haven’t read it yet, but it’s on the stack, I have scanned it, and it looks excellent.

It’s fun showing puzzles to other testers. I get a warm feeling about people who are willing to accept the challenges, and I really appreciate learning about other people’s approaches to solving them. All of these guys were really impressive in their strategies and tactics.

Stuart Reid and I have very strong differences of opinion on certification and other aspects of testing. I perceive that he sees some things as hard that I think are easier, and I perceive that he seeks simplicity and assurance in a world that I see as sometimes complex and messy–and yet still perfectly livable. We had a very animated discussion in front of the exhibit hall at EuroSTAR that might have entertained some of the onlookers and startled others. I appreciate the good will and respect that Stuart offered, and offer the same in return, despite the disagreements. Time will sort those out eventually, and we’ll both learn things along the way. I thank Stuart for the energetic discussion (neither of us had had coffee!) and for the opportunity to keynote at the conference.

Still more to come on EuroSTAR…

For those who read my blog but don’t read Grig Georghiu‘s, he notes an interview with Jerry Weinberg here. Thanks, Grig.