Blog Posts from December, 2007

Who’s to Blame?

Monday, December 24th, 2007

A couple of people, both SDETs at Microsoft, have responded either to Joel Spolsky’s post on SDET culture at Microsoft or to my excerpt of it in a recent blog post of usefulness in software. Those people deserve a response.

The first issue is that they’ve taken Joel to task for promulgating rumours. But these are rumours in the same sense that the cell theory and the theory of natural selection are theories; usefully complete, good-enough summaries of the issue for the purpose of furthering the discussion–in the same sense that we spoke of Conrad Black’s alleged embezzlement. I’ve heard about the issue of SDET hiring focus from at least a dozen people, and most of them work at Microsoft.

I can’t speak for Joel, but I can speak for my own observations: I’m not blaming the SDETs for the problems in Vista. The decision to ship a product, irrespective of the shape that it’s in, is always a business decision. Testers aren’t to blame for the flaws in the product. We don’t put ’em in, and if we find them, we are to some degree lucky.

Wait–why lucky? Doesn’t skill matter? What about experience? Don’t tools help? Sure they do; all those things can help a lot. But… products and the systems that they run on are hideously complex. The set of tests that we could run is potentially infinite. Problems are unexpected, subjective, numerous, and many of them can’t be identified by a non-sapient machine. Most of all, the problems in software are usually hidden. If we’re searching for problems an intractably large space in very limited time, luck will play a part.

Testers don’t (and shouldn’t) direct product development. Program managers, project managers, product owners, senior managers–whatever they might be called–do that. Those people have authority over the budget; the schedule; the scope of the problem that it’s intended to solve; staffing, hiring and firing; business relationships; contractual obligations; annual bonuses; and the decision to ship the product. They also have control over its design, how much it’s going to pester the user, and whose software is going to ask (persistently; very persistently in Vista’s case) for you to install it. For a price.

So the SDETs aren’t responsible for the problems in Vista. What is responsible, I think, is the culture. That culture leads what I perceive to be an overly heavy automation strategy and a testing department dominated by things that programmers value, rather than things that customers value. That’s a problem, but it’s only a symptom of a wider problem. That problem results in a Vista that is so annoying that I spent a substantial amount of time and effort to make sure that it didn’t get installed on machines that I use regularly. Vista isn’t valuable to me (and, I believe, to millions of others), even though it might be functionally correct. It does perfectly all kinds of things that I don’t want it to do; and it fails to do at all things that I want it to do. That was my point, and I think it was one of Joel’s, too.

Joel Nails It

Wednesday, December 19th, 2007

I’ve just had to buy a new computer, and am now becoming intimate with Windows Vista. “Becoming intimate” is a euphemism that people sometimes use when they mean “being screwed”. The biggest problem I’ve encountered so far is with Vista’s User Interface Feature Concealment module, which took the few remaining things that were reasonably intuitive and accessible in Windows XP and hid them.

Joel Spolsky, he of Joel on Software, is someone that I’ve admired for a long time. In this blog post—the first part of a talk that he gave at Yale on November 28, 2007, along with part two and part three—he identifies a key concept about quality that few people identify: the absence of bugs doth not a quality product make. Technical correctness is a wonderful thing, but there has to be value in that correctness. Quoth Joel (from part one):

The Windows Vista team at Microsoft is a case in point. Apparently—and this is all based on blog rumors and innuendo—Microsoft has had a long term policy of eliminating all software testers who don’t know how to write code, replacing them with what they call SDETs, Software Development Engineers in Test, programmers who write automated testing scripts.

The old testers at Microsoft checked lots of things: they checked if fonts were consistent and legible, they checked that the location of controls on dialog boxes was reasonable and neatly aligned, they checked whether the screen flickered when you did things, they looked at how the UI flowed, they considered how easy the software was to use, how consistent the wording was, they worried about performance, they checked the spelling and grammar of all the error messages, and they spent a lot of time making sure that the user interface was consistent from one part of the product to another, because a consistent user interface is easier to use than an inconsistent one.

None of those things could be checked by automated scripts. And so one result of the new emphasis on automated testing was that the Vista release of Windows was extremely inconsistent and unpolished. Lots of obvious problems got through in the final product… none of which was a “bug” by the definition of the automated scripts, but every one of which contributed to the general feeling that Vista was a downgrade from XP. The geeky definition of quality won out over the suit’s definition; I’m sure the automated scripts for Windows Vista are running at 100% success right now at Microsoft, but it doesn’t help when just about every tech reviewer is advising people to stick with XP for as long as humanly possible. It turns out that nobody wrote the automated test to check if Vista provided users with a compelling reason to upgrade from XP.

Jerry Weinberg, who has a habit of Getting There First, is one of the people who does talk about this. Here’s an example from Quality Software Management, Vol. 1: Systems Thinking.

Though copious errors guarantee worthlessness, having zero errors guarantees nothing at all about the value of software. Let’s take one example. Would you offer me $100 for a zero defect program to compute the horoscope of Philip Amberly Warblemaxon, who died in 1927 after a 37-year career as a filing clerk in a hat factory in Akron? I doubt it, because to have value, software must be more than perfect. It must be useful to someone.”

Dave Platt also gets it right in the talk that goes with his wonderful book Why Software Sucks: “As surprising as it might be to you, your customers don’t want to use your software. They want to have used your software.”

I have it from a fairly reliable source that Microsoft, as of a couple of months ago, had 650 open positions for teste{backspace}{backspace}{backspace}{backspace}{backspace}SDETs. After my first looks at Vista (over the last couple of weeks), and of Office 2007, I’ll say this: don’t bother filling those open reqs, guys. Instead of finding even more SDETs to write even more scripts to show that the program still works the way some programmer coded it, try hiring more people who are interested in looking at the whole program, not just the API. There are SDETs like Michael Hunter (as of way before 2019-01-15, no longer at Microsoft), but they’re very few and very far between. You have more than enough people with his coding skills; you need to hire more people, way more people, with his thinking skills.

EuroSTAR: The Good Stuff

Monday, December 17th, 2007

Much of what I’ve reported about EuroSTAR has been on the topic that had the most energy around it–the very gratifying response to my keynote presentation. I’d be remiss in not mentioning some other highlights.

A tip of the hat to Tracy O’Connell and the friendly, helpful, and capable staff of EuroSTAR 2007. The entire conference went off splendidly.

Henrik Andersson, a member of the context-driven community and a real driver of investigative testing at Sogeti, was a very kind and able host. Henrik is a kind of agent provocateur (in all of the good ways, and none of the bad) in the Swedish testing community. If you’re able to hear him speak or to get into a conversation with him, do it. You’ll be rewarded for your time.

Sogeti held a wonderful dinner in downtown Stockholm one of the nights of the conference, and I was very grateful to be invited. Many thanks.

At that dinner, I met Mattias Göransson. We played three testing games, exercises from Rapid Software Testing. He did very well on two of them. One, called The Mysterious Sphere, he destroyed. With one question (which I can’t reveal here, because to do so would compromise the exercise for anyone reading the blog), he undermined the foundation of the problem posed by the exercise, such that the problem entirely collapsed around itself. I was dead impressed.

The night earlier, Björn Hagstrom was also game enough to try one of the testing puzzles. This one uses the dice to model a certain class of testing problem. It’s not an easy problem to start with, and there’s an element of randomness to it that can lead to quick insight or great difficulty in solving it. Bjorn had absolutely the worst luck I’ve seen in almost three years of presenting this exercise. Nonetheless, he hung in there, and used his wiles and his resources (including some observations from a tester sitting next to him, a French fellow named Yannick–I’m sorry I didn’t get his last name, because I’d like to keep in touch with him). Admirable stamina in the face of terrible luc

Torbjörn Ryber also worked on a special puzzle, a work in progress that will eventually become an RST exercise. He provided a bunch of valuable observations that will make the exercise stronger. Better yet, he brought (and gave away) 6oo copies of his just-published-in-English book, Essential Software Test Design. I haven’t read it yet, but it’s on the stack, I have scanned it, and it looks excellent.

It’s fun showing puzzles to other testers. I get a warm feeling about people who are willing to accept the challenges, and I really appreciate learning about other people’s approaches to solving them. All of these guys were really impressive in their strategies and tactics.

Stuart Reid and I have very strong differences of opinion on certification and other aspects of testing. I perceive that he sees some things as hard that I think are easier, and I perceive that he seeks simplicity and assurance in a world that I see as sometimes complex and messy–and yet still perfectly livable. We had a very animated discussion in front of the exhibit hall at EuroSTAR that might have entertained some of the onlookers and startled others. I appreciate the good will and respect that Stuart offered, and offer the same in return, despite the disagreements. Time will sort those out eventually, and we’ll both learn things along the way. I thank Stuart for the energetic discussion (neither of us had had coffee!) and for the opportunity to keynote at the conference.

Still more to come on EuroSTAR…

Certification and Its Discontents I

Thursday, December 6th, 2007

EuroSTAR has been a wonderful conference for me. Alas, I’ve been unable to attend many of the track presentations because I’ve been, well, conferring. I’ve had some very engaging and interesting chats, and it’s been great to connect familiar names with their faces. I look forward to many more visits to Europe.

Today’s activities included a conversation, sometimes quite animated, with Stuart Reid. I’ve been surprised at the strength of my emotional reaction to the conversation. It has deepened my already subterranean feelings about the ISEB/ISTQB Foundation Level certification. I think this aspect of the certification business is particularly odious–tantamount to a tax on the vulnerable and the credulous.

If I understood Stuart correctly, I believe that his principal argument was that the ISEB/ISTQB Foundation Level certification demonstrates an engaged interest in the testing profession (uh, applying for, or holding, a testing job indicates an engaged interest in the testing profession, doesn’t it?) and thereby makes it easier for HR departments to filter applicants for testing jobs. (As usual, if I am mischaracterizing the argument, I welcome a correction, and I’ll publish it.) If the poor businesses don’t have the certification to separate the wheat from the chaff, how are they supposed to qualify the applicants when they receive hundreds of resumes?

Well… I suppose that the hiring staff could select applicants that send a resume and a covering letter that is clearly and neatly written, and

  • that states and articulates an engaged interest in the testing profession, in the applicant’s own words; or
  • that points to relevant life experience; or
  • that points to relevant work experience; or
  • that highlights relevant education; or
  • that points to seemingly irrelevant education or work or life experience, demonstrating a connection that might not be immediately obvious to the hiring manager, yet still intriguing; or
  • that points to self-study via books or online courses on testing; or
  • that provides a sample one-page test report for an open-source software product; or
  • that provides a link to the applicant’s work in the bug-reporting system for an online software product; or
  • that answers a set of three or four questions about testing basics provided in the advertisement; or
  • that provides direct statements of endorsement from a personal reference; or
  • that provides acknowledgment of the requirement for certification along with an argument as to why the hiring manager should consider the candidate anyway, despite the absence of the certification; or
  • that places the letters “ISEB” after the candidate’s name, with a footnote that says “I’m Still Employable, Buddy”;
  • or that uses any number of attention-getting devices that companies have been using to qualify entry-level applicants forever.

This would take too much work and too much time, was one reply. Too much compared to what? If we’re going to hire someone for a responsible technical position with the company, possibly for years, isn’t it worth a couple of minutes per applicant to select a set that we want to invite to an interview–where the real qualification begins? I don’t see any reason why our profession should shore up an incompetent HR department with a specious certification scheme. If the hiring manager doesn’t know how to hire or how to manage, why provide them with any help at all? (Jeez, why doesn’t the hiring manager go get a management certification?)

In addition, the resume-filtering argument ignores the fact that companies frequently hire testers from within. Testers often come from customer support (I did), from internal support, from development, from administration, or from internship. They come pre-qualified. Testers might also come through personal links and references–“I know someone who’d be great for that job.”

Have a look at this page of frequently asked questions from the Australia/New Zealand Testing Qualifications Board. Ask some more questions, less frequently asked: why is the exam a multiple choice test of forty questions for which there is only one correct answer per question? Why is a passing grade is 62.5% (25 out of 40)? Why are approximately half of the questions at the simplest level, such that getting the simplest questions correct, plus chance levels of achievement on the not-simplest questions is a passing grade? Why does it takes three weeks to mark? Why (especially considering the fee) is there no mechanism for feedback? Where have the ISEB/ISTQB and its affiliated boards managed to spend USD200/AUD300/GBP200 times 30,000 testers worldwide?

Anna Docherty gave a very good, though admittedly preliminary, presentation on some survey research that she did herself on certification (not just the foundation level) and its strengths and weaknesses in the field. The questions that she raised were pointed and pertinent, and she acknowledged that many questions remained unasked or unanswered in her research. In that sense, it was just like a good test report, and Anna immediately qualified herself under my personal certification scheme. I’ll discuss some suggestions for further investigation in a later post.

What truly astonished me today, though, was the number, immediacy, and directness of the statements, even from strong ISEB certification advocates, that the Foundation Level certification is “garbage”, “stupid”, and “useless”–those were their words. Come on guys; walk the talk. Trash it. Or make it free. Then trash it.

Why I Am Not Yet Certified — EuroSTAR Presentation

Wednesday, December 5th, 2007

Today, December 4 2007, I gave a presentation at EuroSTAR on “Why I Am Not (Yet) Certified“. James Bach was originally slated to give a different presentation with the same title, but I got the nod due to the untimely illness of James’ wife Lenore, which caused him to cancel his fall schedule (she’s much better now).

Stuart Reid, the chair of the conference, strongly supports the notion of certifications in their current forms. I disagree with that, but I have considerable respect for people who are willing to provide a platform for opposing views, and I therefore thank him for providing the opportunity to speak. I think the controversy opens up the discussion, and thereby strengthens the conference and the craft of testing.

As I said as I finished the presentation, I felt a little like Martin Luther nailing 42 PowerPoint slides to the screen. The talk was generally well received, but there were several conversations that I found rather sobering.

At least two people to whom I spoke–one a former ISEB instructor–told me that they had wanted to effect change in the multiple choice Foundation exams, but their experience was that that couldn’t happen unless the ISEB/ISTQB Syllabus were to change–and changing that proved an insurmountable obstacle for them.

Almost everyone who approached me afterwards said that they were glad that I had said the things that they had been thinking privately for several years. They tended to be enthusiastic but they also tended to check to see whether they were among friends before they spoke freely. The latter is a tendency we need to break. As it was, it felt like revolution and insurrection were in the air–but nobody was quite brave enough to speak up. I encourage people to talk about this stuff, out loud and in public. Open criticism of things that are damaging to the craft is a form of self-certification in my community.

The complacence and chill were disturbing, but once a group of people were together, the complaints started to flow. Many had taken the ISEB/ISTQB certifications. All but one found little to no value in it. They complained about the triviality and the one-and-only-one-answer nature of the Foundation Level exam. Saddest of all, they noted that in Britain and in several countries on the continent, almost all businesses that are hiring testers require applicants for entry-level jobs to have the ISEB/ISTQB certification. I’m pretty certain that this will have several nasty effects. First, it is likely to discourage people from entering the testing field the way many of our best testers have done–by accident and opportunity. In turn, this will make the profession more insular and less diverse. In turn, this will prevent new ideas from reaching the craft. This is very bad.

We’re already learning this business slowly enough. If you attend conferences–especially the major commercial ones–you’ll hear near endless repetition of the same themes: heavyweight planning and estimation for a task that should be nimble, rapid, and responsive; bloated approaches to test documentation and artifacts; relentless focus on confirmation, verification, and validation, and very little talk of investigation, exploration, and discovery. It’s narcotic–the conferences seem addicted to these talks, and they make the craft sleepy. If we’re going to repeat anything, let’s repeat Einstein’s notion that the we can’t solve problems by using the same level of thinking that we used when we created them.

Jerry Weinberg Interview

Saturday, December 1st, 2007

For those who read my blog but don’t read Grig Georghiu‘s, he notes an interview with Jerry Weinberg here. Thanks, Grig.