Blog: A Moment of Jerry Weinberg Zen

The year was 2006. James Bach and I were running a workshop at the Amplifying Your Effectiveness conference (AYE). We were in one of those large-ish, high-ceiling conference rooms with about 15 programmers and software consultants.

We were showing them one of James Lyndsay’s wonderful testing machines. (You can find it here, but you’ll need Flash active to run it.) It looked like this:

James Lyndsday's Machine 1

At first, it’s all very confusing. When you press the buttons on the left, the red and blue balls on the right move in some way. The slider in the middle influences the range of motion somehow. In general, the mission of the exercise is to describe the behaviour of the machine.

Test cases are not testing. To illustrate this important fact, we give class participants the machines to investigate for a few moments, and then ask a question that James (Bach) asked in our AYE session.

“How many test cases,” he asked, “would you need to be able to understand and describe this product completely?”

Brows immediately furrowed. Clicking sounds from the buttons and murmured conversation between pairs of participants filled the room. “Two states to the power of five buttons with how many stops on that slider…?” “Wait, that button is just momentary…” “Seven hundred and sixt… no, that’s wrong.”

Whereupon, in a moment of perfect timing, a door opened, and Jerry Weinberg walked into the room. His walking stick and his bearing reminded me of Yoda and Gandalf and other sages and wizards.

“Hey, here’s Jerry Weinberg!” said James. “The world’s greatest living software tester! Jerry, how many test cases would you need to understand and describe this product completely?”

The room fell silent. Everyone wanted to know the answer. Jerry observed the laptop that James was holding. He didn’t touch the laptop, press a key, or move the mouse. He just looked for a few moments.

Then he said, “Three.” There was a pause.

Having worked with Jerry over a decade or so, James understood. “Three, Jerry?”, he asked dramatically, in mock astonishment.

“Hm.” (pause) “Yeah. Three,” replied Jerry. Another pause.

Then he peered at James. “Why? Were you expecting some other number?”

Rapid Software Testing is coming soon to Sydney, Brisbane, Utrecht, and Copenhagen. Learn more about it, and find places to register.

Want to know more? Learn about upcoming Rapid Software Testing classes here.

9 responses to “A Moment of Jerry Weinberg Zen”

  1. Jim Bullock says:

    I will bet that if pressed, Jerry could have named the three cases he had in mind.

    And what he was up to was in his reply (also in the “session based” “rapid” and related testing approaches.)

    Clever man.

  2. Vladyslav says:

    “Three” meaning himself, You and James? 🙂

    Michael replies: Maybe. Were you expecting some other explanation?

  3. […] A Moment of Jerry Weinberg Zen Written by: Michael Bolton […]

  4. Chad says:

    Fun story. Your live portrayal was hilarious. Happy New Years.

  5. Vladyslav says:

    Nope of course 🙂

  6. Andrew says:

    “Having worked with Jerry over a decade or so, James understood.”

    I’m not sure that I understand but based on my level of familiarity with your writing on the subject of software testing, I’m led to believe that Mr. Weinberg’s answer is to be taken as a mockery of the question itself, correct?

    Michael replies: I’m not sure I’d say “mockery”, although the answer affords that interpretation.

    After all, he did not interact with the puzzle at all prior to his answer. You mentioned, “When you press the buttons on the left, the red and blue balls on the right move in some way. The slider in the middle influences the range of motion somehow.”, but did he know these requirements to be true ahead of time? How are we defining “test case”? Does he have the same definition? And how is it that he could know that he would be able to understand and describe this product _completely_ in 3 test cases without any exploration and experimentation ahead of time? Why would the greatest _tester_ of all time come to an answer without actually doing any _testing_?

    We can’t ask him any more, sad to say, but I have some pretty well founded beliefs about how Jerry might address your questions. I’ll try to channel him by various means.

    Jerry didn’t worry about whether requirements or specifications were “true” or “correct”. What concerned him was whether the product the customer had was the product the customer wanted.

    Jerry also didn’t think very highly of counting test cases. One of the earliest things he ever published was Computer Programming Fundamentals, written with Herbert Leeds. Jerry wrote the chapter on testing. In it, he tells the story of a programmer who, on several occasions, believed he had found the last bug in his program. Eventually, after each repair, another deeper and more damaging bug was discovered. As the epilogue to the story, Jerry says:

    Jim Bullock says:

    “…mockery of the question…”

    It’s Michael’s blog, but I’ll chime in and he can correct.

    Not “mockery” as much as illuminating the question’s deficiencies. This is the point of that kind of training exercise — I’ve seen Michael’s and James’ work — and Jerry’s answer. Deficient questions are a thing.

    The hint is here, I think: “Then he peered at James. “Why? Were you expecting some other number?”” What were you *expecting*?

    Information about what something does is pretty useless unless you define what you want, intend, and expect from the thing.

    With his reply, Jerry actually made a little morality play with himself as the SUT, and James as the tester, illustrating exactly what the exercise in progress was illustrating: “Were you expecting some other number?” Nobody knows what to expect from the box, so any answer to “How much to test?” comes from somewhere other than what the box does.

    I suspect Jerry could name 3 test cases in that situation that were about the testing system itself, which would likewise illustrate the point.

    This is similar to the notion of “foretelling” from LeGuinn’s The Left Hand of Darkness, an art practiced to prove the “…perfect uselessness of knowing the answer to the wrong question”.

    Any particular test case, number of test cases, testing stragegy, testing activity, or testing program, may produce robust answer (if well done) to the wrong question, thus be perfectly useless.

    Michael replies: Thank you Jim. It’s an honour to have you reply to a reply on my blog — and what a wonderful reply you’ve given.

  7. Jim Bullock says:

    Michael: Thank you Jim.

    Well thank you, kind host, and also for making a place where I can chime in sometimes, even though I’m too cranky and broken to wrangle my own web presence.

  8. Andrew says:

    Fascinating. Thanks to both of you, Michael and Jim.

Leave a Reply

Your email address will not be published. Required fields are marked *