Blog Posts from April, 2010

Coding QA Podcast on Exploratory Testing (Part 2)

Sunday, April 11th, 2010

Here’s Part 2 of my notes on the CodingQA podcast, in which Matthew Osborn and Federico Silva Armas chat with James Bach about the exploratory testing and session-based test management.

Skills of Exploratory Testing

  • If you want to develop a list of testing skills, you might find it helpful to start with the Exploratory Skills and Dynamics sheet, by James Bach, Jon Bach, and Michael Bolton.
  • One of the core skills of excellent exploratory testing is self-awareness and self-management; managing your attention and your focus.
  • Another skill of excellent testing is using your emotions as a test tool. Frustration or confusion about something may affect the user in the same kind of way that it affects you. Treat your emotional reaction as a trigger; start looking into why you might be experiencing that emotion.

Test Cases

  • Except in well-defined contexts, counting test cases means absolutely nothing. When someone gives you a number, telling you that they have N test cases, they’re telling you they have N files.
  • Test cases are like briefcases; if you want to know something about them, you have to look inside them to know what’s going on.
  • What people do when they test is different from what’s in the test case. The test is what the tester does (and thinks –MB), not what the test case says.
  • In an automated test, the problem is that there might not be an oracle to observe a bug that’s outside the scope of that test, and there’s no human to say, “That’s odd. I wonder what’s going on there.”
  • When managers ask you to count your test cases, they’re asking you to count checks, and that doesn’t get to the essence of what testing is.

Managing Exploratory Testing

  • If you’re going to manage and train exploratory testers, it would probably be a good idea to break out a list of constituent skills of excellent testing (see above). Then ask, for each skill: How can we observe it? And how can we teach it?
  • If you’re going to manage any highly cognitive work, you’re going to have to participate in the work on a regular basis. If managers aren’t involved in the work themselves, they don’t really know what their people are doing; they know only the story that their people are telling them. The less management is involved in testing, the more likely they are to come up with weird astrology-like methods for managing testing.
  • How do you manage testing on a large scale? Like the military, develop a corps of sergeants. Test leads—skilled, trusted testers—are trained on the job, and train other testers and other test leads. This is also consistent with the way that doctors, pilots, scientists, and other cognitively skilled professionals are trained.
  • Good doctors, welders, and pilots are not trained by filling out templates.
  • The other component to managing exploratory testing is session-based test management: uninterrupted blocks of tester time.

Session-Based Test Management

  • A block of uninterrupted time (a session, typically 90 minutes) is accounted in terms of three activities: test design and execution (T-time); bug investigation and reporting (B-time); and setup (S-time). You account from your time as test design and execution, unless it’s interrupted by B-time or S-time.
  • T-time has the highest precedence, then B-time, then S-time. If you’re doing more than one thing at once (e.g. investigating a bug while you’re setting up), report it as the higher-precendence activity.
  • Accounting for time in this way allows you to identify ways in which more testing can get done. (Here’s the first part of an example, and here’s the second part. Note that the metrics don’t tell you what’s good; they suggest where to look. The purpose of this kind of measurement is not to provide answers, but to provide more focused questions on what we should do next and how we should allocate our time. –MB)
  • In a session format, after you’ve gathered a bit of data, a picture emerges of how much you can get done. A middle-level test strategy language evolves: “That part of the product looks like it will take two recon sessions, three analysis sessions, ten deep coverage sessions, and two closure sessions.” (The CodingQA guys have developed their own language—discovery sessions and targeted sessions. By this kind of language, testing sessions turn into calendar time.
  • In exploratory testing, testing resources develop: test data, test platforms, test ideas in documents.
  • Test leads handle the debriefing of session. Debriefings and formal acceptance of the test notes are essential in developing discipline in note-taking and reporting.
  • The SBTM protocol was initially to debrief at the end of every session, but this wasn’t always possible. More recently, the protocol is to debrief in groups; everyone reporting to a given test lead meets at the end of the day for a half-hour debriefing session.
  • The overall philosophy should be to tinker with the process to get the bureaucracy out of the way of the thinking tester.
  • After two years of success with SBTM, at the place where it was developed, management stopped asking for metrics, but the team kept using the session concept.

For more information on session-based test management, see

Recording Exploratory Testing

  • Every novice’s first set of test notes looks like this: a single page that says, “I DID THE TESTING”. The second set of test notes contains every keystroke and every mouse click. Novices need training and instruction in how to create good notes that strike the right balance between detail and conciseness.
  • The fastest way, in the long run, to train novices is to provide them with pairing and personal supervision, showing them how to do the notes.
  • Testers are not often challenged—and therefore have not developed the skill—to give expert reports on what they actually did as they were testing. For a while, senior testers who are being trained in this kind of reporting can get annoyed and frustrated.
  • The more practice you get in note-taking, the more skill you develop, and the less it interferes with your thought process.
  • Watch the episode “Gallileo Was Right” from the HBO series “From the Earth to the Moon”, and look for the segment in which astronauts are trained to become geological observers and reporters. It’s all about rapid testing!
  • Federico reports an irony: that on his team, exploratory testing (so often mistaken for “testing without writing anything down”) became subject to the opposite problem: it became all about the test notes, which disrupted testing and learning.
  • Test notes in SBTM should be focused not on the literal steps of execution, but on questions, wondering, models, test ideas, coverage ideas, evidence of learning.
  • Correct the problem by sitting with testers and test with them.
  • In excellent exploratory testing, the testing is paramount. If the paperwork gets in the way, then the paperwork has to change until there’s a reasonable balance between paper and testing.
  • Combine the notes with recordings of the session—a videotape; a screen recording system; application and server log files; tools like Burp Proxy or Fiddler. Lean on automation and tools to keep records of the minutiae of what’s going on.
  • Beware of systems that have encoded or encrypted communication between server and client. The transactions might be logged, but might be hard to decode or decrypt.
  • You don’t look at the data unless you experience a problem; you normally file the video and not worry about it unless you need hard evidence or review.
  • With a screen recorder, it might be hard to make sense of what happened. Therefore: keep notes electronically on the same machine that you’re testing on. This causes the notes to pop up on the screen concurrent with the test, which provides insight into what the tester was thinking.
  • The notes should be just enough to figure out the highlights of what you were doing.
  • Timestamps help to link the test idea with what actually happened and when it happened. This allows you to move quickly, taking notes that are more easily reviewable.

Exploratory Testing and Interviews

Wednesday, April 7th, 2010

I’m was interviewed on April 19, 2010 by Gil Broza, an expert Agile coach who is a colleague and friend here in Toronto.

Gil’s request for an interview reminded me of an experience I had a few beforehand. I received an email from a couple of researchers in Sweden who are studying exploratory testing.  I was honoured to be asked for my point of view on the subject. However, I was a little startled when the gentlemen provided me with a list of 27 questions about the problems of exploratory testing.  And 23 about the benefits of exploratory testing. And 16 about the problems of test-case-based testing. And 17 about the benefits of test-case-based testing.

That seemed like a lot of questions to answer. To answer some of them sufficiently would have required a straight Yes or No.  To answer others well would have involved sprawling issues of ontology and epistemology.  Some questions asked for more than one answer (“Please list down at least five problems related to ET.”) Some questions were about experiences, and asked for stories. The only sure thing was a that thorough reply would have been hours of writing (and even though it’s about a subject I love, fairly tedious writing.) I was traveling, and figured I could only give them an hour, so I got them to phone me instead, one morning while I was in Trondheim, Norway.

It was a great experience. We had a grand chat (we went a few minutes over the hour, we were having so much fun) and what’s more, it provided a wonderful set of metaphors for testing.

  • Excellent exploratory testing is like interviewing a program. Imagine that you work at a placement agency, linking candidate workers to your clients.  One of your key tasks is to qualify your candidates before you send them out for jobs or for interviews.  To make sure they’ll be ready for whatever your clients might throw at them, you test them through an interview of your own.  You can plan for that interview by all means, but what happens during the interview is to some degree unpredictable, because for each question, the answer that you get informs decisions about the next question. One way to test (a great way, and an important way, I believe) is to treat your program like a prospective employee for your customers.  You’re not merely going to test that the candidate can answer some questions correctly (that is, is the candidate capable?); you’re going to look at the whole package.  Does the candidate deal appropriately with surprising or malformed situations (that is, is the candidate reliable)?  When he gets stuck, does the candidate know how to ask for help politely and attempt to move forward, or does he just sit there stupidly (in the software world, we’d ask questions about user-friendliness, usability, and error messages)?  Can the candidate deal with being stressed out or overwhelmed, or does the candidate just collapse in a heap (performance?  scalability?  reliability?)?

    Scripted testing is like sending someone a list of 83 written questions and expecting 83 written answers. The answer to one question will not inform the next question, unless you design a mechanism for feedback and course correction, such as only submitting a few questions or answers at a time.  Notice also that like “83 test cases”, the quantity “83 questions” doesn’t really mean very much to you.  Until you’ve seen the questions, you can’t really know anything about them.  You can only know what I’ve told you about them.  Were they good questions?  Bad?  Multiple choice?  Worthy of a quick, a deep, or a practical answer?  Did they reflect what the personnel agency’s clients really wanted to know about the candidate?  What they really needed to know?

    Exploratory testing takes into account the possibility that you might get a different answer to your question from one day to the next, and that that might result in you asking new and different questions. Thus exploratory testing focuses on adaptability. Scripted testing emphasizes the answers that you want to hear from a program, the same way every time you ask, without varation. Therefore, scripted testing focuses on invariance. Note that invariance can be checked, but adaptability must be tested. It makes sense to delegate the most extreme kind of scripted testing—checking—to a machine.  Humans are, as Jerry Weinberg put it, at best variable when it comes to showing that the machine can do the same thing with different numbers.  Automation is great at that stuff.  If humans are indeed unpredicatable and variable, it makes sense to treat those tendencies as assets when it comes to testing, and exploit the natural human capacity for variation and adaptation to test the system’s capacity to adapt.

  • When I received the questions, I felt overwhelmed and, frankly, irritated.  During the interview I felt comfortable and relaxed. I noted that the dialogue, the conversation, felt very natural and immediate. The list of questions, although thoughtfully conceived, had felt stilted and disjointed.  Rather than interacting with a static piece of paper, I could hear human voices at the other end of the line. The different feeling offered by the two modes of communication is natural. We enter the world as listeners and speakers, not as readers and writers. Our minds and our sensory systems are biased by our heritage to prefer immediacy, dealing with people as directly as possible. Computers and documents are media, technologies. They extend our capabilities and our senses in some ways and diminish them in others. A paper document turns a conversation from a set of loops into a linear sequence. A computer allows us to converse in real time over great distances, sending video, audio, text, and documents, but it still lacks the immediacy of actual presence and the rich sensory environment in which people evolved.
  • Since we only had an hour, the researchers and I realized that we had to begin by focusing on the questions that were most important to them. Except we started the interview with different ideas about what might be important. Thus we discovered what was really important along the way. At the end of the session, we agreed that we had covered almost all of the initial questions anyway.  It turned out that a little over an hour of conversation was enough to give them plenty of material to consider. That’s like testing too. We often find that with the right approach, discovering what we want to know might take a lot less time and a lot less effort than we think. Rapid feedback loops—like those available in an exploratory approach—can help us to work out quickly what is important to test and what might not be so important.  (That’s why, when a test team is under time pressure, there’s a powerful and natural desire to explore rather than to stick with an overly scripted process.  The trick is to have the skill to do the testing well, in expert and skillful ways.)

So, interviews are like exploratory testing.  They’re fun, and that’s why I enjoyed Gil’s interview with me on the subject, “Is There a Problem Here?”

Some dust and cobwebs were cleared from this post on 2019-01-15.