Blog: Context-Free Questions for Testing

In Jerry Weinberg and Don Gause’s Exploring Requirements, there’s a set of context-free questions to ask about a product or service. The authors call them context-free questions, but to me, many of them are more like context-revealing questions.

In the Rapid Software Testing class, the participants and the instructors make discoveries courtesy of our exercises and conversations. Here’s a list of questions that come up fairly consistently, or that we try to encourage people to ask. Whether you’re working with something new or re-evaluating your status, you might find these questions helpful to you as you probe the context of the test project, your givens, and your mission.

I leave it as an exercise for the reader to link these questions to specific points in the Heuristic Test Strategy Model and the Satisfice Context Model.


  • Is it okay if I ask you questions?
  • Who is my client?
  • Are you my only client?
  • Who is the customer of the product?
  • Who are the other stakeholders?
  • What is my mission?
  • What else might be part of my mission?
  • What problems are you aware of that would threaten the value of this product or service?
  • Do you want a quick, practical, or deep answer to the mission or question you have in mind?
  • How much time do I have?
  • How long before the next release or deployment?
  • How long before the end of this testing or development cycle?
  • When do you want reports or answers?

  • How do you want me to provide them? How often?
  • When were you thinking of shipping or deploying this product or service?
  • What else do you want me to deliver?
  • How do you want me to deliver it?
  • This thing I’m testing… could I have it myself, please?
  • Is there another one like it?
  • Are there more than that?
  • Is that all there are?
  • How is this one expected to be the same or different from the other ones?
  • Here’s what I believe I see in front of me. What else could it be?
  • Here’s what I’m thinking right now. What else might be true? What if the opposite were true?
  • Could you describe how it works?
  • Could you draw me a diagram of how it works?
  • How would I recognize a problem?
  • I think I’m seeing a problem. Why do I think it’s a problem? For whom might it be a problem?
  • What does this thing depend upon?
  • What tools or materials were used to construct it?
  • Who built this thing?
  • Can I talk to them?
  • Are they easy to talk to? Helpful?
  • Have they ever built anything like this before?
  • Is there anyone that I should actively avoid?
  • Who else knows something about this?
  • Who’s the best person to ask about this?
  • Who are the local experts in this field?
  • Who are the acknowledged experts, even if they don’t work here?
  • Has anyone else tested this?
  • Can I see their results, please?
  • Who else is on my test team?
  • What skills and competencies are expected of me?
  • What other skills and competencies can be found on the test team? Elsewhere?
  • What skills and competencies might we be lacking?
  • What information is available to me?
  • Is there more information available?
  • Where could I find more information? Is that the last source you can think of?
  • In what other forms could I find information?
  • Is that all the information there is? Is there more? Are there more rules? Requirements? Specifications?
  • If information is in some way wanting, what can I do to help you discover or develop the information you need?
  • What equipment and tools are available to help with my testing?
  • What tools would you like me to build? Expect me to build?
  • Is there some data that is being processed by this thing?
  • Can I have some of that data?
  • Can I have a description of the data’s structures?
  • What are your feelings about this thing?
  • Who might feel differently?
  • How might they feel?
  • What do customers say about it?
  • Can I talk to the technical support people?
  • (How do I feel about this thing?)
  • Who can we trust? Is there anyone that we should distrust?
  • Is there anything that you would like to prohibit me explicitly from doing?
  • Are there any other questions I should be asking you?

17 Responses to “Context-Free Questions for Testing”

  1. sunjeet says:

    very useful list indeed!

    Michael replies: Thanks, Sunjeet!

    Some more questions coming to my mind which have been, in my testing experience, very useful for me (could have been covered implicitly in your list)

    1. Is there an existing version of this product in the market/production? What have been/are the issues with it? What are you trying to improve in this release?

    Yes. Your first question here is a special case of the more general “Is there another one like it?”, and the two latter questions are excellent follow-ups to it.

    2. Bug patterns? What usually tends to break and is important?

    This is an interesting one, because the answer can be very interesting meta-information about management of the project. The question to ponder is “Why is there something that usually tends to break?”

    3. What are your testing demands in the face of any “change” (continuous or in patches) to the software?

    4. What are your legal concerns which the test team should be wary of?

    Two more very nice additions. Thanks!

    thanks
    sunjeet

  2. Ken De Souza says:

    This looks a lot like the CIA Phoenix Checklist, except in the context of software testing. Good stuff.

    http://www.misentropy.com/2007/01/the-phoenix-che.html

    Michael replies: Wow! Fantastic list! Thanks, Ken!

  3. kristjanuba says:

    Hi…

    These are good for all sort of things. But most useful for anyone doing the “Mysterious Sphere” exercise :)

    Michael replies: We’re all of us always doing the Mysterious Sphere exercise, to some degree. :)

    One should be aware though, that asking a lot of questions might provoke harsh reactions. Especially if the other party does not know the answers.

    That’s why it’s important to begin with “Is it okay if I ask you questions?” And when the other party doesn’t know the answer, I think it’s important to emphasize the default emotional response that Jerry Weinberg recommends: empathy. It’s okay not to know stuff. To some degree, we should expect our clients not to know the answers to things; if they knew everything, we wouldn’t need to test.

    And what to do if you cannot ask questions – then one has to start making assumptions (and ask the questions later, if possible).

    Asking questions later is one approach. As we note though (often in the debrief to the Mysterious Sphere exercise) : sometimes stating an assumption is a very useful proxy for asking a question. Especially back in my career at Quarterdeck, I found that it was sometimes difficult to get the programmers’ attention long enough for them to answer a question. But when I stated an incorrect assumption in the internal bulletin board system, I’d get an answer pronto. It’s important not to mind being wrong about your assumptions, as long as you’re revisiting them, testing them with others, and getting them corrected when they’re wrong. That’s why we encourage test managers not to insist on people being right, but instead to develop a fault-tolerant environment where people can learn from mistakes without a grevious impact on products or projects.

    Thanks for writing!

  4. [...] list of questions to ask, by Michal Bolton, prompted me to write about assumptions. Most of these ideas come from doing the ‘dropping a [...]

  5. kristjanuba says:

    Just to clarify what I meant by ‘making assumptions’ I followed up on the topic at my blog: http://bit.ly/hTE04s

    > Michael replies: We’re all of us always doing the Mysterious Sphere exercise, to some degree

    True, and as usual, the most crucial bit with every exercise is, how to take the knowledge from them to the real life.

  6. Fantastic list, thx!
    Will try to use this in my daily work!

  7. [...] Context-Free Questions for Testing [...]

  8. [...] the context of the test project, your givens, and your mission” Context-Free Questions for Testing, Michael Bolton, [...]

  9. [...] participated actively. The mission was simple and the testers asked context revealing questions (more here) like ‘any specific quality criteria to focus on’, ‘time [...]

  10. [...] a lot more that can be used in any situation, e.g. Context-Free Questions, and Rule of Three: “If you can’t think of three things that might cause your great [...]

  11. [...] value of the product, you should continuously question your testing. Explore about the context (See more on that in Michael Bolton’s blog, DevelopSense). Probe if you are unsure. In Weekend [...]

  12. [...] sure, context-revealing questions (again, Michael Bolton, Developsense) help. Who is the customer of the product? How is this one [...]

  13. Hi Michael,

    Fantastic post! Thanks for sharing.

    Although many could come up with a list of similar questions themselves what you have done is provided an excellent generic template to gather further information about a project, or anything really. Not only that you’ve also grouped them in a format of an approach to revealing information.

    In my workplace we run monthly challenges. One of these was a lateral thinking style exercise, which blended well for these questions. Taking your list as a template I was able to gather enough information quickly by customizing and expanding on it for the challenge, to quickly determine that from the information gathered rapidly (thanks to you) that no sane person would accept the proposed project.

    Thanks for taking the time out to share this with us all.

    Cheers,

    Darren.

  14. [...] questions. (2011 update, since people keep finding this post in particular: Michael Bolton made a blog post last year including a long list of questions. Curious readers should go there for more [...]

  15. Sigge B says:

    Hi Michael,

    Since I have gotten quite fond of your list of questions and I use them every now and then, I wanted to inherit the wisdom into my current work.

    Michael replies: That’s a nice idea, but I can’t make you wise. The best I can do is point the way. You can become wise, but developing wisdom sometimes takes some work.

    Do you think this set of questions can be prioritized if you have a context in which you will use them?

    Michael replies: What do you think? What if I answered Yes? What if I answered “Nope, they can’t prioritized”? How would you react to that?

    If you for example know that in current context you will just have time and buy-in for answers to a handfull of these, how do you choose them carefully enough?

    Time to turn on your Tester’s Thinking Cap, here: I’d like you to consider at least five ways in which you could prioritize the questions, and at least three ways to determine whether you’ve been careful enough.

    If you have developers that know the context and ask them which of these would be the most useful to ask, would it help in such a prioritizing activity?

    Oh! That’s one right there!

    I was going to post this list, among with some more questions in a questionnaire, and get people to choose which (5-10) would be the most useful to ask about our current projects. To get a hunch on what project members care about. Do you have any input on this?

    There’s two!

  16. [...] I got value from the session, I hope the practicing coaches did too.  One thing I hadn’t thought about when looking at the picture was the emotional side.  I thought I was just observing and describing, but when asked that question I was stumped.  I thought about it some more and then realised I had quite a few issues (emotionally) with the picture.  There were items in it that were off centre, and being a touch OCDish where things need to be symmetrical, I found myself not liking it at all!  Maybe this could be put down to critical thinking, or lateral instead of vertical (something I would have liked to explore further given the time).  Either way, it opened up another line of thought… a line that is crucial when testing products.  Some of Michael Bolton’s context-free questions: [...]

Leave a Reply