I read Whittaker’s blog posts about a Tester Dashboard, again not until I read this post did I realize that was also a Bach idea. It seems weird that someone who has a PhD. and had to do a dissertation wouldn’t naturally reference sources of ideas or inspiration for ideas. (Steve Jobs was known for stealing ideas and making them his own, so it’s not unheard of.)

Looking back you can see the lack of references in most of his books; perhaps the market he is targeting doesnt care? That doesnt make it right mind you. Acknowledging earlier work doesn’t seem like it would take that much extra effort and pays respect to those who came before you.

I heard that again, recently, as a comment on a set of lecture notes I’m creating for a course on Test Design. (And again, in a related context, today–which motivated me to finally publish a comment.)

My course (on video) starts with a demonstration of a feature tour and then gives an inventory of tours that I’ve learned from Bach, Bolton, Kelly and Hendrickson over the years. Why, the commentator asks, do I not explain which tours are better and which are worse.

And indeed, there are people who believe that Research Has Been Done that answers this question to some degree, and that More Research Could Be Done to explore it more thoroughly. An enterprising researcher could probably sell this idea to a corporate research department or even a scientific-research funding agency, on the grounds that we might be able to establish some Best Practices for Touring. I’m sure this can be dressed up and made to sound impressive.

However, I think this is a shallow and unproductive line of work. I have never heard it from anyone who (in my view) knows much about touring. Or to put it differently, anyone who suggests it is telling me that they don’t understand what tours are or why we do them.

A tour is a directed search through the program. Find all the capabilities. Find all the claims about the product. Find all the variables. Find all the intended benefits. Find all the ways to get from A to B. Find all the X. Or maybe not ALL, but find a bunch.

This helps the tester achieve a few things:

(1) it creates an inventory of a class of attributes of the product under test. Later, the tester can work through the inventory, testing each one to some intended level of depth. This is what “coverage”-oriented testing is about. You can test N% of the program’s statements, or N% of the program’s features, or N% of the claims made for the product in its documentation–if you can list it, you can test it and check it off the list.

(2) It familiarizes the tester with this aspect of the product. Testing is about discovering quality-related information about the product. An important part of the process of discovery is learning what is in the product, how people can / will use it, and how it works. Tours give us different angles on that multidimensional learning problem.

(3) It provides a way for the tester to explain to someone else what she has studied in the product so far, what types of things she has learned and what basics haven’t yet been explored. In a field with few trustworthy metrics, this gives us a useful basis for reporting progress, especially progress early in the testing effort.

So which tour is better?

From a test-coverage perspective, I think that depends a lot on contract, regulation, and risk.

(i) To the extent that you have to know (and have to be able to tell people in writing) that all the X’s have been tested and all the X’s work, you need to know what all the X’s are and how to find them in the program. That calls for an X-tour. Which tour is better? The one you need for this product. That probably varies from product to product, no?

(ii) Some programmers are more likely to make X-bugs than Y-bugs. Some programmers are sloppy about initializing variables. Some are sloppy about boundary conditions. Some are sloppy about thread interactions. Some are good coders but they design user interactions that are too confusing. If I’m testing Joe X’s code, I want to look for X-bugs. If Joe blows boundaries, I want to do a variable tour, to find all the variables so I can test all the boundaries. But if Joe’s problem is incomprehensibility, I want to do a benefit tour, to see what benefits people _should_ get from the program and how hard/confusing it is for users to actually get them. Which tour is better? The answer is “yes”.

(2) From a tester-learning perspective, people learn differently from each other. If I set 10 people with the task of learning what’s in a program, how it can be used, and how it works, those people would look for different types of information. They would be confused by different things. They would each find some things more interesting than others. They would already know some things that their colleagues didn’t. Which tour is better? The tour that helps you learn something you’re trying to learn today. Tomorrow, the best tour will be something different.

Testing is an infinite task. Define “distinct tests” this way: two tests are distinct if each can expose at least one bug that the other would miss. For a non-trivial program, there is an infinite number of distinct potential tests. The central problem of test design is boiling down this infinite set to a (relative to infinity) tiny collection of tests that we will actually use. Each test technique highlights a different subset of this infinity. In effect, each test technique represents a different sampling strategy from the space of possible tests.

Tours help the tester gain insight into the multidimensional nature of this complex, infinite space. They help us imagine, envision, and as we gain experience on the tour, prioritize the different sampling strategies we could use when we do more thorough, more intense testing after finishing the tours.

So which tour is best? The ones that give the testers more insight and that achieve a greater stretch of the testers’ imagination. For this, some tours will work better for me, others for you.

The best tour isn’t necessarily the one that finds the most bugs. Or covers the most statements (or take your pick of what coverage attribute) of the product. The best tour is the one that helps the individual human tester learn something new and useful. (That’s why we call it exploration. New and useful knowledge.) And that depends on what you already know, on what risks and complexities characterize this program, and what the priorities are in this project.

A tour that is useful to you might be worthless to me. But that doesn’t stop it from being useful for you.

Rather than looking for a few “best” tours, I think it would be more interesting to develop guidance on how to do tour X given that you want to. (Do you know how to do a tour of uses of the program that trigger multithreaded operations in the system? Would it be interesting to know how?)

Rather than looking fro a few “best” tours, I think it would be more interesting to develop a more diverse collection of tours that we can do, with more insight into what each can teach us.

Rather than seeking to objectify and quantify tours, I think we should embrace their subjectivity and the qualitative nature of the benefits they provide.

Pseudo-academics and best-practicers trivialize enough aspects of testing. They should leave this one alone.

Michael replies: Thanks for the comment, Cem.

But you can *always* include a phrase, wording or hint, that this is something you’ve been inspired to by someone else, and doing so shows you’ve got manners and integrity.Speaking of Microsoft in particular, it appears that most of what they do is invented by others: don’t we all ‘know’ that windows was an apple invention ? (I also remember something called ‘Gem’ that was out prior to windows 3.1). Word and Wordperfect (wordstar).. etc etc – sometimes I think of what they actually did themselves, apart from DOS, of course – no wait – wasn’t there something called unix at the time ?!? Early DOS shows many similarities..

I guess we in our business reinvent the wheel all over time and again. It’s difficult to keep track. And ultimately, who can write a book or a lecture or class or anything today without admitting to be, (quote Newton:) “standing on the shoulders of Giants” ?

Not too long ago I read a most creditable story that someone patented the wheel recently in Australia (http://digg.com/d13xYh – believable ?). I guess, that for those without integrity and manners – honour of invention is up for grasps.. unfortunately..

Everything comes at a cost, even plagiarism. If you want it in a word, plagiarism is stealing. The plagiarist robs the unwitting collaborator of their intellectual property, and profits from their work. There’s a reason we have bibliographies and works cited pages and foot notes and reference lists; it’s not up to the reader to determine where the author’s work came from. Think of it as a paper trail – I cite this blog, the blog cites other authors, those authors cite earlier works and so on. I’m not suggesting that the author has to cite the original source of the idea because that isn’t always certain; I’m merely stated that you shouldn’t present someone else’s work as your own, or fail to mention where you got the idea.

Until I read this blog, I could only assume that Touring and low-tech dashboards were Microsoft inventions.

At any rate, I believe that honesty and integrity are very important, and should never be compromised for the sake of perpetuating a useful idea. Just my 2 cents.

However, I will say, sometimes it can be difficult to get the right source. If Scott Barber introduces my to an idea on performance testing, but he cites some other work, then I often can’t remember who he cited. I’ll cite Scott.

I’d leave it up to the reader to work backwards from there. That is, it’s the reader’s job – if they care – to go open Scott’s paper and see who he cited. I think that speaks to Arjan’s point that most testers only care about the ideas. If I were in an academic setting, I’m sure I’d hold myself to a higher level of historical research, but I suppose I’m ok with citing the source where /I/ got the information.

It’s also difficult for topics I’ve been talking about or have been exposed to and using for a long time. When I talk about ET, it’s hard to cite. Not because I don’t know who did what (most of the time I think I know), but because it /feels/ like mine because I use it so much. Because of that, I often myself using blanket statements like “when working with James…” or “some time ago Kaner…”. It can be difficult to do it all the time.

That said, if you know it’s not your idea and you know where it came from, give the credit. Even if it’s just a footnote or a link.