Monday, July 06, 2009
Testability
On Twitter, Kindly Reader@jrl7 (in real life, John Lambert at Microsoft) asks "Is there an example of testability that doesn't involve improving ability to automate? (improved specs?)".
Yup. If testing is questioning a product in order to evaluate it, then testability is anything that makes it easier to question or evaluate that product. So testability is anything that makes the program faster or easier to test on some level. Anything that slows down testing or makes it harder reduces testability, which gives bugs an opportunity to hide for longer, or to conceal themselves better.
Testability is enabled, extended, enhanced, accelerated or intensified by initiating or improving on some of the things below, either on their own or in combination with others. That suggests that testability ideas are media, in the McLuhan sense. Thus each idea comes with a cautionary note: As McLuhan pointed out, when a medium is stretched beyond its capacities, it reverses into the opposite of the intended effect. So the following ideas are heuristic, which means that any of these things could help, but might fail to help or make things worse if considered or applied automatically or unwisely.
In accordance with Joel Spolsky's Law of Leaky Abstractions, I've classified them into a few leaky categories.
Product Elements
Things that make a program faster or easier to use tend to make it faster or easier to test, especially when testing at the user interface level. Any time you see a usability problem in an application, you may be seeing a testability problem too.
An oracle is a principle or mechanism by which we might recognize a problem. Information about how the system is intended to work is a source of oracles.
Postscript: Bret Pettichord contacted me with a link to this paper, at the end of which he surveys several different definitions of testability.
Yup. If testing is questioning a product in order to evaluate it, then testability is anything that makes it easier to question or evaluate that product. So testability is anything that makes the program faster or easier to test on some level. Anything that slows down testing or makes it harder reduces testability, which gives bugs an opportunity to hide for longer, or to conceal themselves better.
Testability is enabled, extended, enhanced, accelerated or intensified by initiating or improving on some of the things below, either on their own or in combination with others. That suggests that testability ideas are media, in the McLuhan sense. Thus each idea comes with a cautionary note: As McLuhan pointed out, when a medium is stretched beyond its capacities, it reverses into the opposite of the intended effect. So the following ideas are heuristic, which means that any of these things could help, but might fail to help or make things worse if considered or applied automatically or unwisely.
In accordance with Joel Spolsky's Law of Leaky Abstractions, I've classified them into a few leaky categories.
Product Elements
- Scriptable interfaces to the product, so that we can drive it more easily with automation.
- Logging of inputs, outputs, or activities within the program. Structure helps; time stamps help; a variety of levels of detail help.
- Real-time monitoring of the internals of the application via another window, a debug port, or output over the network—anything like that. Printers used to provide "real-time" updates.
- Internal consistency checks within the program. For example, if network connectivity is lost and functions depend on it, the application can let us know that instead of simply failing a function.
- Overall simplicity and modularity of the application, especially in the separation of user interface code from program code. This one needs to be balanced with the knowledge that simpler modules tends to mean more numerous modules, which leads to growth in the number of interfaces, which in terms may mean more interface problems. There are no free lunches, but maybe there are less expensive lunches. Note also that complexity is not an attribute of a program; it's a relationship between the program and the person observing it. What looks horribly complex to a tester might look simple and elegant to a programmer; what looks frightfully complex to the programmer might look straightforward to a marketer.
- Use of resource files for localization support, rather than hard-coding of location-dependent strings, dialogs, currencies, time formats, and the like.
- Readable and maintainable code, thanks to pairing or other forms of technical review, and to refactoring.
- Restraint in platform support. Generally, the fewer computers or operating systems or browsers or application framework versions or third-party library versions that we have to support, the easier the testing job will be.
- Restraint in feature support. Very generally, and all other things being equal, the more features, the longer it takes to test a program.
- Finally, but perhaps most importantly, an application that’s in good shape before the testers get to it, thanks at least in part to diligent testing by programmers, based on unit tests or (perhaps better yet) a test-first approach such as test- or behaviour-driven development. If we're investigating and reporting bugs that we find or (worse) spending time trying to work around blocking bugs, we're compromised in our ability to obtain test coverage.
Things that make a program faster or easier to use tend to make it faster or easier to test, especially when testing at the user interface level. Any time you see a usability problem in an application, you may be seeing a testability problem too.
- Ease of learning—that is, the extent to which the application allows the user to achieve expertise in its use.
- Ease of use—that is, the extent to which the application supports the user in complete a task quickly and reliably.
- Affordance—that is, the extent to which the application advertises its available features and functions to the user.
- Clearer error and/or exception messages. This could include unique identifiers to help us to target specific points in the code, a notion of what the problem was, or which file was not found, thank you.
An oracle is a principle or mechanism by which we might recognize a problem. Information about how the system is intended to work is a source of oracles.
- Better documentation. Note that, for testability purposes, "better" documentation doesn't necessarily mean "more complete" or "more elaborate" or "thicker". It might mean "more concise" or "more targeted towards testing" or "more diagrams and illustrations that allow us to visualize how things happen".
- Clear reference to relevant standards, and information as to when and why those standards might be relevant or not.
- "Live oracles"—people who can help us in determining whether we're seeing appropriate behaviour from the application, when that's the most efficient mode of knowledge transfer. Programmers, business analysts, product owners, technical support people, end-users, more experienced testers—all are candidates for being live oracles.
- Programs that give us a comparable result for some feature, function, or state within our product. Such programs may have been created within our organization or outside; they may have been created for testing or for some other purpose; they may be products that are competitors to our own.
- Availability of old versions is a special case of the comparable program heuristic. Having an old version of a product around for comparison may help to make the current version of our program easier to test.
- Access to existing ad hoc (in the sense of “purpose-built”, not sloppy) test tools, and help in creating them where needed. Note that a test tool is not merely a program that probes the application under test. It might be a data-generation tool, an oracle program that supplies us with a comparable result, or a program that allows us to set up a new platform with minimal fuss.
- Availability of test environments. In big organizations and on big projects, I've never worked with a test organization that believed it had sufficient isolated platforms for testing.
- More rapid building and integration of the product, including automated smoke tests that help us to determine if the program has been built correctly.
- Simpler or accelerated setup of the application.
- The ability to change settings or configuration of the application on the fly.
- Access to source control logs help us to identify where a newly-discovered problem or a regression might have crept into the code.
- Availability of modules separately for early testing, especially at the integration level.
- Information about what has already been tested, so we can leverage the information or avoid repeating someone else's efforts.
- Access to source code for those of us who can read and interpret it.
- Proximity of testers, programmers, and other members of the project community.
- Project community support for testing. Testing is often made much longer and more complicated by constraints imposed by forces that are external to the project. IT managers, for good reasons of their own, are often reluctant to grant autonomy to test groups.
- Tester skill is inter-related with testability. It might not make sense to put a scriptable interface into your product if you're not going to use it yourself and you don't anticipate your testers having the skill to use it either. That might sounds undesirable, yet over the years much great software has been produced without test automation assistance. Still, it's usually worthwhile to have at least some members of the test team skilled in automation, and to give them a program for which those skills are useful.
- Stability, or an absence of turbulence, both in the product and in the team that's producing it. Things that are changing all the time are usually harder to test.
Postscript: Bret Pettichord contacted me with a link to this paper, at the end of which he surveys several different definitions of testability.
# posted by Michael Bolton http://www.developsense.com @ Monday, July 06, 2009 
Comments:
<< Home
I agree with everything you've posted, but I'd like to make up a word and add it to the list: debugability. Not just monitoring over a debug port, but things to help interactively debug running code. Some of these things apply to compiled binaries much more than scripts or web interfaces.
- non-obfuscated code to test
- non-optimized, debug binaries instead of release ones (where applicable)
- whatever needed debug symbols (if any) are readily available
- debugging tools to help diagnose typical problems - I mean actually within a debugger when available
- etc.
- non-obfuscated code to test
- non-optimized, debug binaries instead of release ones (where applicable)
- whatever needed debug symbols (if any) are readily available
- debugging tools to help diagnose typical problems - I mean actually within a debugger when available
- etc.
"Is there an example of testability that doesn't involve improving ability to automate?"
I think your response is dead on.
For fun, I did some research on McLuhan where his illustration of marketing is appropriate to test automation.
A nose for news and a stomach for whiskey: McLuhan analyzes an ad for Time Magazine in which he likens a reporter depicted as a romantic character from a Hemingway novel and asks "Why is it [his] plangent duty to achieve cirrhosis of the liver?"
Just as satirically, in the testing sense, why is it Automation Vendors dismal duty to achieve widespread disruption of sentient testing?
I think your response is dead on.
For fun, I did some research on McLuhan where his illustration of marketing is appropriate to test automation.
A nose for news and a stomach for whiskey: McLuhan analyzes an ad for Time Magazine in which he likens a reporter depicted as a romantic character from a Hemingway novel and asks "Why is it [his] plangent duty to achieve cirrhosis of the liver?"
Just as satirically, in the testing sense, why is it Automation Vendors dismal duty to achieve widespread disruption of sentient testing?
In the third point of Product Elements there is debugging mentioned:
- Real-time monitoring of the internals of the application via another window, a debug port, or output over the network—anything like that. Printers used to provide "real-time" updates.
This point basically reminded myself on my reply to Brian Marick's "An Alternative To Business Facing TDD" last year:
http://www.exampler.com/blog/2008/03/23/an-alternative-to-business-facing-tdd/
- Real-time monitoring of the internals of the application via another window, a debug port, or output over the network—anything like that. Printers used to provide "real-time" updates.
This point basically reminded myself on my reply to Brian Marick's "An Alternative To Business Facing TDD" last year:
http://www.exampler.com/blog/2008/03/23/an-alternative-to-business-facing-tdd/
@offshore...
You're welcome, but I don't see Markus' question. I'd be happy to answer it.
If you're referring to Joseph Ours' question ("Why is it Automation Vendors dismal duty to achieve widespread disruption of sentient testing?"), my answer is that it's not intentional; they know not what they do. Specifically, they're confusing testing and checking.
---Michael B.
You're welcome, but I don't see Markus' question. I'd be happy to answer it.
If you're referring to Joseph Ours' question ("Why is it Automation Vendors dismal duty to achieve widespread disruption of sentient testing?"), my answer is that it's not intentional; they know not what they do. Specifically, they're confusing testing and checking.
---Michael B.
I am not sure as to wether I can accept your definition of testability. "Tesability" to me is the feasibilty of proving the validity (or otherwise) of something. This means lack of ambiguity is a requirement for testability, as is some way of measuring or quantifying a result. For example, requirements should be Specific
Measurable,Attainable, Realistic and Timely in which case they are testable. Being easy to test does not come into it.
Measurable,Attainable, Realistic and Timely in which case they are testable. Being easy to test does not come into it.
# posted by 
: Sunday, November 08, 2009 9:51:00 PM
: Sunday, November 08, 2009 9:51:00 PM << Home
Post a Comment