Working With Test Matrices

A test matrix is a simple, useful, and powerful testing tool.

I'm always surprised when I encounter a testing organization that doesn't use test matrices. A test matrix is a pretty simple concept: it allows you to record test ideas and captures test results, and visualize coverage by laying out information in the form of a table. A test matrix can be used for a number of purposes:

• to record a consistent set of tests;
• to document a desired level of test coverage based on an intersection of two criteria and, when filled out, to be used as evidence that the desired amount of testing has been done;
• as a quick visual indication of how much testing has been done on those criteria;
• to help to identify and refine the specific environments in which a defect occurs;
• when it is completely filled out, as a checklist to assure that some exhaustive task has been completed.

The general idea behind using test matrices is to catalog a list of test ideas on one axis—for example, different tasks to accomplished, listed in the first column; and a list of items to be tested on the other axis—operating systems on which the tasks are to be done, listed on the top row of the table. Every spreadsheet set up in this way uses two axes, down and across. What you place on each axis is up to you. The important thing is to remember that each sheet represents a collection of possible tests, and each intersection of the axes represents a single test from that collection. Generally, the table is more readable if the category with the longer descriptions forms the rows, but the key to the matrix is the tabular format.

Since each axis on a test matrice is a list, each axis helps to record and to display something useful. In our example, a new tester, a test manager, or anyone else involved in a project should be able to see at a glance the operating systems that the test effort will support, simply by looking across the top row of the table. The list of the tasks formed by the first column helps to identify the tests that are being performed or to suggest other similar kinds of tests. The list also helps the reader to recognize that desired tests might be missing.

Here's a tiny example:

DataWhizard 2.0 OS Compatibility Matrix

Once you've identified the axes, the next step is to run as many tests as is feasible, generally with a minimum of one test per row and one test per column. The tester performs the test and is recorded at the intersection of a rown and a column; some common notation such as "Pass" or "P" denotes success; "F" might show a failure. A blank cell indicates that no test has been done. Other categories, such as "W" for "Warn", "N" for "Note", a cell reference, or even a comment within the cell itself indicates that the test has passed, but that something worthy of further investigation has been found.

In the example above, highlighting in each cell helps us to see at a glance how much of the matrix has been covered. This is a tiny matrix, so coverage has been good. Complete testing of an application is impossible. Instead, we have to choose some subset of all possible tests. The matrix helps us to identify better subsets: the more tests per row, the greater the confidence that a given task works correctly on all of the listed platforms; the more tests per column, the greater the confidence that a given platform supports all of the different tasks. When the table is big—lots of rows and lots of columns—it's not possible to run all the tests one might want to perform, and thus it's not possible to cover the entire table. However, it becomes easy to see at a glance the extent of coverage that has been achieved, simply by looking down the rows and across the columns.

Test matrices can also help to localize a problem quickly. Suppose that our program provides a way to lock a file from within the application itself, and our first few tests show these results:

Testing in the same axis as the failure will help us to identify quickly whether the problem is platform specific, or is a general problem with the feature. Let's do two more tests.

Right away we can see that the problem is not a general one; it seems to be a failure on Windows 95 only; moreover, it seems that there's a general problem with file handling on that platform. By contrast, we might a result like this:

This result would suggest that the problem is specific to Windows 95, and to that particular function only. A third possible result might look like this:

This would suggest very strongly that opening read-only files works perfectly on all platforms, but there's some more general glitch associated with opening normal files. From this we could extrapolate (although we could not be certain without a test) that the same function is broken on Windows 2000.

From these examples, it should be clear that test matrices are useful for platform and compatibility testing, but that's not the whole story by any means. Test matrices can be used for any pair of attributes or criteria. Here are some examples for printer testing:

• Operating systems listed across the top; printers listed down the side.
• Specific printers across the top; driver revisions down the side.
• Driver revisions across the top; specific sample documents down the side.

Here are some examples for testing a set of Web pages from an online catalog:

• Pages tested across the top; test categories (link checks; proof-reading of the content; verification that colour choices are available; inventory matching tests; product manager review; and so on) down the side.
• Date of tests across the top; specific tests down the side.
• Browsers listed across the top; specific browser compatibility checks down the side.

Test matrices are also wonderful and highly reusable tools for preserving a test case library. A workbook of similar test matrix sheets can represent a test category; a folder of workbooks could represent a complete test suite for a product. Alternatively, a single workbook could contain the same matrix for each test release of a particular piece of software, in which each filled-out matrix represents a set of tests completed over a test cycle. At the end of one test cycle, simply select a matrix, copy the sheet, clear the pass-or-fail data from the table, and start testing again immediately. Using the third dimension of your spreadsheet software, a summary page can help to ensure that different tests on a single matrix have been performed over the entire development cycle, which helps to assure excellent coverage.

Inevitably some defects will be found for which there are not already tests on a matrix. As these defects are found, tests for them can be added easily to existing matrices. If a product defect is discovered after release, someone is going to want to know whether the test coverage was reasonable or not. The test matrix provides an excellent means of identifying specific tests that were performed (or not), and in the event of an investigation or legal action, can show that the test effort was thorough and reasonable.

With this essay, I've included a sample test matrix, the purpose of which is to provide a set of test cases for filename and pathname handling under Windows 9x and the Windows NT family. The document is in Microsoft Excel format. I've included a number of test requirements to test whether the program accepts valid filenames and rejects invalid ones. In this sheet, I've used Excel's conditional formatting feature to help identify tests that have passed and failed; a P (for Pass) in a cell displays as black text on a green background, an F (for Fail) displays as white text on red, and a W (for Warn) displays as black text on orange. In order to create new test categories, create a new sheet, copy the existing sheet to the new sheet in the workbook, and change the labels in the rows and columns.

You can open the sample test matrix by clicking here.

This essay is copyright ©2003 Michael Bolton. If you liked it, please let me know. If you didn't, please let me know how I might improve upon it. If you'd like to be notified when I've posted another essay that you might enjoy, please click here to be put on my mailing list. If your reply-to address isn't altered to prevent spam, you don't have to put anything in the body of the message.

You are welcome to reprint this article for your own use or for your company's use, as long as you follow the instructions here (http://www.developsense.com/reprints.html).

Best of all, if you (or your company, or your manager, or your employee) need counselling or instruction in this area, I can help with engaging and informative courses on quality assurance and software testing in plain English that can save your company lots of time and money. Contact me for details. Thanks!

Back to the testing page…