Working With Test Matrices

A test matrix is one of the most simple, useful, and powerful testing tools.

I'm always surprised when I encounter a testing organization that doesn't use test matrices. A test matrix is simply a spreadsheet that suggests test and captures test results by laying them out in the form of a table. A test matrix can be used for a number of purposes:

The general idea behind using test matrices is to catalog a list of repeatable tests on one axis—for example, different tasks to accomplished, listed in the first column; and a list of items to be tested on the other axis—operating systems on which the tasks are to be done, listed on the top row of the table. Every spreadsheet set up in this way uses two axes, down and across. What you place on each axis is up to you. The important thing is to remember that each sheet represents a collection of possible tests, and each intersection of the axes represents a single test from that collection. Generally, the table is more readable if the category with the longer descriptions forms the rows, but the key to the matrix is the tabular format.

Since each axis on a test matrice is a list, each axis helps to record and to display something useful. In our example, a new tester, a test manager, or anyone else involved in a project should be able to see at a glance the operating systems that the test effort will support, simply by looking across the top row of the table. The list of the tasks formed by the first column helps to identify the tests that are being performed or to suggest other similar kinds of tests. The list also helps the reader to recognize that desired tests might be missing.

Here's a tiny example:

DataWhizard 2.0 OS Compatibility Matrix

 

Windows 95

Windows NT 4.0 SP6
Windows 2000 SP3
Open a normal file
Open a read-only file
Set read-only flag from inside application
Clear read-only flag from inside application

Once you've identified the axes, the next step is to run as many tests as is feasible, generally with a minimum of one test per row and one test per column. The tester performs the test and is recorded at the intersection of a rown and a column; some common notation such as "Pass" or "P" denotes success; "F" might show a failure. A blank cell indicates that no test has been done. Other categories, such as "W" for "Warn", "N" for "Note", a cell reference, or even a comment within the cell itself indicates that the test has passed, but that something worthy of further investigation has been found.

 

Windows 95

Windows NT 4.0 SP6
Windows 2000 SP3
Open a normal file
P
Open a read-only file
P
P
Set read-only flag from inside application
P
Clear read-only flag from inside application
P

In the example above, highlighting in each cell helps us to see at a glance how much of the matrix has been covered. This is a tiny matrix, so coverage has been good. Complete testing of an application is impossible. Instead, we have to choose some subset of all possible tests. The matrix helps us to identify better subsets: the more tests per row, the greater the confidence that a given task works correctly on all of the listed platforms; the more tests per column, the greater the confidence that a given platform supports all of the different tasks. When the table is big—lots of rows and lots of columns—it's not possible to run all the tests one might want to perform, and thus it's not possible to cover the entire table. However, it becomes easy to see at a glance the extent of coverage that has been achieved, simply by looking down the rows and across the columns.

Test matrices can also help to localize a problem quickly. Suppose that our program provides a way to lock a file from within the application itself, and our first few tests show these results:

 

Windows 95

Windows NT 4.0 SP6
Windows 2000 SP3
Open a normal file
F
Open a read-only file
P
P
Set read-only flag from inside application
Clear read-only flag from inside application

Testing in the same axis as the failure will help us to identify quickly whether the problem is platform specific, or is a general problem with the feature. Let's do two more tests.

 

Windows 95

Windows NT 4.0 SP6
Windows 2000 SP3
Open a normal file
F
P
Open a read-only file
F
P
P
Set read-only flag from inside application
Clear read-only flag from inside application

Right away we can see that the problem is not a general one; it seems to be a failure on Windows 95 only; moreover, it seems that there's a general problem with file handling on that platform. By contrast, we might a result like this:

 

Windows 95

Windows NT 4.0 SP6
Windows 2000 SP3
Open a normal file
F
P
Open a read-only file
P
P
P
Set read-only flag from inside application
Clear read-only flag from inside application

This result would suggest that the problem is specific to Windows 95, and to that particular function only. A third possible result might look like this:

 

Windows 95

Windows NT 4.0 SP6
Windows 2000 SP3
Open a normal file
F
F
Open a read-only file
P
P
P
Set read-only flag from inside application
Clear read-only flag from inside application

This would suggest very strongly that opening read-only files works perfectly on all platforms, but there's some more general glitch associated with opening normal files. From this we could extrapolate (although we could not be certain without a test) that the same function is broken on Windows 2000.

From these examples, it should be clear that test matrices are useful for platform and compatibility testing, but that's not the whole story by any means. Test matrices can be used for any pair of attributes or criteria. Here are some examples for printer testing:

Here are some examples for testing a set of Web pages from an online catalog:

Test matrices are also wonderful and highly reusable tools for preserving a test case library. A workbook of similar test matrix sheets can represent a test category; a folder of workbooks could represent a complete test suite for a product. Alternatively, a single workbook could contain the same matrix for each test release of a particular piece of software, in which each filled-out matrix represents a set of tests completed over a test cycle. At the end of one test cycle, simply select a matrix, copy the sheet, clear the pass-or-fail data from the table, and start testing again immediately. Using the third dimension of your spreadsheet software, a summary page can help to ensure that different tests on a single matrix have been performed over the entire development cycle, which helps to assure excellent coverage.

Inevitably some defects will be found for which there are not already tests on a matrix. As these defects are found, tests for them can be added easily to existing matrices. If a product defect is discovered after release, someone is going to want to know whether the test coverage was reasonable or not. The test matrix provides an excellent means of identifying specific tests that were performed (or not), and in the event of an investigation or legal action, can show that the test effort was thorough and reasonable.

With this essay, I've included a sample test matrix, the purpose of which is to provide a set of test cases for filename and pathname handling under Windows 9x and the Windows NT family. The document is in Microsoft Excel format. I've included a number of test requirements to test whether the program accepts valid filenames and rejects invalid ones. In this sheet, I've used Excel's conditional formatting feature to help identify tests that have passed and failed; a P (for Pass) in a cell displays as black text on a green background, an F (for Fail) displays as white text on red, and a W (for Warn) displays as black text on orange. In order to create new test categories, create a new sheet, copy the existing sheet to the new sheet in the workbook, and change the labels in the rows and columns.

You can open the sample test matrix by clicking here.


This essay is copyright ©2003 Michael Bolton. If you liked it, please let me know. If you didn't, please let me know how I might improve upon it. If you'd like to be notified when I've posted another essay that you might enjoy, please click here to be put on my mailing list. If your reply-to address isn't altered to prevent spam, you don't have to put anything in the body of the message.

You are welcome to reprint this article for your own use or for your company's use, as long as you follow the instructions here (http://www.developsense.com/reprints.html).

Best of all, if you (or your company, or your manager, or your employee) need counselling or instruction in this area, I can help with engaging and informative courses on quality assurance and software testing in plain English that can save your company lots of time and money. Contact me for details. Thanks!

Back to the testing page…

Back to the DevelopSense home page…

Last revised: 28 January 2003
Filename: testing/TestMatrices.html