Blog Posts for the ‘Very Short Blog Posts’ Category

Very Short Blog Posts (35): Make Things Visible

Tuesday, April 24th, 2018

I hear a lot from testers who discover problems late in development, and who get grief for bringing them up.

On one level, the complaints are baseless, like holding an investigate journalist responsible for a corrupt government. On the other hand, there’s a way for testers to anticipate bad news and reduce the surprises. Try producing a product coverage outline and a risk list.

A product coverage outline is an artifact (a mind map, or list, or table) that identifies factors, dimensions, or elements of a product that might be relevant to testing it. Those factors might include the product’s structure, the functions it performs, the data it processes, the interfaces it provides, the platforms upon which it depends, the operations that people might perform with it, and the way the product is affected by time. (Look here for more detail.) Sketches or diagrams can help too.

As you learn more through deeper testing, add branches to the map, or create more detailed maps of particular areas. Highlight areas that have been tested so far. Use colour to indicate the testing effort that has been applied to each area—and where coverage is shallower.

A risk list is a list of bad things that might happen: Some person(s) will experience a problem with respect to something desirable that can be detected in some set of conditions because of a vulnerability in the system. Generate ideas on that, rank them, and list them.

At the beginning of the project or early as possible, post your coverage outline and risk list in places where people will see and read them. Update it daily. Invite questions and conversations. This can help you change “why didn’t you find that bug?” to “why didn’t we find that bug?”

Posted in Accountability, Coverage, Risk, Testing Story, Very Short Blog Posts | 3 Comments »

Very Short Blog Posts (33): Insufficient Information and Insufficient Time

Monday, March 19th, 2018

Here’s a question I get from testers quite a lot:

“What do I do when the developers give me something to test with insufficient information and time to test it?”

Here’s my quick answer: test it.

Here’s my practical answer: test it with whatever time and information you have available. (Testing is evaluating a product by learning about it through exploration and experimentation.) When your time is up, provide a report on what you have learned about the product, with particular focus on any problems you have found.

Identify the important risks and product factors of which you are aware, and which you have covered. (A product factor, or product element, is something that can be examined during a test, or that could influence the outcome of a test.) Identify important risks and product factors that you’re aware of and that you haven’t covered. Note the time and sources of information that you had available to you.

If part of the product or feature is obscure to you because you perceive that you have had insufficient information or time or testabilty to learn about it, include that in your report.

(I’ll provide a deep answer to the question eventually, too.)

Related posts:

How Is the Testing Going?
Testability
Testing Problems Are Test Results

Posted in Reporting, Testing Story, Very Short Blog Posts | 2 Comments »

Very Short Blog Posts (31): Ambiguous or Incomplete Requirements

Monday, December 19th, 2016

This question came up the other day in a public forum, as it does every now and again: “Should you test against ambiguous/incomplete requirements?”

My answer is Yes, you should. In fact, you must, because all requirements documents are to some degree ambiguous or incomplete. And in fact, all requirements are to some degree ambiguous and incomplete.

And that is an important reason why we test: to help discover how the product is inconsistent with people’s current requirements, even though it might be consistent with requirements that they may have specified—ambiguously or incompletely—at some point in the past.

In other words: we test not only to compare the product to documented requirements, but to discover and help refine requirements that may otherwise be ambiguous, unclear, inconsistent, out of date, unrecognized, or emergent.

Posted in Requirements, Very Short Blog Posts | 2 Comments »

Very Short Blog Posts (30): Checking and Measuring Quality

Monday, November 14th, 2016

This is an expansion of some recent tweets.

Do automated tests (in the RST namespace, checks) measure the quality of your product, as people sometimes suggest?

First, the check is automated; the test is not. You are performing a test, and you use a check—or many checks—inside the test. The machinery may press the buttons and return a bit, but that’s not the test. For it to be a test, you must prepare the check to cover some condition and alert you to a potential problem; and after the check, you must evaluate the outcome and learn something from it.

The check doesn’t measure. In the same way, a ruler doesn’t measure anything. The ruler doesn’t know about measuring. You measure, and the ruler provides a scale by which you measure. The Mars rovers do not explore. The Mars rovers don’t even know they’re on Mars. Humans explore, and the Mars rovers are ingeniously crafted tools that extend our capabilities to explore.

So the checks measure neither the quality of the product nor your understanding of it. You measure those things—and the checks are like tiny rulers. They’re tools by which you operate the product and compare specific facts about it to your understanding of it.

Peter Houghton, whom I greatly admire, prompted me to think about this issue. Thanks to him for the inspiration. Read his blog.

Posted in Measurement, Quality, Very Short Blog Posts | 1 Comment »

Very Short Blog Posts (29): Defective Detection Effectiveness

Tuesday, July 14th, 2015

Managers are responsible for hiring testers, for training them, and for removing any obstacles that make testing harder or slower. Managers are also responsible for hiring developers and designers, and providing appropriate training when it’s needed. If there are problems in development, managers are responsible for helping the developers to address them.

Managers are also responsible for the scope of the product, the budget, the staffing, and the schedule. As such, they’re responsible for maintaining awareness of the product, of product development, and anything that threatens the value of either of these. Finally, managers are responsible for the release decision: is this product ready for deployment or release into the market?

Misbegotten metrics like “Defect Detection Percentage” (I won’t dignify references to them with a link) continue to plague the software development world, and are sometimes used to evaluate “testing effectiveness”. But since it’s management’s job to understand the product and to decide when the product ships, a too-low defect detection percentage suggests the possibility of development or testing problems, unaware management, or a rash shipping decision. Testers don’t decide whether or when to ship the product; that’s management’s responsibility. In other words: Defect Detection Percentage—to the degree that it has any validity at all—measures management effectiveness.

Posted in Measurement, Very Short Blog Posts | 8 Comments »

Very Short Blog Posts (28): Users vs. Use Cases

Thursday, May 7th, 2015

As a tester, you’ve probably seen use cases, and they’ve probably informed some of the choices you make about how to test your product or service. (Maybe you’ve based test cases on use cases. I don’t find test cases a very helpful way of framing testing work, but that’s a topic for another post—or for further reading; see page 31. But I digress.)

Have you ever noticed that people represented in use cases are kind of… unusual?

They’re very careful, methodical, and well trained, so they don’t seem to make mistakes, get confused, change their minds, or backtrack in the middle of a task. They never seem to be under pressure from the boss, so they don’t rush through a task. They’re never working on two things at once, and they’re never interrupted. Their phones don’t ring, they don’t check Twitter, and they don’t answer instant messages, so they don’t get distracted, forget important details, or do things out of order. They don’t run into problems in the middle of a task, so they don’t take novel or surprising approaches to get around the problems. They don’t get impatient or frustrated. In other words: they don’t behave like real people in real situations.

So, in addition to use cases, you might also want to imagine and consider misuse cases, abuse cases, obtuse cases, abstruse cases, diffuse cases, confuse cases, and loose cases; and then act on them, as real people would. You can do that—and helpful and powerful as they might be, your automated checks won’t.

Posted in Acceptance Tests, Scenario Testing, Test Framing, Very Short Blog Posts | 11 Comments »

Very Short Blog Posts (27): Saving Time

Wednesday, April 29th, 2015

Instead of studying and learning from every bug, you can save a lot of time by counting and aggregating bug reports.

That’s a good thing in its way, because if you don’t study and learn from every bug, you’ll need all the time you can get to deal with problems that seem to keep happening over and over again.

Posted in Time, Very Short Blog Posts | 1 Comment »

Very Short Blog Posts (26): You Don’t Need Acceptance Criteria to Test

Tuesday, February 24th, 2015

You do not need acceptance criteria to test.

Reporters do not need acceptance criteria to investigate and report stories; scientists do not need acceptance criteria to study and learn about things; and you do not need acceptance criteria to explore something, to experiment with it, to learn about it, or to provide a description of it.

You could use explicit acceptance criteria as a focusing heuristic, to help direct your attention toward specific things that matter to your clients; that’s fine. You might choose to use explicit acceptance criteria as claims, oracles that help you to recognize a problem that happens as you test; that’s fine too. But there are many other ways to identify problems; quality criteria may be tacit, not explicit; and you may discover many problems that explicit acceptance criteria don’t cover.

You don’t need acceptance criteria to decide whether something is acceptable or unacceptable. As a tester you don’t have decision-making authority over acceptability anyway. You might use acceptance criteria to inform your testing, and to identify threats to the value of the product. But you don’t need acceptance criteria to test.

Posted in Acceptance Tests, Coverage, Oracles, Test Framing, Tester Skill, Very Short Blog Posts | 4 Comments »

Very Short Blog Posts (25): Testers Don’t Break the Software

Tuesday, February 17th, 2015

Plenty of testers claim that they break the software. They don’t really do that, of course. Software doesn’t break; it simply does what it has been designed and coded to do, for better or for worse. Testers investigate systems, looking at what the system does; discovering and reporting on where and how the software is broken; identifying when the system will fail under load or stress.

It might be a good idea to consider the psychological and public relations problems associated with claiming that you break the software. Programmers and managers might subconsciously harbour the idea that the software was fine until the testers broke it. The product would have shipped on time, except the testers broke it. Normal customers wouldn’t have problems with the software; it’s just that the testers broke it. There are no systemic problems in the project that lead to problems in the product; nuh-uh, the testers broke it.

As an alternative, you could simply say that you investigate the software and report on what it actually does—instead of what people hope or wish that it does. Or as my colleague James Bach puts it, “We don’t break the software. We break illusions about the software.”

Posted in Test Framing, Very Short Blog Posts, Words and Semantics | 19 Comments »

Very Short Blog Posts (24): You Are Not a Bureaucrat

Saturday, February 7th, 2015

Here’s a pattern I see fairly often at the end of bug reports:

Expected: “Total” field should update and display correct result.
Actual: “Total” field updates and displays incorrect result.

Come on. When you write a report like that, can you blame people for thinking you’re a little slow? Or that you’re a bureaucrat, and that testing work is mindless paperwork and form-filling? Or perhaps that you’re being condescending?

It is absolutely important that you describe a problem in your bug report, and how to observe that problem. In the end, a bug is an inconsistency between a desired state and an observed state; between what we want and what we’ve got. It’s very important to identify the nature of that inconsistency; oracles are our means of recognizing and describing problems. But in the relationship between your observation and the desired state, the expectation is the middleman. Your expectation is grounded in a principle based on some desirable consistency. If you need to make that principle explicit, leave out the expectation, and go directly for a good oracle instead.

Posted in Reporting, Tester Skill, Testing Story, Very Short Blog Posts | 3 Comments »

« Older Entries