Exploratory Testing on an API? (Part 2)

July 17th, 2018

Summary:  Loops of exploration, experimentation, studying, modeling, and learning are the essence of testing, not an add-on to it. The intersection of activity and models (such as the Heuristic Test Strategy Model) help us to perform testing while continuously developing, refining, and reviewing it. Testing is much more than writing a bunch of automated checks to confirm that the product can do something; it’s an ongoing investigation in which we continuously develop our understanding of the product.

Last time out, I began the process of providing a deep answer to this question:

Do you perform any exploratory testing on APIs? How do you do it?

That started with reframing the first question

Do you do any exploratory testing on APIs?

into a different question

Given a product with an API, do you testing?

The answer was, of course, Yes. This time I’ll turn to addressing the question “How do you do it?” I’ll outline my thought process and the activities that I would perform, and how they feed back on each other.

Note that in Rapid Software Testing, a test is an action performed by a human; neither a specific check nor a scripted test procedure. A test is a burst of exploration and experiments that you perform. As part of that activity, a test include thousands of automated checks within it, or just one, or none at all. Part of the test may be written down, encoded as a specific procedure. Testing might be aided by tools, by documents or other artifacts, or by process models. But the most important part of testing is what testers think and what testers do.

(Note here that when I say “testers” here, I mean any person who is either permanently or temporarily in a testing role. “Tester” applies to a dedicated tester; a solo programmer switching from the building mindset; or a programmer or DevOps person examining the product in a group without dedicated testers.)

It doesn’t much matter where I start, because neither learning nor testing happen in straight lines. They happen in loops, cycles, epicycles; some long and some short; nested inside each other; like a fractal. Testing and learning entail alternation between focusing and defocusing; some quick flashes of insight, some longer periods of reflection; smooth progress at some times, and frequent stumbling blocks at others. Testing, by nature, is an exploratory process involving conversation, study, experimentation, discovery, investigation that leads to more learning and more testing.

As for anything else I might test, when I’m testing a product through an API, I must develop a strategy. In the Rapid Software Testing namespace, your strategy is the set of ideas that guide the design, development, and selection of your tests.

Having the the Heuristic Test Strategy Model in my head and periodically revisiting it helps me to develop useful ideas about how to cover the product with testing. So as I continue to describe my process, I’ll annotate what I’m describing below with some of the guideword heuristics from the HTSM. The references will look like this.

A word of caution, though:  the HTSM isn’t a template or a script.  As I’m encountering the project and the product, test ideas are coming to me largely because I’ve internalized them through practice, introspection, review, and feedback.  I might use the HTSM generatively, to help ideas grow if I’m having a momentary drought; I might use it retrospectively as a checklist against which I review and evaluate my strategy and coverage ideas; or I might use it as a means of describing testing work and sharing ideas with other people, as I’m doing here.

Testing the RST way starts with evaluating my context. That starts with taking stock of my mission, and that starts with the person giving me my mission. Who is my client—that is, to whom am I directly answerable? What does my client want me to investigate?

I’m helping someone—my client, developers, or other stakeholders—to evaluate the quality of the product. Often when we think about value, we think about value to paying customers and to end users, but there are plenty of people who might get value from the product, or have that value threatened. Quality is value to some person who matters, so whose values do we know might matter? Who might have been overlooked? Project Environment/Mission

Before I do anything else, I’ll need to figure out—at least roughly—how much time I’ll have to accomplish the mission. While I’m at it, I’ll ask other time-related questions about the project: are there any deadlines approaching? How often do builds arrive? How much time should I dedicate to preparing reports or other artifacts? Project Environment/Schedule

Has anyone else tested this product? Who are they? Where are they? Can I talk to them? If not, did they produce results or artifacts that will help me? Am I on a team? What skills do we have? What skills do we need? Project Environment/Test Team

What does my client want to me to provide? A test report, almost certainly, and bug reports, probably—but in what form? Oral conversations or informally written summaries? I’m biased towards keeping things light, so that I can offer rapid feedback to clients and developers. Would the client prefer more formal appoaches, using particular reporting or management tools? As much as the client might like that, I’ll also note whenever I see costs of formalization.

What else might the client, developers, and other stakeholders want to see, now or later on? Input that I’ve generated for testing? Code for automated checks? Statistical test results? Visualizations of those results? Tools that I’ve crafted and documentation for them? A description of my perception of the product? Formal reports for regulators and auditors? Project Environment/Deliverables I’ll continue to review my mission and the desired deliverables throughout the project.

So what is this thing I’m about to test? Project Environment/Test Item Having checked on my mission, I proceed to simple stuff so that I can start the process of learning about the product. I can start with any one of these things, or with two or more of them in parallel.

I talk to the developers, if they’re available. Even better, I particpate in design and planning sessions for the product, if I can. My job at such meetings is to learn, to advocate for testability, to bring ideas and ask questions about problems and risks. I ask about testing that the developers have done, and the checking that they’ve set up. Project Environment/Developer Relations

If I’ve been invited to the party late or not at all, I’ll make a note of it. I want to be as helpful as possible, but I also want to keep track of anything that makes my testing harder or slower, so that everyone can learn from that. Maybe I can point out that my testing will be better-informed the earlier and the more easily I can engage with the product, the project, and the team.

I examine the documentation for the API and for the rest of the product. Project Environment/Information I want to develop an understanding of the product: the services it offers, the means of controlling it, and its role in the systems that surround it. I annotate the documentation or take separate notes, so that I can remember and discuss my findings later on. As I do so, I pay special attention to things that seem inconsistent or confusing.

If I’m confused, I don’t worry about being confused. I know that some of my confusion will dissipate as I learn about the product. Some of my confusion might suggest that there are things that I need to learn. Some of my confusion might point to the risk that the users of the product will be confused too. Confusion can be a resource, a motivator, as long as I don’t mind being confused.

As I’m reading the documentation, I ask myself “What simple, ordinary, normal things can I do with the product?” If I have the product available, I’ll do sympathetic testing by trying a few basic requests, using a tool that provides direct interaction with the product through its API. Perhaps it’s a tool developed in-house; perhaps it’s a tool crafted for API testing like Postman or SOAPUI; or maybe I’ll use an interpreter like Ruby’s IRB along with some helpful libraries like HTTParty. Project Environment/Equipment and Tools

I might develop a handful of very simple scripts, or I might retain logs that the tool or the interpreter provides. I’m just as likely to throw this stuff away as I am to keep it. At this stage, my focus is on learning more than on developing formal, reusable checks. I’ll know better how to test and check the product after I’ve tried to test it.

If I find a bug—any kind of inconsistency or misbehaviour that threatens the value of the product—I’ll report it right away, but that’s not all I’ll report. If I have any problems with trying to do sympathetic testing, I’ll report them immediately. They may be usability problems, testability problems, or both at once. At this stage of the project, I’ll bias my choices towards the fastest, least expensive, and least formal reporting I can do.

My primary goal at this point, though, is not to find bugs, but to figure out how people might use the API to get access to the product, how they might get value from it, and how that value might be threatened. I’m developing my models of the product; how it’s intended to work, how to use it, and how to test it. Learning about the product in a comprehensive way prepares me to find better bugs—deeper, subtler, less frequent, more damaging.

To help the learning stick, I aspire to be a good researcher: taking notes; creating diagrams; building lists of features, functions, and risks; making mind maps; annotating existing documentation. Periodically I’ll review these artifacts with programmers, managers, or other colleagues, in order to test my learning.

Irrespective of where I’ve started, I’ll iterate and go deeper, testing the product and refining my models and strategies as I go. We’ll look at that in the next installment.

Are you a tester—solo or in a group?  Or are you a developer, manager, business person, documenter, support person, or someone in DevOps who wants to get very good at testing?  Attend Rapid Software Testing in Seattle, presented by James Bach and me, September 26-28, 2018.  Sign up!

Exploratory Testing on an API? (Part 1)

July 9th, 2018

In one forum or another recently, a tester, developer or DevOps person (I wasn’t sure which of these he/she was) asked:

Do you perform any exploratory testing on APIs? How do you do it?

Here’s an in-depth version of the reply I gave, with the benefit of links to deeper material.

To begin: application programming interfaces (APIs) are means by which we can use software to send commands to a product to make it do something. We do perform some testing on APIs themselves; interfaces represent one set of dimensions or factors or elements of the product. In a deeper sense, we’re not simply testing the APIs; we’re using them to control and observe the product, so that we can learn lots of things about it.

Now, there’s a problem with the first question: it implies that there is some kind of testing that is not exploratory. But all testing is exploratory. The idea that testing isn’t exploratory comes, in part, from a confusion between demonstrating that a product can work and testing to learn how it does work, and how it might not work.

When a product—a microservice, a component, an application framework, a library, a GUI application,…—offers an API, we can use tools to interact with it. We develop examples of how the product should work, and use the API to provide an appropriate set of inputs to the product, operating and observing it algorithmically and repetitively, which affords a kind of demonstration.

If we observe the behaviours and outputs that we anticipated and desired, we declare the demonstration successful, and we conclude that the product works. If we chose to speak more carefully, we would say that the product can work. If there has been some unanticipated, undesirable change in the product, our demonstration may go off the rails. When it does, we suspect a bug in the product.

It is tempting to call demonstrations “testing”, but as testing goes, it’s pretty limited.  To test the product, we must challenge it; we must attempt to disprove the idea that it works. The trouble is that people often assume that demonstration is all there is to testing; that checking and testing are the same thing.

In the Rapid Software Testing namespace, testing is evaluating a product by learning about it through exploration and experimentation, which includes to some degree: questioning, study, modeling, observation, inference, and plenty of other stuff.

Testing often includes checking as a tactic. Checking is the process of evaluating specific factors of the product by applying algorithmic decision rules to specific observations of a product, and then algorithmically indicating the results of those decision rules. Checking is embedded in testing.

Testing is fundamentally an exploratory process.  An exploratory process might include demonstration from time to time, and some demonstrations can be done with checking.  Where do checks come from? They come from testing!

As you’re first encountering an API, you are learning about it. There is documentation in some form that describes the functions, how to provide input, and what kinds of output to anticipate. There may be a description of how the product interacts with other elements of some larger system. The documentation probably includes some new terminology, and some examples of how to use the API. And it probably contains an error or two. Making sense of that documentation and discovering problems in it is an exploratory process.

You perform some experiments to see the API in action, maybe through a scripting language (like Python or Ruby) or maybe through a tool (like Postman). You provide input, perform some API calls, examine the output, and evaluate it. You might experience confusion, or bump into an inconsistency between what the service does and what the documentation says it does. Wrapping your head around those new things, and stumbling over problems in them, is an exploratory process.

You may have heuristics, words and patterns and techniques that guide your learning. Still, there’s no explicit script to read or set procedure to follow when you’re learning about an API. Learning, and discovering problems in the product as you learn, is an exploratory process.

As you learn about the product, you begin to develop ideas about scenarios in which your product will be used. Some products perform single, simple, atomic functions. Others offer more elaborate sets of transactions, maintaining data or state between API calls. You will imagine and experiment with the conversations that programmers and other products might have with your product. Developing models of how the product will be used is an exploratory process.

As you learn more about the product, you begin to anticipate risks; bad things that might happen and good things that might not happen. You may have heuristics that help you to develop risk models generally, but applying heuristics to develop a product-specific risk model is an exploratory process.

As you learn about risks, you develop strategies and ideas for testing for them.  Developing strategies is an exploratory process.

You try those test ideas. Some of those test ideas reveal things that you anticipated. Some of them don’t. Why not? You investigate to find out if there’s a problem with the API, with your test idea, or with the implementation of your test idea. Interpreting results and investigating potential bugs is an exploratory process.

As your learning deepens, you identify specific factors to be checked. You design and write code to check those factors. Developing those checks, like all programming, isn’t simply a matter of getting an idea and converting it into code. Developing checks is an exploratory process.

You may choose to apply those checks entirely in a repetitious, confirmatory way. On the other hand, every now and then — and especially when a new API call is being introduced or modified — you could induce variation, diversification, randomization, aggregation, data gathering, analysis, visualization, experimentation, and investigation into your work. Refreshing your strategies and tactics is an exploratory process.

In other words, practically everything that we do in testing a product except the mechanistic running of the checks is exploratory work. “Do you do any exploratory testing on APIs?” is equivalent to asking “Given a product with an API, do you do testing?”

The answer is, of course, Yes.  How do you do it?  Find out in Part 2.

Are you a tester—solo or in a group?  Or are you a developer, manager, business person, documenter, support person, or someone in DevOps who wants to get very good at testing?  Attend Rapid Software Testing in Seattle, presented by James Bach and me, September 26-28, 2018.  Sign up!

Test Cases and Coverage

June 28th, 2018

A tester recently asked about creating an in-house testing process, wanting to know how to start writing test cases so that testing effort could be estimated. My reply—which could just as easily apply to a developer or business analysts in a testing role—went something like this:

Test cases are not testing!  While that’s true, just saying so probably won’t help you very much, so let me offer an alternative to thinking about your assignment in terms of test cases.

At the beginning of the project, you don’t yet know how to test it. To develop a relevant test strategy, you must learn the product. There may be many ways for you to learn about the product—by attending meetings, talking to people, reviewing project plans, studying designs or sketches. You may even have an opportunity to work with the product directly; with a previous version; with a mockup or prototype; or with a part of the product.

Without a good deal of that learning, you cannot know what test cases should describe or contain—so instead of the typical fine granularity of test cases, consider the broader notion of coverage areas.

As you learn about a product or feature, create a coverage outline based on product factors. (A product factor is something that could be examined in a test, or something that could affect the outcome of a test.)  Your outline may take one or more of many forms—a hierarchical list, a table, an annotated diagram; these days we like mind maps.  The outline identifies areas of the product and its relationship to its context that can be covered with testing activity. In general, these areas will tend to fall into categories like structure, functions, data, interfaces, platforms, operations, and time.

You might want to note any specific factors or conditions that you believe must be checked—but since you’re at the beginning of the project, also note that no one can be aware of all of them yet.

While you’re at it, begin advocating for testabilty; things that can make testing faster and easier. A product tends to be faster and easier to test when developers provide controllability through scriptable interfaces (APIs); visibility through log files; and code that they have already tested for basic problems and coding errors. When programmers do more of their own testing (including automated checks), testers won’t have to investigate and report on shallow bugs—which leaves more time for deep testing.

Your coverage outline goes hand in hand with a risk list—in which you identify technological and business risks that will influence your strategies for covering the test space. As development progresses, we learn about potential problems that threaten the value of the product. It’s a special responsibility of someone in the testing role to explore ideas about problems and where they might be lurking; to perform experiments to identify problems; and to determine the nature of those problems so that developers and managers can make informed decisions on what to do about them.

Consider the kinds of activities that testers perform: making contact with the team; learning about the product; analyzing how to test it; preparing equipment and tools; developing ideas for deep coverage; doing deep testing; and winding down the testing and helping prepare the product for deployment.  Then, instead of estimating by test cases, or by hours or days, consider estimating by sessions.

Sessions are bursts of nominally uninterrupted testing work, guided by a mission statement or charter. The time span of a session is intentionally imprecise, and we say “nominally uninterrupted” because, in the real world, it’s unusual for work not to get interrupted.  When planning a session, we base its charter on the idea that it will take around 90 minutes.  In reality, a session may last from 45 minutes to two and a half hours.  That’s because we can identify what we’d like to happen in a session, but our knowledge of what will happen is uncertain.

For instance, the bugs that you encounter during a session will affect how much you’re able to cover during the session. When bugs require investigation and reporting, coverage suffers. So, after the session, the charter may change to reflect what actually happened.  The difference between what you hope to do and what you’ll be able to do can’t be known very well in advance—so guess how many sessions you’ll need to cover the stuff that you know about.  Then double your guess.

What will happen over the course of a collection of sessions is uncertain too. You’ll need to allocate plenty of time to account for setting up for each session; dealing with obstacles; debriefing; debugging with developers; planning meetings; breaks and lunch; and so forth. As with the specifics of what you’re going to cover, that stuff is uncertain too.  However, it’s reasonable to assume, given the non-session work, a tester will have time for two or three sessions at most in a day.

In the early stages, you’re going to have to guess about a lot of this. That’s normal; testing is about exploration, discovery, investigation, and learning.  Those things aren’t calculable except in the most general way, and work tends to expand or shrink to fit the time available for it. Recognize that little of your early planning will play out as anticipated.

As you go through loops of learning and testing, your coverage outline and risk list will evolve. Commit to providing daily reports on what is being learned; and on the gap between what has been covered and what has not yet been covered—product coverage and risk coverage. Daily reports will keep managers alert to problems that need their attention—problems that might threaten the value of the product or the success of the project.  Managers need to know about those, not about test case counts.

As a key element in your reporting, provide notes on testability and its absence—anything that makes the product easier or harder to test. Reduced testability means testing takes longer or reduces coverage.  Reduced coverage means bugs will survive undetected for longer—maybe into production.

Every day, as you learn about the product and update your models of it, you will be informing your clients on what testing has been done, and what testing could be done.  In the end, your clients will make the decision about when they have sufficient information about coverage and risk to ship the product.  It’s your job to keep them up to date and well-informed; it’s their job to decide whether the product they’ve got is the product they want.

All that said, you might not find it necessary to estimate the amount of testing that the project needs. Since both you and the client already have an idea of what the schedule and budget will be for this project anyway (based on past experience and other factors), your estimate may already be done for you. If not, and you’re asked to provide and estimate, don’t provide one; provide three: a reasonable estimate; a safer estimate; and a riskier estimate.

I anticipate someone will respond “all this sounds harder than preparing a list of test cases.” I don’t believe it is—I’d say it’s easier, in fact. A coverage outline is certainly easier to maintain than a library of test cases.  Working from an outline acknowledges the fact that we’ll learn and refine our plans as we go, where the test case approach turns us into ostriches when it comes to anticipating change.

Some people will respond “But managers want test cases!” Yes, some do; and kids want candy instead of real food, and addicts want illegal drugs, too. We are not obliged to do harmful things, and fixation on test cases puts more emphasis on documentation and procedures than on actual testing. I believe most managers want test cases because they’re used to them, and because they’re not aware of alternatives. Testers provide test cases because managers keep asking about them. I’d offer real testing work, continuous collaboration, refinement of the tasks at hand—and solid information about coverage, problems, and risks.

Related reading:

No Test Cases Required (PowerPoint presentation)
Questioning Test Cases
Testers Learn, but Test Cases Don’t Teach

Rapid Software Testing is designed for managers, developers, testers, and teams who want help with making testing faster, easier, cheaper, and above all, effective. Read about RST classes here, and consider the upcoming public class taught by both James Bach and Michael Bolton.

Which Test Cases Should I Automate?

June 15th, 2018

When someone asks “I have a big suite of manual tests; which tests (or worse, which test cases) should I automate?”, I often worry about several things.

The first thing is that focusing on test cases is often a pretty lousy way to think about testing.  That’s because test cases are often cast in terms of following an explicit procedure in order to observe a specific result.  At best, this confirms that the product can work if someone follows that procedure, and it also assumes that any result unobserved in the course of that procedure and after it is unimportant.

The trouble is that there are potentially infinite variations on the procedure, and many factors might make a difference in a test or in its outcome. Will people use the product in only one way? Will this specific data expose a problem?  Might other data expose a problem that this data does not?  Will a bug appear every time we follow this procedure? The test case often actively suppresses discoveryTest cases are not testing , and bugs don’t follow the test cases.

Second: testing is neither manual nor automated. A test cannot be automated.  Elements of the procedure within the test (in particular, checks for specific facts) can be automated.  But your test is not just the machine performing virtual key presses or comparing an output to some reference.

Your test is a process of activity and reasoning:  analyzing risk; designing an experiment; performing the experiment; observing what happens before, during, and after the experiment; interpreting the results; and preparing and a relevant report.  This depends on your human intentions, your mindset, and your skill set.  Tools can help every activity along the way, but the test is something you do, not something the machine does.

Third:  lots of existing test cases are shallow, pointless, out of date, ponderous, inefficient, cryptic, and unmotivated by risk.  Often they are focused on the user interface, a level of the product that is often quite unfriendly to tools. Because the test cases exist, they are often pointlessly repeated, long after they have lost any power to find a bug.  Why execute pointless test cases more quickly?

It might be a much better idea to create new automated checks that are focused on specific factors of the product, especially at low levels, with the goal of providing quick feedback to the developer.  It might be a good idea to prepare those checks as a collaboration between the developer and the tester (or between two developers, one of whom is in the builder’s role, with the other taking on a testing role). It might be a good idea to develop those checks as part of the process of developing some code. And it might be a really good idea to think about tools in a way that goes far beyond faster execution of a test script.

So I encourage people to reframe the question.  Instead of thinking what (existing) test cases should I automate? try thinking:

  • What reason do we have for preserving these test cases at all? If we’re going to use tools effectively, why not design checks with tools in mind from the outset?  What tool-assisted experiments could we design to help us learn about the product and discover problems in it?
  • What parts of a given experiment could tools accelerate, extend, enhance, enable, or intensify?
  • What do we want to cover? What product factors could we check? (A product factor is something that can be examined during a test, or that might influence the outcome of a test.)
  • To what product factors (like data, or sequences, or platforms,…) could we apply tools to help us to induce variation into our testing?
  • How could tools help us to generate representative, valid data and exceptional or pathological data?
  • In experiments that we might perform on the product, how might tools help us make observations and recognize facts that might otherwise escape our notice?  How can tools make invisible things visible?
  • How can tools help us to generate lots of outcomes that we can analyze to find interesting patterns in the data?  How can tools help us to visualize those outcomes?
  • How could the tool help us to stress out the product; overwhelm it; perturb it; deprive it of things that it needs?
  • How can developers make various parts of the system more amenable to being operated, observed, and evaluated with the help of tools?
  • How might tools induce blindness to problems that we might be able to see without them?
  • What experiments could we perform on the product that we could not perform at all without help from tools?

Remember:  if your approach to testing is responsible, clever, efficient, and focused on discovering problems that matter, then tools will help you to find problems that matter in an efficient, clever, and responsible way.  If you try to “automate” bad testing, you’ll find yourself doing bad testing faster and worse than you’ve ever done it before.

More reading:
A Context-Driven Approach to Automation in Testing
Manual and Automated Testing
The End of Manual Testing

Learn about Rapid Software Testing

Sign up for the RST class, taught by both James Bach and Michael Bolton, in Seattle September 26-28, 2018.  It’s not just for testers!  Anyone who wants to get better at testing is welcome and encouraged to attend—developers, managers, designers, ops people, tech support folk…

Very Short Blog Posts (36): Positive, Negative, and Sympathetic Testing

May 12th, 2018

In Rapid Software Testing, a “positive test” is one that honours every required and explicitly declared condition or factor for a desired outcome. A “negative test” is one that violates (or dishonours, disrespects, ignores, omits, undermines…) at least one required and explicitly declared condition.

That said, we don’t talk very much about positive and negative testing. We do talk about “sympathetic testing“, a closely related idea. Sympathetic testing helps us to learn about the product and how people would obtain value from it. Rich models derived from that learning help us to focus on finding more important bugs.

“Positive testing” is really about demonstration or confirmation; showing that the product can work. It might be a good idea to situate a fair amount of positive testing in a suite of maintainable automated checks, to cover conditions in which people will do exactly what they’re supposed to do at just the right time.

That may be all right as far as it goes, but it doesn’t fit very well with the real world. In the real world, people make mistakes. They get confused, become curious, forget stuff, experience frustration, feel impatient, act mischievously… They do things that the product’s design does not anticipate or specify, and they do so accidentally, incidentally, inadvertently, inventively, desperately, obliviously, intentionally, maliciously,… They don’t follow “the rules”—which aren’t really rules at all; just someone’s hopes or desires that everyone will behave themselves.

People will violate at least one explicitly required condition a lot more often than you think. So, as testers, it’s crucial for us to challenge the product and vary our tests to find out how it responds.

Excellent testing isn’t limited to showing the the product can produce an expected result. Excellent testing includes learning what the product does when people do something unexpected.

Rapid Software Testing in Toronto, June 4-6

May 3rd, 2018

I don’t usually give upcoming events an entire blog post. I usually post upcoming events in the panel on the right. However, this one’s special.

I’m presenting a three-day Rapid Software Testing class, organized by TASSQ in Toronto, June 4-6, 2018.

Rapid Software Testing is focused on doing the fastest, least expensive testing that still completely fulfills testing’s mission: evaluating the product by learning about it through exploration and experimentation. Developers and managers—including those from organizations where there are few testers or none at all—are welcome and warmly invited. It’s the only public RST class being offered in Canada in 2018. Early bird pricing is available until Friday, May 4. Register here.

How Long Will the Testing Take?

April 27th, 2018

Today yet another tester asked “When a client asks ‘How long will the testing take for our development project?’, how should I reply?”

The simplest answer I can offer is this: if you’re testing as part of a development project, testing will take exactly as long as development will take. That’s because effective, efficient testing is not separate from development; it is woven into development.

When we develop a software product or service, we build things: designs, prototypes, functions, modules, components, interfaces, integrated components, services, complete systems… There is some degree of risk—the possibility of problems—associated with everything we build. To be successful, each element of a system must fit with other elements of the system, and must satisfy the people who are using and building the system.

Despite our best intentions, we’re likely to make some mistakes and experience some misunderstandings along the way. If we review things as we go, we’ll eliminate many problems before we turn them into code. Still, there’s risk in using elements when we haven’t built, configured, operated, observed, and evaluated them. Unless we actually try to use what we’ve built, look for problem in it, and develop a well-grounded, empirical understanding of it, we risk fooling ourselves about how it really works—and doesn’t.

So it’s a good idea to start testing things as developer are creating or assembling them—and to test how they fit with other elements of the system—from the moment we start to build them or put them together.

Testing, for a given element or system, stops when we have a reasoned belief that there is no more development work to be done. Testing stops when people are satisfied that they know enough about the system and its elements to have discovered and resolved all the important problems. That satisfaction will be easier to establish relatively quickly, for any given element or system, when the product is designed to be easy to test. If you’ve been testing each part of the product deeply, investigating risk, and addressing problems throughout development, the whole product will be easier to test.

For the project to be successful, it’s important for the whole team to keep discovering and identifying ways in which people might be dissatisfied. That includes not only finding problems in the product, but also finding problems in our concept of what it might mean to be “done”. It’s not the tester’s job to build confidence in the product, but to reveal where confidence is unwarranted. That’s central to testing work, even though it’s difficult, disappointing, and to some degree socially disruptive.

Asking how long testing will take is an odd question for managers to ask, because it’s just like asking how long management will take. Management of a development project starts as development starts, and ends as development ends. Testing enables awareness about the product and problems in it, so that managers and developers can make decisions about it. So testing starts when the project starts, and testing stops when managers and developers have made their final decisions about it. Testing doesn’t stop until development and management of the project is done.

People often stop testing a product after it’s been put into production. Now: it might be a good idea to monitor and test some aspects of the system after it’s been put into production, too. (We call that live-site testing.) Live-site testing is often a very good idea, but like all other forms of testing, ultimately, it’s optional. Here’s one good reason to continue live-site testing: when you believe that there will be more development work done on the system.

So: development and management and testing go hand in hand. When a manager asks “How long will the testing take?”, it seems to me that the appropriate answer is “Testing will take just as long as development will take.” When people are satisfied that there’s no more important development work to do, they will also be satisfied that there’s no more important testing work to do either.

It’s important to remember, though: determining when people will be satisfied is something that we can guess, but cannot calculate. Satisfaction is a feeling, not a finish line.

Further reading:

How Long Will The Testing Take?
Project Estimation and Black Swans (Part 1)
Project Estimation and Black Swans (Part 5): Test Estimation

Very Short Blog Posts (35): Make Things Visible

April 24th, 2018

I hear a lot from testers who discover problems late in development, and who get grief for bringing them up.

On one level, the complaints are baseless, like holding an investigate journalist responsible for a corrupt government. On the other hand, there’s a way for testers to anticipate bad news and reduce the surprises. Try producing a product coverage outline and a risk list.

A product coverage outline is an artifact (a mind map, or list, or table) that identifies factors, dimensions, or elements of a product that might be relevant to testing it. Those factors might include the product’s structure, the functions it performs, the data it processes, the interfaces it provides, the platforms upon which it depends, the operations that people might perform with it, and the way the product is affected by time. (Look here for more detail.) Sketches or diagrams can help too.

As you learn more through deeper testing, add branches to the map, or create more detailed maps of particular areas. Highlight areas that have been tested so far. Use colour to indicate the testing effort that has been applied to each area—and where coverage is shallower.

A risk list is a list of bad things that might happen: Some person(s) will experience a problem with respect to something desirable that can be detected in some set of conditions because of a vulnerability in the system. Generate ideas on that, rank them, and list them.

At the beginning of the project or early as possible, post your coverage outline and risk list in places where people will see and read them. Update it daily. Invite questions and conversations. This can help you change “why didn’t you find that bug?” to “why didn’t we find that bug?”

Very Short Blog Posts (34): Checking Inside Exploration

April 23rd, 2018

Some might believe that checking and exploratory work are antithetical. Not so.

In our definition, checking is “the algorithmic process of operating and observing a product, applying decision rules to those observations, and reporting the outcome of those decision rules”.

We might want to use some routine checks, but not all checks have to be rote. We can harness algorithms and tools to induce variation that can help us find bugs. Both during development of a feature and when we’re studying and evaluating it, we can run checks that use variable input; in randomized sequences; at varying pace; all while attempting to stress out or overwhelm the product.

For instance, consider inducing variation and invalid data into checks embedded in a performance test while turning up the stress. When we do that, we can discover when the product falls to its knees under how much load—how the product fails, and what happens next. That in turn affords the opportunity to find out whether the product deals with overhead associated with error handling—which may result in feedback loops and cascades of stress and performance problems.

We can use checks as benchmarks, too. If a function takes significantly and surprisingly more or less time to do its work after a change, we have reason to suspect a problem.

We could run checks in confirmatory ways, which, alas, is the only way most people seem to use them. But we can also design and run checks taking a disconfirmatory and exploratory approach, affording discovery of bugs and problems. Checking is always embedded in testing, which is fundamentally exploratory to some degree. If we want to find problems, it would be a good idea to explore the test space, not just tread the same path over and over.

Interview on Rapid Software Testing

March 27th, 2018

Not too long ago, I had a conversation with Joe Colantonio. Joe asks questions that prompt answers about the nature of testing, accountability, and telling the testing story.

Enjoy!