Exploratory Testing on an API? (Part 3)

July 30th, 2018

In the last two posts, I’ve been answering the question

Do you perform any exploratory testing on an API? How do you do it?

Last time I described the process of learning about factors in the project environment that affect my strategy as I go about the testing of an API. This time, I’ll go deeper, learn more, and begin to interact with the product. That learning and interaction feeds back into more learning and more interaction.

Along the way, I’ll develop ideas that are reflected in the Heuristic Test Strategy Model. I get ideas about potential problems and risks based on Quality Criteria (and things that threaten them). I identify Product Factors and get ideas about coverage—how to the cover the product with testing. I apply those ideas using approaches that come from families of Test Techniques.

Notice, though, that even though I’ve tagged them, these ideas don’t come to me from the HTSM as such. Ideas occur to me as I encounter elements of the product and its context. Each interaction with and observation of the product triggers ideas about risk and how to test for it. Some of these ideas come from my testing and development experience; others from stories told by my colleagues or reported in the news; still others from the product and its documentation; and those all lead to new ideas that are synthesized in the moment.

You may find that this post is long. That length reflects the volume of ideas that occur in an exploratory process.

Whether they are end users—getting access to the product through the GUI—or developers—getting access to the product through an API—people use software to make things happen. They have tasks that they want to get done. So what can I imagine the API’s user wanting to do? Product Factors/Operations; Test Techniques/Scenario Testing.

An API’s user wants to get some data, make something happen, or effect some change. Anything that happens or changes in the product is driven by specific functions in the code. What functions does the product perform? Product Factors/Functions I find out by looking at the product and API documentation if it’s available, or by looking at source code if that’s available. Project Environment/Information If I can’t get to either one, I might ask developers or designers for help. Project Environment/Developer Relations

I take notes about what I’m learning, perhaps in the form of a mind map; perhaps in point-form text; perhaps in a table. The process of taking notes helps to cement what I’m learning. In addition, I want to notice inconsistencies between my sources of information. Sometimes different people tell me the different things. Some people contradict what is written in the requirements documents or specifications. Whenever those things happen, bugs are breeding. I’ll collect lots of specific statements about the product should do, and look for controversy. Test Techniques/Claims Testing

I’ll review what I already know. To what functions does the API offer access? The user of the API wants to obtain some kind of value from interacting with the product, and those values are expressible in terms of quality criteria. Quality Criteria/Capability What could go wrong with those functions and threaten something do do with one of those quality criteria?

What are the protocols for calling those functions? Product Factors/Interfaces Is there anything, like setting up or logging in, that should happen before an API call? What if they don’t happen? I’ll make a note to try that later. Test Techniques/Flow Testing “Before” reminds me to ask myself about “after”, too: Are there there things to be cleaned up after an API call? Product Factors/Time Does data need to be saved or purged?

Given a set of available functions, I’ll try to use a small selection of the functions that I’m aware of. Test Techniques/Function Testing; Test Techniques/Claims Testing. I’ll try some simple stuff using a tool that affords direct interaction; these days it would be Postman, or some quick calls from IRB (the Ruby interpreter). Does the product appear to do what it should? Can it work? Quality Criteria/Capability

I’ll start simple with representative inputs; then I’ll explore around the extents and limits that I’ve been told about or that are documented. Test Techniques/Domain Testing. I’ll put in some preposterously long strings, or large quantities, or deliberately malformed input. I’ll leave out mandatory fields in the call; put in extra fields that I’ve made up; provide letters where where only numbers are anticipated; feed in symbols or accented characters where only normal letters are anticipated. If it’s a Web app, I might focus on a smattering of characters that are associated with cross-site scripting, SQL injection, or other security risks. Quality Criteria/Security

If I do something that should trigger error handling, does error handling happen? Does the product return an error code? After that, does continue to do what it should do? Test Techniques/Claims Testing; Quality Criteria/Reliability/Robustness I pay special attention when the product appears to contradict the documentation or what someone has told me. Test Techniques/Claims Testing

At this point, if I find problems, I note them. Even if I don’t, I start to gather some ideas about how I might use tools and automated checks to vary the input and cover whole sets of possible inputs when it’s time for deep testing.

I pause and remember that that there’s a difference between the product doing what it should and appearing to do what it should. Every step of the way, I’d like to be able to observe as much as I can of the product’s state. Are there any special API function calls that I can use to examine or control the state of the product before, during, and after the usual calls? Quality Criteria/Development/Testability These may not be noted in the public documentation; I may have to ask developers about them Project Environment/Developer Relations or look at source code. Project Environment/Information

Thinking about testability like this reminds me to ask: Does the product offer logging? I have a look. Is the format of the logs consistent with other logs in the same system? Consistent log files will help to make my testing go more quickly and easily, which helps me to get to deeper testing. Quality Criteria/Development/Testability If there is no logging and there is resistance from the developers or from management to putting it in, I’ll keep that in mind and note it for the future, when I design automated checks; the logs for my checks may have to be more specific and detailed.

I’ve wandered from thinking about functions to testability, so I pull on the leash to bring my mind back to functions again. What services do the functions perform for the rest of the system? What are the bits and pieces of the system? Product Factors/Structure If I haven’t started one already, I’ll sketch a diagram of the product’s structure, and use that to help guide my understanding of the product. As my understanding develops, I’ll share the diagram with others to determine whether we’re all on the same page, and to help to maintain our awareness of the product’s structure. I might also post a copy up in a prominent place and invite people to let me know as soon as they see something wrong. Every now and again, I’ll deliberately review the diagram.

Thinking about the parts of the system reminds to to ask myself: Upon what parts of the whole system do those functions depend? In Rapid Software Testing, something that our product depends upon that is outside the control of our current project is called a platform. Product Factors/Platform. Platforms might include the operating system, the browser, application frameworks, third-party libraries, stuff that was built in-house on another project. Platforms might also include hardware elements, too: displays, keyboards, mice, touchscreens, printers, or other peripherals. I’ll develop lists of platform dependencies, preparing myself for deeper coverage later.

I return to exercising the functions. Most functions do something to something: data. What data can I supply via the API? In what format? What responses will the API provide? How are responses structured? Product Factors/Data. Answers to these questions will influence my selection of tools for deeper testing. Project Environment/Equipment and Tools

Is the API providing access to a database? If so, and I’m focused on testing the functions in the API, I might be able use direct access to the database as an oracle. Test Techniques/Function Testing

Thinking about databases prompts me to think about stored data and its quality. I focus for a moment on the quality of the data, and about what data I might want to use or generate for automated checks. Test Techniques/Domain Testing; Test Techniques/Automated Checking If I want to examine risk related to data quality, going directly to the database might be more efficient than trying to get at the data through the API’s functions. Of course, I could do a smattering of both, for which I’ll need different tools. Project Environment/Equipment and Tools

Although I’m doing it in a somewhat more painstaking way, as I’m learning to use the API, I’m doing what one of its users—typically a developer—would do to learn about it. Test Techniques/User Testing Just like its intended user, I make mistakes—some intentionally, some quite unintentionally. How difficult might it be for a developer to use this API correctly? How easy might it be to use it incorrectly, or in a way that provides misleading, ambiguous, or confusing results? Quality Criteria/Usability Test Techniques/Risk-Based Testing

I imagine a set of programming tasks that I might want to accomplish using the API. At that same time, I imagine a person performing those tasks; a role. It’s quite possible that not all users of the API will be expert programmers, so it makes sense for me to create a “novice programmer” role. Test Techniques/User Testing What if I make the kinds of mistakes a novice programmer might make, and ask for something in a way that the API doesn’t support? What if I hand it pathological or malformed data; data structures with missing elements, dates in the wrong format, or characters that might be used in countries other than my own? Quality Criteria/Localizability Does the product report an error?

How well do things get cleaned up after an error condition? Are error conditions handled gracefully? Quality Criteria/Reliability/Error Handling How are error conditions relayed to the user of the API—the software calling it, and the human writing that software? Are there error codes to make troubleshooting easier? Quality Criteria/Development/Supportability? Human-readable strings or data structures? Are they consistent with the documentation? Quality Criteria/Usability. Are they reasonably elegant and efficient, or are they clunky, confusing, misspelled, or embarrassing in any way? Quality Criteria/Charisma

Some APIs are used for atomic interactions; single calls to a service followed by single responses from it. Other APIs work more like a conversation, retaining data or maintaining the product in a certain state from one call to the next. A conversation has its risks: what happens when I interrupt it? What happens if I leave out a particular call or response? Test Techniques/Flow Testing What if I varied the sequence of interactions in a conversation? Product Factors/Time Has anyone made any claims about timeouts, either informally or in the documentation? Test Techniques/Claims Testing

How long will the system let me leave a conversation dangling? Product Factors; Time What happens to the internal state of the system or to the data that’s being managed after a conversation has been abandoned for a while? What happens when I try to pick up a previously abandoned conversation? Quality Criteria; Reliability

Thinking about sequence leads me to think about time generally. How long does it take for the system to process an API call and return a result? Product Factors/Time Tools help make the round-trip time easily visible—but I don’t trust the first result. Because the tool makes it so easy, I often do several quick tests to observe results from the same call several times. If there’s substantial variation, I make a note of it; it might be a bug, or it might be a target for deeper testing.

Has anyone made explicit claims about performance? Project Environment/Information Even though I might refer to those claims, I don’t need them to do performance testing; I’ll operate the product, collect observations about performance, and include those observations in my report.

In addition to the API, there may also be human user interfaces: a GUI, or a command line interface. I’ll exercise those interfaces too, and look for inconsistencies between the way they behave and the way the API behaves. Product Factors/Interfaces Inconsistency within the product is a powerful oracle.

Considering the human interfaces reminds me of other interfaces. The API interacts with certain components in the product, and the product inevitably interacts with other products: the operating system, the file system, third-party frameworks and libraries, networks, printers, ports, Bluetooth… These interfaces afford access to the platforms upon which the product depends. Product Factors/Platform

What if one of those dependencies were missing? What if I deprived the of something it needs by preventing access to files or databases? Test Techniques; Stress Testing Would it stay up? Quality Criteria; Reliability

Thinking about reliability prompts me to think about how the product might behave when I place it under load. Quality Criteria; Performance Could I stress it out by overwhelming it with large amounts of data in specific fields, or with an extreme rate of calls? Test Techniques; Stress Testing I’ll definitely need tools for that. Project Environment/Equipment and Tools

I could use those tools to do the same checks over and over again, comparing the product’s output to a specified, hard-coded result. But I could broaden my coverage by varying the inputs for the checks, and using a parallel algorithm to compute the desired result. Exploring the product prepares me to identify those opportunities.

After a session or two of this kind of activity, I’ve learned a lot of useful stuff about the product. My learning allows me to make much better decisions about where I might want to apply automated checks to address particular risks; how I might want to induce variation in those checks; how I might want to extend ideas about checking.

It may seem like this exercise sounds like a long process. It doesn’t have to be. Code takes time to develop. Quick little informal experiments can fast and cheap. Ideas about risk, coverage, quality criteria, and oracles can be developed much more quickly and less expensively than code can. Writing about these things, as I’ve done here, takes far longer than thinking them through! Yet taking notes as I go often helps by inducing pauses in which more ideas occur to me.

This is testing! I’m not only testing to find bugs here; I’m testing to learn how to test the product. If I’m going to use automated checking as part of a strategy for learning specific about the product quickly, I must test to learn how to develop those checks. This is an exploratory process! Notice how often each idea above is connected to the one before. Testing is fundamentally exploratory.

Do I do exploratory testing on an API?  Yes—because if I’m testing it, I’m doing exploratory work! If I’m doing anything other than that, it’s probably automated checking—and I can’t begin to do excellent checking until I’ve done that exploratory work.

I’ll have some follow-up notes in the final post in this series.

Are you a tester—solo or in a group? Or are you a developer, manager, business person, documenter, support person, or someone in DevOps who wants to get very good at testing? Attend Rapid Software Testing in Seattle, presented by James Bach and me, September 26-28, 2018.  Sign up!

Exploratory Testing on an API? (Part 2)

July 17th, 2018

Summary:  Loops of exploration, experimentation, studying, modeling, and learning are the essence of testing, not an add-on to it. The intersection of activity and models (such as the Heuristic Test Strategy Model) help us to perform testing while continuously developing, refining, and reviewing it. Testing is much more than writing a bunch of automated checks to confirm that the product can do something; it’s an ongoing investigation in which we continuously develop our understanding of the product.

Last time out, I began the process of providing a deep answer to this question:

Do you perform any exploratory testing on APIs? How do you do it?

That started with reframing the first question

Do you perform any exploratory testing on APIs?

into a different question

Given a product with an API, do you do testing?

The answer was, of course, Yes. This time I’ll turn to addressing the question “How do you do it?” I’ll outline my thought process and the activities that I would perform, and how they feed back on each other.

Note that in Rapid Software Testing, a test is an action performed by a human; neither a specific check nor a scripted test procedure. A test is a burst of exploration and experiments that you perform. As part of that activity, a test include thousands of automated checks within it, or just one, or none at all. Part of the test may be written down, encoded as a specific procedure. Testing might be aided by tools, by documents or other artifacts, or by process models. But the most important part of testing is what testers think and what testers do.

(Note here that when I say “testers” here, I mean any person who is either permanently or temporarily in a testing role. “Tester” applies to a dedicated tester; a solo programmer switching from the building mindset; or a programmer or DevOps person examining the product in a group without dedicated testers.)

It doesn’t much matter where I start, because neither learning nor testing happen in straight lines. They happen in loops, cycles, epicycles; some long and some short; nested inside each other; like a fractal. Testing and learning entail alternation between focusing and defocusing; some quick flashes of insight, some longer periods of reflection; smooth progress at some times, and frequent stumbling blocks at others. Testing, by nature, is an exploratory process involving conversation, study, experimentation, discovery, investigation that leads to more learning and more testing.

As for anything else I might test, when I’m testing a product through an API, I must develop a strategy. In the Rapid Software Testing namespace, your strategy is the set of ideas that guide the design, development, and selection of your tests.

Having the the Heuristic Test Strategy Model in my head and periodically revisiting it helps me to develop useful ideas about how to cover the product with testing. So as I continue to describe my process, I’ll annotate what I’m describing below with some of the guideword heuristics from the HTSM. The references will look like this.

A word of caution, though:  the HTSM isn’t a template or a script.  As I’m encountering the project and the product, test ideas are coming to me largely because I’ve internalized them through practice, introspection, review, and feedback.  I might use the HTSM generatively, to help ideas grow if I’m having a momentary drought; I might use it retrospectively as a checklist against which I review and evaluate my strategy and coverage ideas; or I might use it as a means of describing testing work and sharing ideas with other people, as I’m doing here.

Testing the RST way starts with evaluating my context. That starts with taking stock of my mission, and that starts with the person giving me my mission. Who is my client—that is, to whom am I directly answerable? What does my client want me to investigate?

I’m helping someone—my client, developers, or other stakeholders—to evaluate the quality of the product. Often when we think about value, we think about value to paying customers and to end users, but there are plenty of people who might get value from the product, or have that value threatened. Quality is value to some person who matters, so whose values do we know might matter? Who might have been overlooked? Project Environment/Mission

Before I do anything else, I’ll need to figure out—at least roughly—how much time I’ll have to accomplish the mission. While I’m at it, I’ll ask other time-related questions about the project: are there any deadlines approaching? How often do builds arrive? How much time should I dedicate to preparing reports or other artifacts? Project Environment/Schedule

Has anyone else tested this product? Who are they? Where are they? Can I talk to them? If not, did they produce results or artifacts that will help me? Am I on a team? What skills do we have? What skills do we need? Project Environment/Test Team

What does my client want to me to provide? A test report, almost certainly, and bug reports, probably—but in what form? Oral conversations or informally written summaries? I’m biased towards keeping things light, so that I can offer rapid feedback to clients and developers. Would the client prefer more formal appoaches, using particular reporting or management tools? As much as the client might like that, I’ll also note whenever I see costs of formalization.

What else might the client, developers, and other stakeholders want to see, now or later on? Input that I’ve generated for testing? Code for automated checks? Statistical test results? Visualizations of those results? Tools that I’ve crafted and documentation for them? A description of my perception of the product? Formal reports for regulators and auditors? Project Environment/Deliverables I’ll continue to review my mission and the desired deliverables throughout the project.

So what is this thing I’m about to test? Project Environment/Test Item Having checked on my mission, I proceed to simple stuff so that I can start the process of learning about the product. I can start with any one of these things, or with two or more of them in parallel.

I talk to the developers, if they’re available. Even better, I particpate in design and planning sessions for the product, if I can. My job at such meetings is to learn, to advocate for testability, to bring ideas and ask questions about problems and risks. I ask about testing that the developers have done, and the checking that they’ve set up. Project Environment/Developer Relations

If I’ve been invited to the party late or not at all, I’ll make a note of it. I want to be as helpful as possible, but I also want to keep track of anything that makes my testing harder or slower, so that everyone can learn from that. Maybe I can point out that my testing will be better-informed the earlier and the more easily I can engage with the product, the project, and the team.

I examine the documentation for the API and for the rest of the product. Project Environment/Information I want to develop an understanding of the product: the services it offers, the means of controlling it, and its role in the systems that surround it. I annotate the documentation or take separate notes, so that I can remember and discuss my findings later on. As I do so, I pay special attention to things that seem inconsistent or confusing.

If I’m confused, I don’t worry about being confused. I know that some of my confusion will dissipate as I learn about the product. Some of my confusion might suggest that there are things that I need to learn. Some of my confusion might point to the risk that the users of the product will be confused too. Confusion can be a resource, a motivator, as long as I don’t mind being confused.

As I’m reading the documentation, I ask myself “What simple, ordinary, normal things can I do with the product?” If I have the product available, I’ll do sympathetic testing by trying a few basic requests, using a tool that provides direct interaction with the product through its API. Perhaps it’s a tool developed in-house; perhaps it’s a tool crafted for API testing like Postman or SOAPUI; or maybe I’ll use an interpreter like Ruby’s IRB along with some helpful libraries like HTTParty. Project Environment/Equipment and Tools

I might develop a handful of very simple scripts, or I might retain logs that the tool or the interpreter provides. I’m just as likely to throw this stuff away as I am to keep it. At this stage, my focus is on learning more than on developing formal, reusable checks. I’ll know better how to test and check the product after I’ve tried to test it.

If I find a bug—any kind of inconsistency or misbehaviour that threatens the value of the product—I’ll report it right away, but that’s not all I’ll report. If I have any problems with trying to do sympathetic testing, I’ll report them immediately. They may be usability problems, testability problems, or both at once. At this stage of the project, I’ll bias my choices towards the fastest, least expensive, and least formal reporting I can do.

My primary goal at this point, though, is not to find bugs, but to figure out how people might use the API to get access to the product, how they might get value from it, and how that value might be threatened. I’m developing my models of the product; how it’s intended to work, how to use it, and how to test it. Learning about the product in a comprehensive way prepares me to find better bugs—deeper, subtler, less frequent, more damaging.

To help the learning stick, I aspire to be a good researcher: taking notes; creating diagrams; building lists of features, functions, and risks; making mind maps; annotating existing documentation. Periodically I’ll review these artifacts with programmers, managers, or other colleagues, in order to test my learning.

Irrespective of where I’ve started, I’ll iterate and go deeper, testing the product and refining my models and strategies as I go. We’ll look at that in the next installment.

Are you a tester—solo or in a group?  Or are you a developer, manager, business person, documenter, support person, or someone in DevOps who wants to get very good at testing?  Attend Rapid Software Testing in Seattle, presented by James Bach and me, September 26-28, 2018.  Sign up!

Exploratory Testing on an API? (Part 1)

July 9th, 2018

In one forum or another recently, a tester, developer or DevOps person (I wasn’t sure which of these he/she was) asked:

Do you perform any exploratory testing on APIs? How do you do it?

Here’s an in-depth version of the reply I gave, with the benefit of links to deeper material.

To begin: application programming interfaces (APIs) are means by which we can use software to send commands to a product to make it do something. We do perform some testing on APIs themselves; interfaces represent one set of dimensions or factors or elements of the product. In a deeper sense, we’re not simply testing the APIs; we’re using them to control and observe the product, so that we can learn lots of things about it.

Now, there’s a problem with the first question: it implies that there is some kind of testing that is not exploratory. But all testing is exploratory. The idea that testing isn’t exploratory comes, in part, from a confusion between demonstrating that a product can work and testing to learn how it does work, and how it might not work.

When a product—a microservice, a component, an application framework, a library, a GUI application,…—offers an API, we can use tools to interact with it. We develop examples of how the product should work, and use the API to provide an appropriate set of inputs to the product, operating and observing it algorithmically and repetitively, which affords a kind of demonstration.

If we observe the behaviours and outputs that we anticipated and desired, we declare the demonstration successful, and we conclude that the product works. If we chose to speak more carefully, we would say that the product can work. If there has been some unanticipated, undesirable change in the product, our demonstration may go off the rails. When it does, we suspect a bug in the product.

It is tempting to call demonstrations “testing”, but as testing goes, it’s pretty limited.  To test the product, we must challenge it; we must attempt to disprove the idea that it works. The trouble is that people often assume that demonstration is all there is to testing; that checking and testing are the same thing.

In the Rapid Software Testing namespace, testing is evaluating a product by learning about it through exploration and experimentation, which includes to some degree: questioning, study, modeling, observation, inference, and plenty of other stuff.

Testing often includes checking as a tactic. Checking is the process of evaluating specific factors of the product by applying algorithmic decision rules to specific observations of a product, and then algorithmically indicating the results of those decision rules. Checking is embedded in testing.

Testing is fundamentally an exploratory process.  An exploratory process might include demonstration from time to time, and some demonstrations can be done with checking.  Where do checks come from? They come from testing!

As you’re first encountering an API, you are learning about it. There is documentation in some form that describes the functions, how to provide input, and what kinds of output to anticipate. There may be a description of how the product interacts with other elements of some larger system. The documentation probably includes some new terminology, and some examples of how to use the API. And it probably contains an error or two. Making sense of that documentation and discovering problems in it is an exploratory process.

You perform some experiments to see the API in action, maybe through a scripting language (like Python or Ruby) or maybe through a tool (like Postman). You provide input, perform some API calls, examine the output, and evaluate it. You might experience confusion, or bump into an inconsistency between what the service does and what the documentation says it does. Wrapping your head around those new things, and stumbling over problems in them, is an exploratory process.

You may have heuristics, words and patterns and techniques that guide your learning. Still, there’s no explicit script to read or set procedure to follow when you’re learning about an API. Learning, and discovering problems in the product as you learn, is an exploratory process.

As you learn about the product, you begin to develop ideas about scenarios in which your product will be used. Some products perform single, simple, atomic functions. Others offer more elaborate sets of transactions, maintaining data or state between API calls. You will imagine and experiment with the conversations that programmers and other products might have with your product. Developing models of how the product will be used is an exploratory process.

As you learn more about the product, you begin to anticipate risks; bad things that might happen and good things that might not happen. You may have heuristics that help you to develop risk models generally, but applying heuristics to develop a product-specific risk model is an exploratory process.

As you learn about risks, you develop strategies and ideas for testing for them.  Developing strategies is an exploratory process.

You try those test ideas. Some of those test ideas reveal things that you anticipated. Some of them don’t. Why not? You investigate to find out if there’s a problem with the API, with your test idea, or with the implementation of your test idea. Interpreting results and investigating potential bugs is an exploratory process.

As your learning deepens, you identify specific factors to be checked. You design and write code to check those factors. Developing those checks, like all programming, isn’t simply a matter of getting an idea and converting it into code. Developing checks is an exploratory process.

You may choose to apply those checks entirely in a repetitious, confirmatory way. On the other hand, every now and then — and especially when a new API call is being introduced or modified — you could induce variation, diversification, randomization, aggregation, data gathering, analysis, visualization, experimentation, and investigation into your work. Refreshing your strategies and tactics is an exploratory process.

In other words, practically everything that we do in testing a product except the mechanistic running of the checks is exploratory work. “Do you do any exploratory testing on APIs?” is equivalent to asking “Given a product with an API, do you do testing?”

The answer is, of course, Yes.  How do you do it?  Find out in Part 2.

Are you a tester—solo or in a group?  Or are you a developer, manager, business person, documenter, support person, or someone in DevOps who wants to get very good at testing?  Attend Rapid Software Testing in Seattle, presented by James Bach and me, September 26-28, 2018.  Sign up!

Test Cases and Coverage

June 28th, 2018

A tester recently asked about creating an in-house testing process, wanting to know how to start writing test cases so that testing effort could be estimated. My reply—which could just as easily apply to a developer or business analysts in a testing role—went something like this:

Test cases are not testing!  While that’s true, just saying so probably won’t help you very much, so let me offer an alternative to thinking about your assignment in terms of test cases.

At the beginning of the project, you don’t yet know how to test it. To develop a relevant test strategy, you must learn the product. There may be many ways for you to learn about the product—by attending meetings, talking to people, reviewing project plans, studying designs or sketches. You may even have an opportunity to work with the product directly; with a previous version; with a mockup or prototype; or with a part of the product.

Without a good deal of that learning, you cannot know what test cases should describe or contain—so instead of the typical fine granularity of test cases, consider the broader notion of coverage areas.

As you learn about a product or feature, create a coverage outline based on product factors. (A product factor is something that could be examined in a test, or something that could affect the outcome of a test.)  Your outline may take one or more of many forms—a hierarchical list, a table, an annotated diagram; these days we like mind maps.  The outline identifies areas of the product and its relationship to its context that can be covered with testing activity. In general, these areas will tend to fall into categories like structure, functions, data, interfaces, platforms, operations, and time.

You might want to note any specific factors or conditions that you believe must be checked—but since you’re at the beginning of the project, also note that no one can be aware of all of them yet.

While you’re at it, begin advocating for testabilty; things that can make testing faster and easier. A product tends to be faster and easier to test when developers provide controllability through scriptable interfaces (APIs); visibility through log files; and code that they have already tested for basic problems and coding errors. When programmers do more of their own testing (including automated checks), testers won’t have to investigate and report on shallow bugs—which leaves more time for deep testing.

Your coverage outline goes hand in hand with a risk list—in which you identify technological and business risks that will influence your strategies for covering the test space. As development progresses, we learn about potential problems that threaten the value of the product. It’s a special responsibility of someone in the testing role to explore ideas about problems and where they might be lurking; to perform experiments to identify problems; and to determine the nature of those problems so that developers and managers can make informed decisions on what to do about them.

Consider the kinds of activities that testers perform: making contact with the team; learning about the product; analyzing how to test it; preparing equipment and tools; developing ideas for deep coverage; doing deep testing; and winding down the testing and helping prepare the product for deployment.  Then, instead of estimating by test cases, or by hours or days, consider estimating by sessions.

Sessions are bursts of nominally uninterrupted testing work, guided by a mission statement or charter. The time span of a session is intentionally imprecise, and we say “nominally uninterrupted” because, in the real world, it’s unusual for work not to get interrupted.  When planning a session, we base its charter on the idea that it will take around 90 minutes.  In reality, a session may last from 45 minutes to two and a half hours.  That’s because we can identify what we’d like to happen in a session, but our knowledge of what will happen is uncertain.

For instance, the bugs that you encounter during a session will affect how much you’re able to cover during the session. When bugs require investigation and reporting, coverage suffers. So, after the session, the charter may change to reflect what actually happened.  The difference between what you hope to do and what you’ll be able to do can’t be known very well in advance—so guess how many sessions you’ll need to cover the stuff that you know about.  Then double your guess.

What will happen over the course of a collection of sessions is uncertain too. You’ll need to allocate plenty of time to account for setting up for each session; dealing with obstacles; debriefing; debugging with developers; planning meetings; breaks and lunch; and so forth. As with the specifics of what you’re going to cover, that stuff is uncertain too.  However, it’s reasonable to assume, given the non-session work, a tester will have time for two or three sessions at most in a day.

In the early stages, you’re going to have to guess about a lot of this. That’s normal; testing is about exploration, discovery, investigation, and learning.  Those things aren’t calculable except in the most general way, and work tends to expand or shrink to fit the time available for it. Recognize that little of your early planning will play out as anticipated.

As you go through loops of learning and testing, your coverage outline and risk list will evolve. Commit to providing daily reports on what is being learned; and on the gap between what has been covered and what has not yet been covered—product coverage and risk coverage. Daily reports will keep managers alert to problems that need their attention—problems that might threaten the value of the product or the success of the project.  Managers need to know about those, not about test case counts.

As a key element in your reporting, provide notes on testability and its absence—anything that makes the product easier or harder to test. Reduced testability means testing takes longer or reduces coverage.  Reduced coverage means bugs will survive undetected for longer—maybe into production.

Every day, as you learn about the product and update your models of it, you will be informing your clients on what testing has been done, and what testing could be done.  In the end, your clients will make the decision about when they have sufficient information about coverage and risk to ship the product.  It’s your job to keep them up to date and well-informed; it’s their job to decide whether the product they’ve got is the product they want.

All that said, you might not find it necessary to estimate the amount of testing that the project needs. Since both you and the client already have an idea of what the schedule and budget will be for this project anyway (based on past experience and other factors), your estimate may already be done for you. If not, and you’re asked to provide and estimate, don’t provide one; provide three: a reasonable estimate; a safer estimate; and a riskier estimate.

I anticipate someone will respond “all this sounds harder than preparing a list of test cases.” I don’t believe it is—I’d say it’s easier, in fact. A coverage outline is certainly easier to maintain than a library of test cases.  Working from an outline acknowledges the fact that we’ll learn and refine our plans as we go, where the test case approach turns us into ostriches when it comes to anticipating change.

Some people will respond “But managers want test cases!” Yes, some do; and kids want candy instead of real food, and addicts want illegal drugs, too. We are not obliged to do harmful things, and fixation on test cases puts more emphasis on documentation and procedures than on actual testing. I believe most managers want test cases because they’re used to them, and because they’re not aware of alternatives. Testers provide test cases because managers keep asking about them. I’d offer real testing work, continuous collaboration, refinement of the tasks at hand—and solid information about coverage, problems, and risks.

Related reading:

No Test Cases Required (PowerPoint presentation)
Questioning Test Cases
Testers Learn, but Test Cases Don’t Teach

Rapid Software Testing is designed for managers, developers, testers, and teams who want help with making testing faster, easier, cheaper, and above all, effective. Read about RST classes here, and consider the upcoming public class taught by both James Bach and Michael Bolton.

Which Test Cases Should I Automate?

June 15th, 2018

When someone asks “I have a big suite of manual tests; which tests (or worse, which test cases) should I automate?”, I often worry about several things.

The first thing is that focusing on test cases is often a pretty lousy way to think about testing.  That’s because test cases are often cast in terms of following an explicit procedure in order to observe a specific result.  At best, this confirms that the product can work if someone follows that procedure, and it also assumes that any result unobserved in the course of that procedure and after it is unimportant.

The trouble is that there are potentially infinite variations on the procedure, and many factors might make a difference in a test or in its outcome. Will people use the product in only one way? Will this specific data expose a problem?  Might other data expose a problem that this data does not?  Will a bug appear every time we follow this procedure? The test case often actively suppresses discoveryTest cases are not testing , and bugs don’t follow the test cases.

Second: testing is neither manual nor automated. A test cannot be automated.  Elements of the procedure within the test (in particular, checks for specific facts) can be automated.  But your test is not just the machine performing virtual key presses or comparing an output to some reference.

Your test is a process of activity and reasoning:  analyzing risk; designing an experiment; performing the experiment; observing what happens before, during, and after the experiment; interpreting the results; and preparing and a relevant report.  This depends on your human intentions, your mindset, and your skill set.  Tools can help every activity along the way, but the test is something you do, not something the machine does.

Third:  lots of existing test cases are shallow, pointless, out of date, ponderous, inefficient, cryptic, and unmotivated by risk.  Often they are focused on the user interface, a level of the product that is often quite unfriendly to tools. Because the test cases exist, they are often pointlessly repeated, long after they have lost any power to find a bug.  Why execute pointless test cases more quickly?

It might be a much better idea to create new automated checks that are focused on specific factors of the product, especially at low levels, with the goal of providing quick feedback to the developer.  It might be a good idea to prepare those checks as a collaboration between the developer and the tester (or between two developers, one of whom is in the builder’s role, with the other taking on a testing role). It might be a good idea to develop those checks as part of the process of developing some code. And it might be a really good idea to think about tools in a way that goes far beyond faster execution of a test script.

So I encourage people to reframe the question.  Instead of thinking what (existing) test cases should I automate? try thinking:

  • What reason do we have for preserving these test cases at all? If we’re going to use tools effectively, why not design checks with tools in mind from the outset?  What tool-assisted experiments could we design to help us learn about the product and discover problems in it?
  • What parts of a given experiment could tools accelerate, extend, enhance, enable, or intensify?
  • What do we want to cover? What product factors could we check? (A product factor is something that can be examined during a test, or that might influence the outcome of a test.)
  • To what product factors (like data, or sequences, or platforms,…) could we apply tools to help us to induce variation into our testing?
  • How could tools help us to generate representative, valid data and exceptional or pathological data?
  • In experiments that we might perform on the product, how might tools help us make observations and recognize facts that might otherwise escape our notice?  How can tools make invisible things visible?
  • How can tools help us to generate lots of outcomes that we can analyze to find interesting patterns in the data?  How can tools help us to visualize those outcomes?
  • How could the tool help us to stress out the product; overwhelm it; perturb it; deprive it of things that it needs?
  • How can developers make various parts of the system more amenable to being operated, observed, and evaluated with the help of tools?
  • How might tools induce blindness to problems that we might be able to see without them?
  • What experiments could we perform on the product that we could not perform at all without help from tools?

Remember:  if your approach to testing is responsible, clever, efficient, and focused on discovering problems that matter, then tools will help you to find problems that matter in an efficient, clever, and responsible way.  If you try to “automate” bad testing, you’ll find yourself doing bad testing faster and worse than you’ve ever done it before.

More reading:
A Context-Driven Approach to Automation in Testing
Manual and Automated Testing
The End of Manual Testing

Learn about Rapid Software Testing

Sign up for the RST class, taught by both James Bach and Michael Bolton, in Seattle September 26-28, 2018.  It’s not just for testers!  Anyone who wants to get better at testing is welcome and encouraged to attend—developers, managers, designers, ops people, tech support folk…

Very Short Blog Posts (36): Positive, Negative, and Sympathetic Testing

May 12th, 2018

In Rapid Software Testing, a “positive test” is one that honours every required and explicitly declared condition or factor for a desired outcome. A “negative test” is one that violates (or dishonours, disrespects, ignores, omits, undermines…) at least one required and explicitly declared condition.

That said, we don’t talk very much about positive and negative testing. We do talk about “sympathetic testing“, a closely related idea. Sympathetic testing helps us to learn about the product and how people would obtain value from it. Rich models derived from that learning help us to focus on finding more important bugs.

“Positive testing” is really about demonstration or confirmation; showing that the product can work. It might be a good idea to situate a fair amount of positive testing in a suite of maintainable automated checks, to cover conditions in which people will do exactly what they’re supposed to do at just the right time.

That may be all right as far as it goes, but it doesn’t fit very well with the real world. In the real world, people make mistakes. They get confused, become curious, forget stuff, experience frustration, feel impatient, act mischievously… They do things that the product’s design does not anticipate or specify, and they do so accidentally, incidentally, inadvertently, inventively, desperately, obliviously, intentionally, maliciously,… They don’t follow “the rules”—which aren’t really rules at all; just someone’s hopes or desires that everyone will behave themselves.

People will violate at least one explicitly required condition a lot more often than you think. So, as testers, it’s crucial for us to challenge the product and vary our tests to find out how it responds.

Excellent testing isn’t limited to showing the the product can produce an expected result. Excellent testing includes learning what the product does when people do something unexpected.

Rapid Software Testing in Toronto, June 4-6

May 3rd, 2018

I don’t usually give upcoming events an entire blog post. I usually post upcoming events in the panel on the right. However, this one’s special.

I’m presenting a three-day Rapid Software Testing class, organized by TASSQ in Toronto, June 4-6, 2018.

Rapid Software Testing is focused on doing the fastest, least expensive testing that still completely fulfills testing’s mission: evaluating the product by learning about it through exploration and experimentation. Developers and managers—including those from organizations where there are few testers or none at all—are welcome and warmly invited. It’s the only public RST class being offered in Canada in 2018. Early bird pricing is available until Friday, May 4. Register here.

How Long Will the Testing Take?

April 27th, 2018

Today yet another tester asked “When a client asks ‘How long will the testing take for our development project?’, how should I reply?”

The simplest answer I can offer is this: if you’re testing as part of a development project, testing will take exactly as long as development will take. That’s because effective, efficient testing is not separate from development; it is woven into development.

When we develop a software product or service, we build things: designs, prototypes, functions, modules, components, interfaces, integrated components, services, complete systems… There is some degree of risk—the possibility of problems—associated with everything we build. To be successful, each element of a system must fit with other elements of the system, and must satisfy the people who are using and building the system.

Despite our best intentions, we’re likely to make some mistakes and experience some misunderstandings along the way. If we review things as we go, we’ll eliminate many problems before we turn them into code. Still, there’s risk in using elements when we haven’t built, configured, operated, observed, and evaluated them. Unless we actually try to use what we’ve built, look for problem in it, and develop a well-grounded, empirical understanding of it, we risk fooling ourselves about how it really works—and doesn’t.

So it’s a good idea to start testing things as developer are creating or assembling them—and to test how they fit with other elements of the system—from the moment we start to build them or put them together.

Testing, for a given element or system, stops when we have a reasoned belief that there is no more development work to be done. Testing stops when people are satisfied that they know enough about the system and its elements to have discovered and resolved all the important problems. That satisfaction will be easier to establish relatively quickly, for any given element or system, when the product is designed to be easy to test. If you’ve been testing each part of the product deeply, investigating risk, and addressing problems throughout development, the whole product will be easier to test.

For the project to be successful, it’s important for the whole team to keep discovering and identifying ways in which people might be dissatisfied. That includes not only finding problems in the product, but also finding problems in our concept of what it might mean to be “done”. It’s not the tester’s job to build confidence in the product, but to reveal where confidence is unwarranted. That’s central to testing work, even though it’s difficult, disappointing, and to some degree socially disruptive.

Asking how long testing will take is an odd question for managers to ask, because it’s just like asking how long management will take. Management of a development project starts as development starts, and ends as development ends. Testing enables awareness about the product and problems in it, so that managers and developers can make decisions about it. So testing starts when the project starts, and testing stops when managers and developers have made their final decisions about it. Testing doesn’t stop until development and management of the project is done.

People often stop testing a product after it’s been put into production. Now: it might be a good idea to monitor and test some aspects of the system after it’s been put into production, too. (We call that live-site testing.) Live-site testing is often a very good idea, but like all other forms of testing, ultimately, it’s optional. Here’s one good reason to continue live-site testing: when you believe that there will be more development work done on the system.

So: development and management and testing go hand in hand. When a manager asks “How long will the testing take?”, it seems to me that the appropriate answer is “Testing will take just as long as development will take.” When people are satisfied that there’s no more important development work to do, they will also be satisfied that there’s no more important testing work to do either.

It’s important to remember, though: determining when people will be satisfied is something that we can guess, but cannot calculate. Satisfaction is a feeling, not a finish line.

Further reading:

How Long Will The Testing Take?
Project Estimation and Black Swans (Part 1)
Project Estimation and Black Swans (Part 5): Test Estimation

Very Short Blog Posts (35): Make Things Visible

April 24th, 2018

I hear a lot from testers who discover problems late in development, and who get grief for bringing them up.

On one level, the complaints are baseless, like holding an investigate journalist responsible for a corrupt government. On the other hand, there’s a way for testers to anticipate bad news and reduce the surprises. Try producing a product coverage outline and a risk list.

A product coverage outline is an artifact (a mind map, or list, or table) that identifies factors, dimensions, or elements of a product that might be relevant to testing it. Those factors might include the product’s structure, the functions it performs, the data it processes, the interfaces it provides, the platforms upon which it depends, the operations that people might perform with it, and the way the product is affected by time. (Look here for more detail.) Sketches or diagrams can help too.

As you learn more through deeper testing, add branches to the map, or create more detailed maps of particular areas. Highlight areas that have been tested so far. Use colour to indicate the testing effort that has been applied to each area—and where coverage is shallower.

A risk list is a list of bad things that might happen: Some person(s) will experience a problem with respect to something desirable that can be detected in some set of conditions because of a vulnerability in the system. Generate ideas on that, rank them, and list them.

At the beginning of the project or early as possible, post your coverage outline and risk list in places where people will see and read them. Update it daily. Invite questions and conversations. This can help you change “why didn’t you find that bug?” to “why didn’t we find that bug?”

Very Short Blog Posts (34): Checking Inside Exploration

April 23rd, 2018

Some might believe that checking and exploratory work are antithetical. Not so.

In our definition, checking is “the algorithmic process of operating and observing a product, applying decision rules to those observations, and reporting the outcome of those decision rules”.

We might want to use some routine checks, but not all checks have to be rote. We can harness algorithms and tools to induce variation that can help us find bugs. Both during development of a feature and when we’re studying and evaluating it, we can run checks that use variable input; in randomized sequences; at varying pace; all while attempting to stress out or overwhelm the product.

For instance, consider inducing variation and invalid data into checks embedded in a performance test while turning up the stress. When we do that, we can discover when the product falls to its knees under how much load—how the product fails, and what happens next. That in turn affords the opportunity to find out whether the product deals with overhead associated with error handling—which may result in feedback loops and cascades of stress and performance problems.

We can use checks as benchmarks, too. If a function takes significantly and surprisingly more or less time to do its work after a change, we have reason to suspect a problem.

We could run checks in confirmatory ways, which, alas, is the only way most people seem to use them. But we can also design and run checks taking a disconfirmatory and exploratory approach, affording discovery of bugs and problems. Checking is always embedded in testing, which is fundamentally exploratory to some degree. If we want to find problems, it would be a good idea to explore the test space, not just tread the same path over and over.