Blog Posts for the ‘Exploratory Testing’ Category

Exploratory Testing on an API? (Part 4)

Wednesday, November 21st, 2018

As promised, (at last!) here are some follow-up notes on previous installments in the series that starts here.

Let’s revisit the original question:

Do you perform any exploratory testing on APIs? How do you do it?

To review: there’s a problem with the question. Asking about “exploratory testing” is a little like asking about “vegetarian cauliflower”, “carbon-based human beings”, or “metallic copper”. Testing is fundamentally exploratory. Testing is an attempt by a self-guided agent to discover something unknown; to learn something about the product, with a special focus on finding problems and revealing risk.

One way to learn about the product is to develop and run a set of automated checks that access the product via the API. That is, a person writes code the machine to operate and observe the product, apply decision rules, and report the outputs. This produces a script, but it’s not a scripted process. Learning about the product, designing the checks, and developing the code to implement them checks are all exploratory processes.

When we use a machine to perform automated checks, there’s no discovery or learning involved in the performance of the check itself. Checks are like the system on your car that monitors certain aspects of your engine and illuminates the “check engine” light. Checks are, in essence, tools by which people become aware of specific conditions in the product. The machine learns no more than the dashboard light learns. It’s the humans, aided by tools that might include checking, who learn.

People learn as they interpret outcomes of the checks and investigate problems that the checks seem to indicate. The machinery has no way of knowing whether a reported “failed” output represent a problem in the product or a problem in the check. Checks don’t know the difference between “changed” and “broken”, between “different” and “fixed”, between “good” and “bad”. The machinery also has no way of knowing whether a reported “passed” output really means that the product is trouble-free; a problem in the check could be masking a problem in the product.

Testing via an API is exploratory testing via an API, because exploratory testing is, simply, testing. (Exploratory) testing not simply “acting like a user”, “testing without tools”, or “a manual testing technique”.

Throughout this whole series, what I’ve been doing is not “manual testing”, and it’s not “automated testing” either. I use tools to get access to the product via the API, but that’s not automated testing. There is no automated testing. Testing is neither manual nor automated. No one talks about “automated” or “manual” experiments. No one talks about “manual” or “automated” research. Testing is done neither by the hands, nor by the machinery, but by minds.

Tools do play an important role in testing. We use tools to extend, enhance, enable, accelerate, and intensify the testing we do. Tools play an important role in programming too, but no one refers to “manual programming”. No one calls compiling “automated programming”. Compiling is something that a machine can do; it is a translation between on set of strings (source code) and another set of strings (machine code). This is not to dismiss the role of the compiler, or of compiler writers; indeed, writing a sophisticated compiler is a job for an advanced programmer.

Programming starts in the complex, imprecise, social world of humans. Designers and programmers repair messy human communication into a form that’s so orderly that a brainless machine can follow the necessary instructions and perform the intended tasks. Throughout the development process, testers explore and experiment with the product to find problems, to help the development team and the business to decide whether the product they’ve got is the product they want. Tools can help, but none these processes cannot be automated.

In the testing work that I’ve described in the previous posts, I haven’t been “testing like a user”. Who uses APIs? It might be tempting to answer “application programs” (it’s an application programming interface, after all), or “machines”. But the real users of an API are human beings. These include the direct users—the various developers who write code to take advantage of what a product offers—and the indirect users—the people who use the products that programmers develop. For sure, some of my testing has been informed by ideas about actions of users of an API. That’s part of testing like a tester.

In several important ways, there’s a lot of opportunity for testability through APIs. Very generally, components and services with APIs tend to be of a smaller scale than entire applications, so studying and understanding them can be much more tractable. An API is deterministic and machine-specific. That means means that certain kinds of risks due to human variability are of lower concern than they might be through a GUI, where all kinds of things can happen at any time.

The API is by definition a programming interface, so it’s natural to use that interface for automated checking. You can use validator checks to detect problems with the syntax of the output, or parallel algorithms to check the semantics of transactions through an API.

Once they’re written, it’s easy to repeat such checks, especially to detect regressions, but be careful. In Rapid Software Testing, regression testing isn’t simply repetition of checks. To us, regression testing means testing focused on risk related to change; “going backwards” (which is what “regress” means; the opposite of “progress”).

A good regression testing strategy is focused on what has changed, how it has changed, and what might be affected. That would involve understanding what has changed; testing the change itself; exploring around that; and a smattering of testing of stuff that should be unaffected by the change (to reveal hidden or misunderstood risk). This applies whether you are testing via the API or not; whether you have a set of automated checks or not; whether you run checks continuously or not.

If you are using automated checks, remember that they can help to detect unanticipated variations from specified results, but they don’t show that everything works, and they don’t show that nothing has broken. Instead, checks verify that output from given functions are consistent from one build to the next. Do not simply confirm that everything is OK; actively search for problems. Explore around. Are all the checks passing? Ask “What else could go wrong?”

Automated checks can take on special relevance when they’re in the form of contract testschecks. The idea here is to solicit checks from actual consumers of an API that represent specified, desired results, and to check the contract from both the supplier and consumer ends. Nonetheless, remember that such checks are heavily focused on confirmation, and not on discovery of problems and risks that aren’t covered by the contracts.

On the other hand, now that you’ve gone to the trouble of writing code to check for specific outputs, why stop there? I’ve used checks in an exploratory way by:

  • varying the input systematically to look for problems related to missing or malformed data, extreme values, messed-up character handling, and other foreseeable data-related bugs;
  • varying the input more randomly (“fuzzing” is one instance of this technique), to help discover surprising hidden boundaries or potential security vulnerabilities;
  • varying the order and sequences of input, to look for subtle state-related bugs;
  • writing routines to stress the product, pumping lots of transactions and lots of data through it, to find performance-related bugs;
  • capturing data (like particular values) or metadata (like transaction times) associated with the checks, visualizing it, and analyzing it, to see problems and understand risks in new ways.

A while back, Peter Houghton told me an elegant example of using checking in exploration. Given an API to a component, he produces a simple script that calls the same function thousands of times from a loop and benchmarks the time that the process took. Periodically he re-runs the script and compares the timing to the first run. If he sees a significant change in the timing, he investigates. About half the time, he says, he finds a bug.

So, to sum up: all testing is exploratory. Exploration is aided by tools, and automated checking is an approach to using tools. Investigation of the unknown and discover of new knowledge is of the essence of exploration. We must explore to find bugs. All testing on APIs is exploratory.

Exploratory Testing on an API? (Part 3)

Monday, July 30th, 2018

In the last two posts, I’ve been answering the question

Do you perform any exploratory testing on an API? How do you do it?

Last time I described the process of learning about factors in the project environment that affect my strategy as I go about the testing of an API. This time, I’ll go deeper, learn more, and begin to interact with the product. That learning and interaction feeds back into more learning and more interaction.

Along the way, I’ll develop ideas that are reflected in the Heuristic Test Strategy Model. I get ideas about potential problems and risks based on Quality Criteria (and things that threaten them). I identify Product Factors and get ideas about coverage—how to the cover the product with testing. I apply those ideas using approaches that come from families of Test Techniques.

Notice, though, that even though I’ve tagged them, these ideas don’t come to me from the HTSM as such. Ideas occur to me as I encounter elements of the product and its context. Each interaction with and observation of the product triggers ideas about risk and how to test for it. Some of these ideas come from my testing and development experience; others from stories told by my colleagues or reported in the news; still others from the product and its documentation; and those all lead to new ideas that are synthesized in the moment.

You may find that this post is long. That length reflects the volume of ideas that occur in an exploratory process.

Whether they are end users—getting access to the product through the GUI—or developers—getting access to the product through an API—people use software to make things happen. They have tasks that they want to get done. So what can I imagine the API’s user wanting to do? Product Factors/Operations; Test Techniques/Scenario Testing.

An API’s user wants to get some data, make something happen, or effect some change. Anything that happens or changes in the product is driven by specific functions in the code. What functions does the product perform? Product Factors/Functions I find out by looking at the product and API documentation if it’s available, or by looking at source code if that’s available. Project Environment/Information If I can’t get to either one, I might ask developers or designers for help. Project Environment/Developer Relations

I take notes about what I’m learning, perhaps in the form of a mind map; perhaps in point-form text; perhaps in a table. The process of taking notes helps to cement what I’m learning. In addition, I want to notice inconsistencies between my sources of information. Sometimes different people tell me the different things. Some people contradict what is written in the requirements documents or specifications. Whenever those things happen, bugs are breeding. I’ll collect lots of specific statements about the product should do, and look for controversy. Test Techniques/Claims Testing

I’ll review what I already know. To what functions does the API offer access? The user of the API wants to obtain some kind of value from interacting with the product, and those values are expressible in terms of quality criteria. Quality Criteria/Capability What could go wrong with those functions and threaten something do do with one of those quality criteria?

What are the protocols for calling those functions? Product Factors/Interfaces Is there anything, like setting up or logging in, that should happen before an API call? What if they don’t happen? I’ll make a note to try that later. Test Techniques/Flow Testing “Before” reminds me to ask myself about “after”, too: Are there there things to be cleaned up after an API call? Product Factors/Time Does data need to be saved or purged?

Given a set of available functions, I’ll try to use a small selection of the functions that I’m aware of. Test Techniques/Function Testing; Test Techniques/Claims Testing I’ll try some simple stuff using a tool that affords direct interaction; these days it would be Postman, or some quick calls from IRB (the Ruby interpreter). Does the product appear to do what it should? Can it work? Quality Criteria/Capability

I’ll start simple with representative inputs; then I’ll explore around the extents and limits that I’ve been told about or that are documented. Test Techniques/Domain Testing. I’ll put in some preposterously long strings, or large quantities, or deliberately malformed input. I’ll leave out mandatory fields in the call; put in extra fields that I’ve made up; provide letters where where only numbers are anticipated; feed in symbols or accented characters where only normal letters are anticipated. If it’s a Web app, I might focus on a smattering of characters that are associated with cross-site scripting, SQL injection, or other security risks. Quality Criteria/Security

If I do something that should trigger error handling, does error handling happen? Does the product return an error code? After that, does continue to do what it should do? Test Techniques/Claims Testing; Quality Criteria/Reliability/Robustness I pay special attention when the product appears to contradict the documentation or what someone has told me. Test Techniques/Claims Testing

At this point, if I find problems, I note them. Even if I don’t, I start to gather some ideas about how I might use tools and automated checks to vary the input and cover whole sets of possible inputs when it’s time for deep testing.

I pause and remember that that there’s a difference between the product doing what it should and appearing to do what it should. Every step of the way, I’d like to be able to observe as much as I can of the product’s state. Are there any special API function calls that I can use to examine or control the state of the product before, during, and after the usual calls? Quality Criteria/Development/Testability These may not be noted in the public documentation; I may have to ask developers about them Project Environment/Developer Relations or look at source code. Project Environment/Information

Thinking about testability like this reminds me to ask: Does the product offer logging? I have a look. Is the format of the logs consistent with other logs in the same system? Consistent log files will help to make my testing go more quickly and easily, which helps me to get to deeper testing. Quality Criteria/Development/Testability If there is no logging and there is resistance from the developers or from management to putting it in, I’ll keep that in mind and note it for the future, when I design automated checks; the logs for my checks may have to be more specific and detailed.

I’ve wandered from thinking about functions to testability, so I pull on the leash to bring my mind back to functions again. What services do the functions perform for the rest of the system? What are the bits and pieces of the system? Product Factors/Structure If I haven’t started one already, I’ll sketch a diagram of the product’s structure, and use that to help guide my understanding of the product. As my understanding develops, I’ll share the diagram with others to determine whether we’re all on the same page, and to help to maintain our awareness of the product’s structure. I might also post a copy up in a prominent place and invite people to let me know as soon as they see something wrong. Every now and again, I’ll deliberately review the diagram.

Thinking about the parts of the system reminds to to ask myself: Upon what parts of the whole system do those functions depend? In Rapid Software Testing, something that our product depends upon that is outside the control of our current project is called a platform. Product Factors/Platform.

Platforms might include the operating system, the browser, application frameworks, third-party libraries, stuff that was built in-house on another project. Platforms might also include hardware elements, too: displays, keyboards, mice, touchscreens, printers, or other peripherals. I’ll develop lists of platform dependencies, preparing myself for deeper coverage later.

I return to exercising the functions. Most functions do something to something: data. What data can I supply via the API? In what format? What responses will the API provide? How are responses structured? Product Factors/Data. Answers to these questions will influence my selection of tools for deeper testing. Project Environment/Equipment and Tools

Is the API providing access to a database? If so, and I’m focused on testing the functions in the API, I might be able use direct access to the database as an oracle. Test Techniques/Function Testing

Thinking about databases prompts me to think about stored data and its quality. I focus for a moment on the quality of the data, and about what data I might want to use or generate for automated checks. Test Techniques/Domain Testing; Test Techniques/Automated Checking If I want to examine risk related to data quality, going directly to the database might be more efficient than trying to get at the data through the API’s functions. Of course, I could do a smattering of both, for which I’ll need different tools. Project Environment/Equipment and Tools

Although I’m doing it in a somewhat more painstaking way, as I’m learning to use the API, I’m doing what one of its users—typically a developer—would do to learn about it. Test Techniques/User Testing Just like its intended user, I make mistakes—some intentionally, some quite unintentionally. How difficult might it be for a developer to use this API correctly? How easy might it be to use it incorrectly, or in a way that provides misleading, ambiguous, or confusing results? Quality Criteria/Usability Test Techniques/Risk-Based Testing

I imagine a set of programming tasks that I might want to accomplish using the API. At that same time, I imagine a person performing those tasks; a role. It’s quite possible that not all users of the API will be expert programmers, so it makes sense for me to create a “novice programmer” role. Test Techniques/User Testing What if I make the kinds of mistakes a novice programmer might make, and ask for something in a way that the API doesn’t support? What if I hand it pathological or malformed data; data structures with missing elements, dates in the wrong format, or characters that might be used in countries other than my own? Quality Criteria/Localizability Does the product report an error? Will the novice understand it? A potential misunderstanding might point to a problem in the product or in the documentation for it.

How well do things get cleaned up after an error condition? Are error conditions handled gracefully? Quality Criteria/Reliability/Error Handling How are error conditions relayed to the user of the API—the software calling it, and the human writing that software? Are there error codes to make troubleshooting easier? Quality Criteria/Development/Supportability? Human-readable strings or data structures? Are they consistent with the documentation? Quality Criteria/Usability. Are they reasonably elegant and efficient, or are they clunky, confusing, misspelled, or embarrassing in any way? Quality Criteria/Charisma

Some APIs are used for atomic interactions; single calls to a service followed by single responses from it. Other APIs work more like a conversation, retaining data or maintaining the product in a certain state from one call to the next. A conversation has its risks: what happens when I interrupt it? What happens if I leave out a particular call or response? Test Techniques/Flow Testing What if I varied the sequence of interactions in a conversation? Product Factors/Time Has anyone made any claims about timeouts, either informally or in the documentation? Test Techniques/Claims Testing

How long will the system let me leave a conversation dangling? Product Factors; Time What happens to the internal state of the system or to the data that’s being managed after a conversation has been abandoned for a while? What happens when I try to pick up a previously abandoned conversation? Quality Criteria; Reliability

Thinking about sequence leads me to think about time generally. How long does it take for the system to process an API call and return a result? Product Factors/Time Tools help make the round-trip time easily visible—but I don’t trust evaluating performance based on the first result. Because the tool makes it so easy, I often do several quick tests to observe results from the same call several times. If there’s substantial variation, I make a note of it; it might be a bug, or it might be a target for deeper testing.

Has anyone made explicit claims about performance? Project Environment/Information Even though I might refer to those claims, I don’t need them to do performance testing; I’ll operate the product, collect observations about performance, and include those observations in my report.

In addition to the API, there may also be human user interfaces: a GUI, or a command line interface. I’ll exercise those interfaces too, and look for inconsistencies between the way they behave and the way the API behaves. Product Factors/Interfaces Inconsistency within the product is a powerful oracle.

Considering the human interfaces reminds me of other interfaces. The API interacts with certain components in the product, and the product inevitably interacts with other products: the operating system, the file system, third-party frameworks and libraries, networks, printers, ports, Bluetooth… These interfaces afford access to the platforms upon which the product depends. Product Factors/Platform

What if one of those dependencies were missing? What if I deprived the of something it needs by preventing access to files or databases? Test Techniques; Stress Testing Would it stay up? Quality Criteria; Reliability

Thinking about reliability prompts me to think about how the product might behave when I place it under load. Quality Criteria; Performance Could I stress it out by overwhelming it with large amounts of data in specific fields, or with an extreme rate of calls? Test Techniques; Stress Testing I’ll definitely need tools for that. Project Environment/Equipment and Tools

I could use those tools to do the same checks over and over again, comparing the product’s output to a specified, hard-coded result. But I could broaden my coverage by varying the inputs for the checks, and using a parallel algorithm to compute the desired result. Exploring the product prepares me to identify those opportunities.

After a session or two of this kind of activity, I’ve learned a lot of useful stuff about the product. My learning allows me to make much better decisions about where I might want to apply automated checks to address particular risks; how I might want to induce variation in those checks; how I might want to extend ideas about checking.

Based on the length of this post, it may seem like this exercise is necessarily a long process. It doesn’t have to be. Code takes time to develop. Quick little informal experiments can fast and cheap. Ideas about risk, coverage, quality criteria, and oracles can be developed much more quickly and less expensively than code can. Writing about these things, as I’ve done here, takes far longer than thinking them through! Yet taking notes as I go often helps by inducing pauses in which more ideas occur to me.

This is testing! I’m not only testing to find bugs here; I’m testing to learn how to test the product. If I’m going to use automated checking as part of a strategy for learning specific things about the product quickly, I must test to learn how to develop those checks. This is an exploratory process! Notice how often each idea above is connected to the one before. Testing is fundamentally exploratory.

Do I do exploratory testing on an API?  Yes—because if I’m testing it, I’m doing exploratory work! If I’m doing anything other than that, it’s probably automated checking—and I can’t begin to do excellent checking until I’ve done that exploratory work.

I’ll have some follow-up notes in the final post in this series.

Exploratory Testing on an API? (Part 2)

Tuesday, July 17th, 2018

Summary:  Loops of exploration, experimentation, studying, modeling, and learning are the essence of testing, not an add-on to it. The intersection of activity and models (such as the Heuristic Test Strategy Model) help us to perform testing while continuously developing, refining, and reviewing it. Testing is much more than writing a bunch of automated checks to confirm that the product can do something; it’s an ongoing investigation in which we continuously develop our understanding of the product.

Last time out, I began the process of providing a deep answer to this question:

Do you perform any exploratory testing on APIs? How do you do it?

That started with reframing the first question

Do you perform any exploratory testing on APIs?

into a different question

Given a product with an API, do you do testing?

The answer was, of course, Yes. This time I’ll turn to addressing the question “How do you do it?” I’ll outline my thought process and the activities that I would perform, and how they feed back on each other.

Note that in Rapid Software Testing, a test is an action performed by a human; neither a specific check nor a scripted test procedure. A test is a burst of exploration and experiments that you perform. As part of that activity, a test include thousands of automated checks within it, or just one, or none at all. Part of the test may be written down, encoded as a specific procedure. Testing might be aided by tools, by documents or other artifacts, or by process models. But the most important part of testing is what testers think and what testers do.

(Note here that when I say “testers” here, I mean any person who is either permanently or temporarily in a testing role. “Tester” applies to a dedicated tester; a solo programmer switching from the building mindset to the tester mindset; or a programmer or DevOps person examining the product in a group without dedicated testers.)

It doesn’t much matter where I start, because neither learning nor testing happen in straight lines. They happen in loops, cycles, epicycles; some long and some short; nested inside each other; like a fractal. Testing and learning entail alternation between focusing and defocusing; some quick flashes of insight, some longer periods of reflection; smooth progress at some times, and frequent stumbling blocks at others. Testing, by nature, is an exploratory process involving conversation, study, experimentation, discovery, investigation that leads to more learning and more testing.

As for anything else I might test, when I’m testing a product through an API, I must develop a strategy. In the Rapid Software Testing namespace, your strategy is the set of ideas that guide the design, development, and selection of your tests.

Having the the Heuristic Test Strategy Model in my head and periodically revisiting it helps me to develop useful ideas about how to cover the product with testing. So as I continue to describe my process, I’ll annotate what I’m describing below with some of the guideword heuristics from the HTSM. The references will look like this.

A word of caution, though:  the HTSM isn’t a template or a script.  As I’m encountering the project and the product, test ideas are coming to me largely because I’ve internalized them through practice, introspection, review, and feedback.  I might use the HTSM generatively, to help ideas grow if I’m having a momentary drought; I might use it retrospectively as a checklist against which I review and evaluate my strategy and coverage ideas; or I might use it as a means of describing testing work and sharing ideas with other people, as I’m doing here.

Testing the RST way starts with evaluating my context. That starts with taking stock of my mission, and that starts with the person giving me my mission. Who is my client—that is, to whom am I directly answerable? What does my client want me to investigate?

I’m helping someone—my client, developers, or other stakeholders—to evaluate the quality of the product. Often when we think about value, we think about value to paying customers and to end users, but there are plenty of people who might get value from the product, or have that value threatened. Quality is value to some person who matters, so whose values do we know might matter? Who might have been overlooked? Project Environment/Mission

Before I do anything else, I’ll need to figure out—at least roughly—how much time I’ll have to accomplish the mission. While I’m at it, I’ll ask other time-related questions about the project: are there any deadlines approaching? How often do builds arrive? How much time should I dedicate to preparing reports or other artifacts? Project Environment/Schedule

Has anyone else tested this product? Who are they? Where are they? Can I talk to them? If not, did they produce results or artifacts that will help me? Am I on a team? What skills do we have? What skills do we need? Project Environment/Test Team

What does my client want to me to provide? A test report, almost certainly, and bug reports, probably—but in what form? Oral conversations or informally written summaries? I’m biased towards keeping things light, so that I can offer rapid feedback to clients and developers. Would the client prefer more formal appoaches, using particular reporting or management tools? As much as the client might like that, I’ll also note whenever I see costs of formalization.

What else might the client, developers, and other stakeholders want to see, now or later on? Input that I’ve generated for testing? Code for automated checks? Statistical test results? Visualizations of those results? Tools that I’ve crafted and documentation for them? A description of my perception of the product? Formal reports for regulators and auditors? Project Environment/Deliverables I’ll continue to review my mission and the desired deliverables throughout the project.

So what is this thing I’m about to test? Project Environment/Test Item Having checked on my mission, I proceed to simple stuff so that I can start the process of learning about the product. I can start with any one of these things, or with two or more of them in parallel.

I talk to the developers, if they’re available. Even better, I participate in design and planning sessions for the product, if I can. My job at such meetings is to learn, to advocate for testability, to bring ideas and ask questions about problems and risks. I ask about testing that the developers have done, and the checking that they’ve set up. Project Environment/Developer Relations

If I’ve been invited to the party late or not at all, I’ll make a note of it. I want to be as helpful as possible, but I also want to keep track of anything that makes my testing harder or slower, so that everyone can learn from that. Maybe I can point out that my testing will be better-informed the earlier and the more easily I can engage with the product, the project, and the team.

I examine the documentation for the API and for the rest of the product. Project Environment/Information I want to develop an understanding of the product: the services it offers, the means of controlling it, and its role in the systems that surround it. I annotate the documentation or take separate notes, so that I can remember and discuss my findings later on. As I do so, I pay special attention to things that seem inconsistent or confusing.

If I’m confused, I don’t worry about being confused. I know that some of my confusion will dissipate as I learn about the product. Some of my confusion might suggest that there are things that I need to learn. Some of my confusion might point to the risk that the users of the product will be confused too. Confusion can be a resource, a motivator, as long as I don’t mind being confused.

As I’m reading the documentation, I ask myself “What simple, ordinary, normal things can I do with the product?” If I have the product available, I’ll do sympathetic testing by trying a few basic requests, using a tool that provides direct interaction with the product through its API. Perhaps it’s a tool developed in-house; perhaps it’s a tool crafted for API testing like Postman or SOAPUI; or maybe I’ll use an interpreter like Ruby’s IRB along with some helpful libraries like HTTParty. Project Environment/Equipment and Tools

I might develop a handful of very simple scripts, or I might retain logs that the tool or the interpreter provides. I’m just as likely to throw this stuff away as I am to keep it. At this stage, my focus is on learning more than on developing formal, reusable checks. I’ll know better how to test and check the product after I’ve tried to test it.

If I find a bug—any kind of inconsistency or misbehaviour that threatens the value of the product—I’ll report it right away, but that’s not all I’ll report. If I have any problems with trying to do sympathetic testing, I’ll report them immediately. They may be usability problems, testability problems, or both at once. At this stage of the project, I’ll bias my choices towards the fastest, least expensive, and least formal reporting I can do.

My primary goal at this point, though, is not to find bugs, but to figure out how people might use the API to get access to the product, how they might get value from it, and how that value might be threatened. I’m developing my models of the product; how it’s intended to work, how to use it, and how to test it. Learning about the product in a comprehensive way prepares me to find better bugs—deeper, subtler, less frequent, more damaging.

To help the learning stick, I aspire to be a good researcher: taking notes; creating diagrams; building lists of features, functions, and risks; making mind maps; annotating existing documentation. Periodically I’ll review these artifacts with programmers, managers, or other colleagues, in order to test my learning.

Irrespective of where I’ve started, I’ll iterate and go deeper, testing the product and refining my models and strategies as I go. We’ll look at that in the next installment.

Exploratory Testing on an API? (Part 1)

Monday, July 9th, 2018

In one forum or another recently, a tester, developer or DevOps person (I wasn’t sure which of these he/she was) asked:

Do you perform any exploratory testing on APIs? How do you do it?

Here’s an in-depth version of the reply I gave, with the benefit of links to deeper material.

To begin: application programming interfaces (APIs) are means by which we can use software to send commands to a product to make it do something. We do perform some testing on APIs themselves; interfaces represent one set of dimensions or factors or elements of the product. In a deeper sense, we’re not simply testing the APIs; we’re using them to control and observe the product, so that we can learn lots of things about it.

Now, there’s a problem with the first question: it implies that there is some kind of testing that is not exploratory. But all testing is exploratory. The idea that testing isn’t exploratory comes, in part, from a confusion between demonstrating that a product can work and testing to learn how it does work, and how it might not work.

When a product—a microservice, a component, an application framework, a library, a GUI application,…—offers an API, we can use tools to interact with it. As we develop the product, we also develop examples of how the product should work. We use the API to provide an appropriate set of inputs to the product, operating and observing it algorithmically and repetitively, which affords a kind of demonstration.

If we observe the behaviours and outputs that we anticipated and desired, we declare the demonstration successful, and we conclude that the product works. If we chose to speak more carefully, we would say that the product can work. If there has been some unanticipated, undesirable change in the product, our demonstration may go off the rails. When it does, we suspect a bug in the product.

It is tempting to call demonstrations “testing”, but as testing goes, it’s pretty limited.  To test the product, we must challenge it; we must attempt to disprove the idea that it works. The trouble is that people often assume that demonstration is all there is to testing; that checking and testing are the same thing.

In the Rapid Software Testing namespace, testing is evaluating a product by learning about it through exploration and experimentation, which includes to some degree: questioning, study, modeling, observation, inference, and plenty of other stuff.

Testing often includes checking as a tactic. Checking is the process of evaluating specific factors of the product by applying algorithmic decision rules to specific observations of a product, and then algorithmically indicating the results of those decision rules. Checking is embedded in testing.

Testing is fundamentally an exploratory process.  An exploratory process might include demonstration from time to time, and some demonstrations can be done with checking. Where do checks come from? They come from testing!

As a tester first encountering an API, you are learning about it. There is documentation in some form that describes the functions, how to provide input, and what kinds of output to anticipate. There may be a description of how the product interacts with other elements of some larger system. The documentation probably includes some new terminology, and some examples of how to use the API. And it probably contains an error or two. Making sense of that documentation and discovering problems in it is an exploratory process.

You perform some experiments to see the API in action, maybe through a scripting language (like Python or Ruby) or maybe through a tool (like Postman). You provide input, perform some API calls, examine the output, and evaluate it. You might experience confusion, or bump into an inconsistency between what the service does and what the documentation says it does. Wrapping your head around those new things, and stumbling over problems in them, is an exploratory process.

You may have heuristics, words and patterns and techniques that guide your learning. Still, there’s no explicit script to read or set procedure to follow when you’re learning about an API. Learning, and discovering problems in the product as you learn, is an exploratory process.

As you learn about the product, you begin to develop ideas about scenarios in which your product will be used. Some products perform single, simple, atomic functions. Others offer more elaborate sets of transactions, maintaining data or state between API calls. You will imagine and experiment with the conversations that programmers and other products might have with your product. Developing models of how the product will be used is an exploratory process.

As you learn more about the product, you begin to anticipate risks; bad things that might happen and good things that might not happen. You may have heuristics that help you to develop risk models generally, but applying heuristics to develop a product-specific risk model is an exploratory process.

As you learn about risks, you develop strategies and ideas for testing for them.  Developing strategies is an exploratory process.

You try those test ideas. Some of those test ideas reveal things that you anticipated. Some of them don’t. Why not? You investigate to find out if there’s a problem with the API, with your test idea, or with the implementation of your test idea. Interpreting results and investigating potential bugs is an exploratory process.

As your learning deepens, you may identify specific factors to be checked. You design and write code to check those factors. Developing those checks, like all programming, isn’t simply a matter of getting an idea and converting it into code. Developing check code is an exploratory process.

You may choose to apply those checks entirely in a repetitious, confirmatory way. On the other hand, every now and then — and especially when a new API call is being introduced or modified — you could induce variation, diversification, randomization, aggregation, data gathering, analysis, visualization, experimentation, and investigation into your work. Refreshing your strategies and tactics is an exploratory process.

In other words, practically everything that we do in testing a product except the mechanistic running of the checks is exploratory work. “Do you do any exploratory testing on APIs?” is equivalent to asking “Given a product with an API, do you do testing?”

The answer is, of course, Yes. How do you do it?  Find out in Part 2.

Finding the Happy Path

Wednesday, February 28th, 2018

In response to yesterday’s post on The Happy Path colleague and friend Albert Gareev raises an important issue:

Until we sufficiently learned about the users, the product, and the environment, we have no idea what usage pattern is a “happy path” and what would be the “edge cases”.

I agree with Albert. (See more of what he has to say here.) This points to a kind of paradox in testing and development: some say that we can’t test the product unless we know what the requirements are—yet we don’t know what many of the requirements are until we’ve tested! Testing helps to reduce ambiguity, uncertainty, and confusion about the product and about its requirements—and yet we don’t know how to test until we’ve tried to test!

Here’s how I might address Albert’s point:

To test a whole product or system means more than demonstrating that it can work, based on the most common or optimistic patterns of its use. We might start testing the whole system there, but if we wanted to develop a comprehensive understanding of it, we wouldn’t stop at that.

On the other hand, the whole system consists of lots of sub-systems, elements, components, and interactions with other things. Each of those can be seen as a system in itself, and studying those systems contributes to our understanding of the larger system.

We build systems, and we build ideas on how to test them. At each step, considering only the most capable, attentive, well-trained users; preparing only the most common or favourable environments; imagining only the most optimistic scenarios; performing only happy-path testing on each part of the product as we build it; all of these present the risk of misunderstanding not only the product but also the happy paths and edge cases for the greater system. If we want to do excellent testing, all of these things—and our understanding of them—must not only be demonstrated, but must be tested as well. This means we must do more than creating a bunch of high-level, automated, confirmatory checks at the beginning of the sprint, and then declaring victory when they all “pass”.

Quality above depends on quality below; excellent testing above depends on excellent testing below. It’s testing all the way down—and all the way up, too.

The Honest Manual Writer Heuristic

Monday, May 30th, 2016

Want a quick idea for a a burst of activity that will reveal both bugs and opportunities for further exploration? Play “Honest Manual Writer”.

Here’s how it works: imagine you’re the world’s most organized, most thorough, and—above all—most honest documentation writer. Your client has assigned you to write a user manual, including both reference and tutorial material, that describes the product or a particular feature of it. The catch is that, unlike other documentation writers, you won’t base your manual on what the product should do, but on what it does do.

You’re also highly skeptical. If other people have helpfully provided you with requirements documents, specifications, process diagrams or the like, you’re grateful for them, but you treat them as rumours to be mistrusted and challenged. Maybe someone has told you some things about the product. You treat those as rumours too. You know that even with the best of intentions, there’s a risk that even the most skillful people will make mistakes from time to time, so the product may not perform exactly as they have intended or declared. If you’ve got use cases in hand, you recognize that they were written by optimists. You know that in real life, there’s a risk that people will inadvertently blunder or actively misuse the product in ways that its designers and builders never imagined. You’ll definitely keep that possibility in mind as you do the research for the manual.

You’re skeptical about your own understanding of the product, too. You realize that when the product appears to be doing something appropriately, it might be fooling you, or it might be doing something inappropriate at the same time. To reduce the risk of being fooled, you model the product and look at it from lots of perspectives (for example, consider its structure, functions, data, interfaces, platform, operations, and its relationship to time; and business risk, and technical risk). You’re also humble enough to realize that you can be fooled in another way: even when you think you see a problem, the product might be working just fine.

Your diligence and your ethics require you to envision multiple kinds of users and to consider their needs and desires for the product (capability, reliability, usability, charisma, security, scalability, performance, installability, supportability…). Your tutorial will be based on plausible stories about how people would use the product in ways that bring value to them.

You aspire to provide a full accounting of how the product works, how it doesn’t work, and how it might not work—warts and all. To do that well, you’ll have to study the product carefully, exploring it and experimenting with it so that your description of it is as complete and as accurate as it can be.

There’s a risk that problems could happen, and if they do, you certainly don’t want either your client or the reader of your manual to be surprised. So you’ll develop a diversified set of ways to recognize problems that might cause loss, harm, annoyance, or diminished value. Armed with those, you’ll try out the product’s functions, using a wide variety of data. You’ll try to stress out the product, doing one thing after another, just like people do in real life. You’ll involve other people and apply lots of tools to assist you as you go.

For the next 90 minutes, your job is to prepare to write this manual (not to write it, but to do the research you would need to write it well) by interacting with the product or feature. To reduce the risk that you’ll lose track of something important, you’ll probably find it a good idea to map out the product, take notes, make sketches, and so forth. At the end of 90 minutes, check in with your client. Present your findings so far and discuss them. If you have reason to believe that there’s still work to be done, identify what it is, and describe it to your client. If you didn’t do as thorough a job as you could have done, report that forthrightly (remember, you’re super-honest). If anything that got in the way of your research or made it more difficult, highlight that; tell your client what you need or recommend. Then have a discussion with your client to agree on what you’ll do next.

Did you notice that I’ve just described testing without using the word “testing”?

On Scripting

Saturday, July 4th, 2015

A script, in the general sense, is something that constrains our actions in some way.

In common talk about testing, there’s one fairly specific and narrow sense of the word “script”—a formal sequence of steps that are intended to specify behaviour on the part of some agent—the tester, a program, or a tool. Let’s call that “formal scripting”. In Rapid Software Testing, we also talk about scripts as something more general, in the same kind of way that some psychologists might talk about “behavioural scripts”: things that direct, constrain, or program our behaviour in some way. Scripts of that nature might be formal or informal, explicit or tacit, and we might follow them consciously or unconsciously. Scripts shape the ways in which people behave, influencing what we might expect people to do in a scenario as the action plays out.

As James Bach says in the comments to our blog post Exploratory Testing 3.0, “By ‘script’ we are speaking of any control system or factor that influences your testing and lies outside of your realm of choice (even temporarily). This does not refer only to specific instructions you are given and that you must follow. Your biases script you. Your ignorance scripts you. Your organization’s culture scripts you. The choices you make and never revisit script you.” (my emphasis, there)

When I’m driving to a party out in the country, the list of directions that I got from the host scripts me. Many other things script me too. The starting time of the party—combined with cultural norms that establish whether I should be very prompt or fashionably late—prompts me to leave home at a certain time. The traffic laws and the local driving culture condition my behaviour and my interactions with other people on the road. The marked detour along the route scripts me, as do the weather and the driving conditions. My temperament and my current emotional state script me too. In this more general sense of “scripting”, any activity can become heavily scripted, even if it isn’t written down in a formal way.

Scripts are not universally bad things, of course. They often provide compelling advantages. Scripts can save cognitive effort; the more my behaviour is scripted, the less I have to think, do research, make choices, or get confused. In my driving example, a certain degree of scripting helps me to get where I’m going, to get along with other drivers, and to avoid certain kinds of trouble. Still, if I want to get to the party without harm to myself or other people, I must bring my own agency to the task and stay vigilant, present, and attentive, making conscious and intentional choices. Scripts might influence my choices, and may even help me make better choices, but they should not control me; I must remain in control. Following a script means giving up engagement and responsibility for that part of the action.

From time to time, testing might include formal testing—testing that must be done in a specific way, or to check specific facts. On those occasions, formal scripting—especially the kind of formal script followed by a machine—might be a reasonable approach enabling certain kinds of tasks and managing them successfully. A highly scripted approach could be helpful for rote activities like operating the product following explicitly declared steps and then checking for specific outputs. A highly scripted approach might also enable or extend certain kinds of variation—randomizing data, for example. But there are many other activities in testing: learning about the product, designing a test strategy, interviewing a domain expert, recognizing a new risk, investigating a bug—and dealing with problems in formally scripted activities. In those cases, variability and adaptation are essential, and an overly formal approach is likely to be damaging, time-consuming, or outright impossible. Here’s something else that is almost never formally scripted: the behaviour of normal people using software.

Notice on the one hand that formal testing is, by its nature, highly scripted; most of the time, scripting constrains or even prevents exploration by constraining variation. On the other hand, if you want to make really good decisions about what to test formally, how to test formally, why to test formally, it helps enormously to learn about the product in unscripted and informal ways: conversation, experimentation, investigation… So excellent scripted testing and excellent checking are rooted in exploratory work. They begin with exploratory work and depend on exploratory work. To use language as Harry Collins might, scripted testing is parasitic on exploration.

We say that any testing worthy of the name is fundamentally exploratory. We say that to test a product means to evaluate it by learning about it through experimentation and exploration. To explore a product means to investigate it, to examine it, to create and travel over maps and models of it. Testing includes studying the product, modeling it, questioning it, making inferences about it, operating it, observing it. Testing includes reporting, which itself includes choosing what to report and how to contextualize it. We believe these activities cannot be encoded in explicit procedural scripting in the narrow sense that I mentioned earlier, even though they are all scripted to some degree in the more general sense. Excellent testing—excellent learning—requires us to think and to make choices, which includes thinking about what might be scripting us, and deciding whether to control those scripts or to be controlled by them. We must remain aware of the factors that are scripting us so that we can manage them, taking advantage of them when they help and resisting them when they interfere with our mission.

Exploratory Testing 3.0

Tuesday, March 17th, 2015

This blog post was co-authored by James Bach and me. In the unlikely event that you don’t already read James’ blog, I recommend you go there now.

The summary is that we are beginning the process of deprecating the term “exploratory testing”, and replacing it with, simply, “testing”. We’re happy to receive replies either here or on James’ site.

Very Short Blog Posts (12): Scripted Testing Depends on Exploratory Testing

Sunday, February 23rd, 2014

People commonly say that exploratory testing “is a luxury” that “we do after we’ve finished our scripted testing”. Yet there is no scripted procedure for developing a script well. To develop a script, we must explore requirements, specifications, or interfaces. This requires us to investigate the product and the information available to us; to interpret them and to seek ambiguity, incompleteness, and inconsistency; to model the scope of the test space, the coverage, and our oracles; to conjecture, experiment, and make discoveries; and to perform testing and obtain feedback on how the scripts relate to the actual product, rather than the one imagined or described or modeled in an artifact; to observe and interpret and report the test results, and to feed them back into the process; and to do all of those things in loops and bursts of testing activity. All of these are exploratory activities. Scripted testing is preceded by and embedded in exploratory processes that are not luxuries, but essential.

Related posts:

http://www.satisfice.com/blog/archives/856
http://www.developsense.com/blog/2011/05/exploratory-testing-is-all-around-you/
http://www.satisfice.com/blog/archives/496

Very Short Blog Posts (11): Passing Test Cases

Wednesday, January 29th, 2014

Testing is not about making sure that test cases pass. It’s about using any means to find problems that harm or annoy people. Testing involves far more than checking to see that the program returns a functionally correct result from a calculation. Testing means putting something to the test, investigating and learning about it through experimentation, interaction, and challenge. Yes, tools may help in important ways, but the point is to discover how the product serves human purposes, and how it might miss the mark. So a skilled tester does not ask simply “Does this check pass or fail?” Instead, the skilled tester probes the product and asks a much more rich and fundamental question: Is there a problem here?