Breaking the Test Case Addiction (Part 1)

January 15th, 2019

Recently, during a coaching session, a tester was wrestling with something that was a mystery to her. She asked:

Why do some tech leaders (for example, CTOs, development managers, test managers, and test leads) jump straight to test cases when they want to provide traceability, share testing efforts with stakeholders, and share feature knowledge with testers?

I’m not sure. I fear that most of the time, fixation on test cases is simply due to ignorance. Many people literally don’t know any other way to think about testing, and have never bothered to try. Alarmingly, that seems to apply not only to leaders, but to testers, too. Much of the business of testing seems to limp along on mythology, folklore, and inertia.

Testing, as we’ve pointed out (many times), is not test cases; testing is a performance. Testing, as we’ve pointed out, is the process of learning about a product through exploration and experimentation, which includes to some degree questioning, studying, modeling, observation, inference, etc. You don’t need test cases for that.

The obsession with procedurally scripted test cases is painful to see, because a mandate to follow a script removes agency, turning the tester into a robot instead of an investigator. Overly formalized procedures run a serious risk of over-focusing testing and testers alike. As James Bach has said, “testing shouldn’t be too focused… unless you want to miss lots of bugs.”

There may be specific conditions, elements of the product, notions of quality, interactions with other products, that we’d like to examine during a test, or that might change the outcome of a test. Keeping track of these could be very important. Is a procedurally scripted test case the only way to keep track? To guide the testing? The best way? A good way, even?

Let’s look at alternatives for addressing the leaders’ desires (traceability, shared knowledge of testing effort, shared feature knowledge).

Traceability. It seems to me that the usual goal of traceability is be able to narrate and justify your testing by connecting test cases to requirements. From a positive perspective, it’s a good thing to make those connections to make sure that the tester isn’t wasting time on unimportant stuff.

On the other hand, testing isn’t only about confirming that the product is consistent with the requirements documents. Testing is about finding problems that matter to people. Among other things, that requires us to learn about things that the requirements documents get wrong or don’t discuss at all. If the requirements documents are incorrect or silent on a given point, “traceable” test cases won’t reveal problems reliably.

For that reason, we’ve proposed a more powerful alternative to traceability: test framing, which is the process of establishing and describing the logical connections between the outcome of the test at the bottom and the overarching mission of testing at the top.

Requirements documents and test cases may or may not appear in the chain of connections. That’s okay, as long as the tester is able to link the test with the testing mission explicitly. In a reasonable working environment, much of the time, the framing will be tacit. If you don’t believe that, pause for a moment and note how often test cases provide a set of instructions for the tester to follow, but don’t describe the motivation for the test, or the risk that informs it.

Some testers may not have sufficient skill to describe their test framing. If that’s so, giving test cases to those testers papers over that problem in an unhelpful and unsustainable way. A much better way to address the problem would, I believe, would be to train and supervise the testers to be powerful, independent, reliable agents, with freedom to design their work and responsibility to negotiate it and account for it.

Sharing efforts with stakeholders. One key responsibility for a tester is to describe the testing work. Again, using procedurally scripted test cases seems to be a peculiar and limited means for describing what a tester does. The most important things that testers do happen inside their heads: modeling the product, studying it, observing it, making conjectures about it, analyzing risk, designing experiments… A collection of test cases, and an assertion that someone has completed them, don’t represent the thinking part of testing very well.

A test case doesn’t tell people much about your modeling and evaluation of risk. A suite of test cases doesn’t either, and typical test cases certainly don’t do so efficiently. A conversation, a list, an outline, a mind map, or a report would tend to be more fitting ways of talking about your risk models, or the processes by which you developed them.

Perhaps the worst aspect of using test cases to describe effort is that tests—performances of testing activity—become reified, turned into things, widgets, testburgers. Effort becomes recast in terms of counting test cases, which leads to no end of mischief.

If you want people to know what you’ve done, record and report on what you’ve done. Tell the testing story, which is not only about the status of the product, but also about how you performed the work, and what made it more and less valuable; harder or easier; slower or faster.

Sharing feature knowledge with testers. There are lots of ways for testers to learn about the product, and almost all of them would foster learning better than procedurally scripted test cases. Giving a tester a script tends to focus the tester on following the script, rather than learning about the product, how people might value it, and how value might be threatened.

If you want a tester to learn about a product (or feature) quickly, provide the tester with something to examine or interact with, and give the tester a mission. Try putting the tester in front of

  • the product to be tested (if that’s available)
  • an old version of the product (while you’re waiting for a newer one)
  • a prototype of the product (if there is one)
  • a comparable or competitive product or feature (if there is one)
  • a specification to be analyzed (or compared with the product, if it’s available)
  • a requirements document to be studied
  • a standard to review
  • a user story to be expanded upon
  • a tutorial to walk through
  • a user manual to digest
  • a diagram to be interpreted
  • a product manager to be interviewed
  • another tester to pair with
  • a domain expert to outline a business process

Give the tester the mission to learn something based on one or more of these things. Require the tester to take notes, and then to provide some additional evidence of what he or she learned.

(What if none of the listed items is available? If none of that is available, is any development work going on at all? If so, what is guiding the developers? Hint: it won’t be development cases!)

Perhaps some people are concerned not that there’s too little information, but too much. A corresponding worry might be that the available information is inconsistent. When important information about the product is missing, or unclear, or inconsistent, that’s a test result with important information about the project. Bugs breed in those omissions or inconsistencies.

What could be used as evidence that the tester learned something? Supplemented by the tester’s notes, the tester could

  • have a conversation with a test lead or test manager
  • provide a report on the activities the tester performed, and what the tester learned (that is, a test report)
  • produce a description of the product or feature, bugs and all (see The Honest Manual Writer Heuristic)
  • offer proposed revisions, expansions, or refinements of any of the artifacts listed above
  • identify a list of problems about the product that the tester encountered
  • develop a list of ways in which testers might identify inconsistencies between the product and something desirable (that is, a list of useful oracles)
  • report on a list of problems that the tester had in fulfilling the information mission
  • in a mind map, outline a set of ideas about how the tester might learn more about the product (that is, a test strategy)
  • list out a set of ideas about potential problems in the product (that is, a risk list)
  • develop a set of ideas about where to look for problems in product (that is, a product coverage outline)

Then review the tester’s work. Provide feedback, coaching and mentoring. Offer praise where the tester has learned something well; course correction where the tester hasn’t. Testers will get a lot more from this interactive process than from following step-by-step instructions in a test case.

My coaching client had some more questions about test cases. We’ll get to those next time.

A Moment of Jerry Weinberg Zen

January 10th, 2019

The year was 2006. James Bach and I were running a workshop at the Amplifying Your Effectiveness conference (AYE). We were in one of those large-ish, high-ceiling conference rooms with about 15 programmers and software consultants.

We were showing them one of James Lyndsay’s wonderful testing machines. (You can find it here, but you’ll need Flash active to run it.) It looked like this:

James Lyndsday's Machine 1

At first, it’s all very confusing. When you press the buttons on the left, the red and blue balls on the right move in some way. The slider in the middle influences the range of motion somehow. In general, the mission of the exercise is to describe the behaviour of the machine.

Test cases are not testing. To illustrate this important fact, we give class participants the machines to investigate for a few moments, and then ask a question that James (Bach) asked in our AYE session.

“How many test cases,” he asked, “would you need to be able to understand and describe this product completely?”

Brows immediately furrowed. Clicking sounds from the buttons and murmured conversation between pairs of participants filled the room. “Two states to the power of five buttons with how many stops on that slider…?” “Wait, that button is just momentary…” “Seven hundred and sixt… no, that’s wrong.”

Whereupon, in a moment of perfect timing, a door opened, and Jerry Weinberg walked into the room. His walking stick and his bearing reminded me of Yoda and Gandalf and other sages and wizards.

“Hey, here’s Jerry Weinberg!” said James. “The world’s greatest living software tester! Jerry, how many test cases would you need to understand and describe this product completely?”

The room fell silent. Everyone wanted to know the answer. Jerry observed the laptop that James was holding. He didn’t touch the laptop, press a key, or move the mouse. He just looked for a few moments.

Then he said, “Three.” There was a pause.

Having worked with Jerry over a decade or so, James understood. “Three, Jerry?”, he asked dramatically, in mock astonishment.

“Hm.” (pause) “Yeah. Three,” replied Jerry. Another pause.

Then he peered at James. “Why? Were you expecting some other number?”

Rapid Software Testing is coming soon to Sydney, Brisbane, Reykjavik, Utrecht, and Copenhagen. Learn more about it, and find places to register.

Pressing the Green Button

December 19th, 2018

For years at conferences and meetups and in social media, I have been hearing regularly from testers who tell me that they must “sign off” on the product or deployment before it is released to production, or to review by the client. The testers claim that, after they have performed some degree of testing work, they must “approve” or “reject” the product. Here’s a fairly typical verbatim report from one tester:

In my current context, despite my reasoned explanations to the contrary, I am viewed as the work product gatekeeper and am newly positioned as such within a software controlled workflow that literally presents a green “approve” or red “reject” button for me to select after I “do QC” on the work product which as a bonus might be provided with a list of ambiguous client requests and/or a sketchy mock-up with many scattered revision notes (often superseded by verbal requests not logged).

It’s important to note that in project work, a mess of competing information is normal — not completely desirable, necessarily, but normal. When information is unclear, it’s part of the tester’s job to identify where and how it’s unclear. Confusion and uncertainty ratchet up product and project risk. After all, if you’re not clear on what the product is supposed to do, how can the developers be clear on it? In the unlikely event that the developers know how the product should work and you don’t, how will you recognize all of the important bugs? Whether you or the developers or both aren’t straight on what the client wants, bugs will have time and opportunity to breed and to survive.

The tester continues:

Delivery of the product to the client for their review is generally held up until I press the green “approve” button. The expectation when I “approve” is that the product (which I did not build) is “error free”, meets contradictory loosely defined “standards” and satisfies the client (whom I have not met). The structure is such that I am not to directly communicate to the client, so all clarifications and questions I have are filtered through a project manager.

I am now beginning to frustrate the developers with whom I have previously built a great rapport by repeatedly rejecting their work products for even a single minor infraction. I also frustrate project managers delaying product delivery and going over budget. I predict there will soon be pressure from all sides to “just approve” and then later repercussions of “how/why did this get approved”. I combat this by providing long lists of observations, potential issues, questions, obstacles and coverage notes with each “rejection” or “approval” and I communicate to project managers that they do not need my approval to proceed and may override at anytime.

Some testers seem happy with the authority implicit in “approving” or “rejecting” the product. Most express some level of discomfort with the decision. To me, the discomfort is appropriate.

In the Rapid Software Testing view of the world, it is not the job of the tester to approve or disapprove of things. It is the job of the tester to identify reasons to believe that some person who matters might approve or disapprove of something, and to explain the bases for that belief. Decisions about what to do with a product, including approving or rejecting it, lie with a role called management.

If you’re in a situation like the tester above, and someone offers you the responsibility to approve and reject products, and you desire to be a manager, this is your big chance! Seize the opportunity—but don’t do it without the manager’s title and authority—and salary, while you’re at it. If you’re offered the approval or rejection decision without becoming a manager, though, I’d recommend that you politely decline the “offer”, and make a counteroffer—perhaps one like this:

“Thank you for honouring me with the offer to approve or reject the product. However, as a tester, I don’t believe that it is appropriate for me to make such decisions without management authority. Here’s what I will do, though.

“I will gladly test the product, learning about it through exploration and experimentation. I will evaluate the product for consistency with these (contradictory, loosely-defined) standards. If the product appears to be inconsistent with them, I will certainly let you know about that. If I see inconsistencies or contradictions in those standards, I will let you know about those too, so that you can decide how the standards apply to our product. But I won’t limit my testing to that.

“I will tell you about any important problems that I find. I will tell you about problems that appear inconsistent with things desirable to important people. Here’s an example of how I might categorize desirable things, and here’s an example of how I might recognize problems in the product.

“I will report on anything that appears consistent with some notion of an ‘error’. However, I will not assert that the product is error-free. I don’t know how I could do that. I don’t know how anyone can do that.

“I would prefer to interact freely and directly with stakeholders, for the purposes of obtaining clarifications and answers to questions I have without bothering the project manager. (I will, of course, keep responsible records of my interactions; and I will not presume to make decisions about product or project scope, since that’s a management function.)

“If you would prefer to restrict or mediate my access to stakeholders, that’s OK; I can work that way. Doing so will likely come with a cost of extra time on the part of the project manager, and the risk of broken-telephone-style miscommunication between the stakeholders and me. However, if you’re prepared to take responsibility for that risk, I’m fine with it too.

“Since I am manager of neither the product, nor the project, nor the developers, I do not have the authority to direct them. However, I am happy to report on everything I know about the product—and the apparent problems and risks in it—to those who do have the required authority and responsibility, and they can make the appropriate decisions based on everything they know about the product and business and its needs.

“I am not a gatekeeper, or owner, or ‘approver’ of the quality of the product. I am not a manager or decision maker. I am a reporter on the status of the product, and of the testing, and of the quality of the testing, and I’ll report accordingly. My “approval” is immaterial; what matters is what managers and the business want. It is they, not I, who decide whether a problem is a showstopper or something we’re prepared to live with. It is they, not I, who decide whether problems are significant enough to extend the schedule or increase the budget for the project.

“It’s my job to contribute information to any decision to approve or reject, but it’s not my job to make that decision. I would like someone else to be responsible for the ‘approve’ or ‘reject’ checkbox as such. However, if the tool that we’re using restricts me to ‘approve’ and ‘reject’, let me tell you what those mean, because what they say is inconsistent with their normal English meanings, and we should all be aware of that.

“Pressing ‘Approve’ means this, and only this: ‘I am not aware of any problem in this area that threatens the value of the product, project, or business to any person that matters.’

Pressing ‘Reject’ means ‘I am aware of a specific problem or I have some reason to believe that there could be a problem in this area that I have not had the opportunity to identify yet.’ In other words, ‘reject’ means that I see risk; there’s something about the product or about the testing that I believe managers or the programmers should be aware of. ‘Reject’ means no more than that.

“In either case, we should frequently discuss my observations, potential issues, questions, obstacles and coverage notes, to avoid the possibility that I’m overlooking something important, or that I’m over-emphasizing the significance of particular problems.”

How you’re viewed depends on the commitments (another example here) that you make and declare about what you do, what you’re willing to do, and what you’re not willing to do. If your role, your profile and your commitments don’t match, getting them lined up is your most urgent and important job.

Related reading:
Signing Off
When Testers Are Asked For a Ship/No-Ship Opinion
Testers: Get Out of the Quality Assurance Business

Rapid Software Testing classes are coming to Brisbane, Sydney, Utrecht, Reykjavik, and Copenhagen—and elsewhere. Join us!

I Represent the User! And We All Do

December 15th, 2018

As a tester, I try to represent the interests of users. Saying the user, in the singular, feels like a trap to me. There are usually lots of users, and they tend to have diverse and sometimes competing interests. I’d like to represent and highlight the interests of users that might have been forgotten or overlooked.

There’s another trap, though. As Cem Kaner has pointed out, it’s worth remembering that practically everyone else on the team represents the interests of end users in some sense. “End users want this product in a timely way at a reasonable price, so let’s get things happening on schedule and on budget,” say the project managers. “End users like lots of features,” say the marketers. “End users want this specific feature right away,” say the sales people. “End users want this feature optimized like I’m making it now,” say the programmers. I’d be careful about claiming that I represent the end user—and especially insinuating that I’m the only one who does—when lots of other people can credibly make that claim.

Meanwhile, I aspire to test and find problems that threaten the value of the product for anyone who matters. That includes anyone who might have an interest in the success of the product, like managers and developers, of course. It also includes anyone whose interests might have been forgotten or neglected. Technical support people, customer service representatives, and documentors spring to mind as examples. There are others. Can you think of them? People who live in other countries or speak other languages, whether they’re end users or business partners or colleagues in remote offices, are often overlooked or ignored.

All of the people in our organization play a role in assuring quality. I can assure the quality of my own work, but not of the product overall. For that reason, it seems inappropriate to dub myself and my testing colleagues as “quality assurance”. The “quality assurance” moniker causes no end of confusion and angst. Alas, not much has changed over the last 35 years or so: no one, including the most of the testing community, seems willing to call testers what they are: testers.

That’s a title I believe we should wear proudly and humbly. Proudly, because we cheerfully and diligently investigate the product, learning deeply about it where most others merely prod it a little. Humbly, because we don’t create the product, design it, code it, or fix it if it has problems. Let’s honour those who do that, and not make the over-reaching claim that we assure the quality of their work.

Rapid Software Testing classes are coming to Brisbane, Sydney, Utrecht, Reykjavik, and Copenhagen—and elsewhere. Join us!

Who Needs the Testers?

December 6th, 2018

This post is a lightly-edited transcript from a LinkedIn article, which was itself adapted and extended from a recent thread on Twitter.

Another day, another story that goes like this. A colleague tells me that he’s working with an organization by training developers in how to do testing. That sounds like a pretty good idea at first, although most developers are already pretty good at the kind of testing that developers need to do, especially when they collaborate and review each other’s work.

But this is not the kind of testing that the training is focused on. It turns out that developers are being trained in higher-level integration and system testing—the kind of deep testing that requires significant domain knowledge and a substantial shift from the developer mindset. Why are the developers getting this training? Because the organization got rid of the testers who were formerly doing this work, and is hoping that the developers will do it.

Why were the testers canned? The testers were unsuccessful at a mandate handed down from management: “We have to do more automated testing! Automate all the testing! Automate everything!” This means that the testers were being asked to do programming. And, unsurprisingly, most of the testers weren’t great at that.

Part of the problem was that, at best, the testers got shallow training in programming. Another part is that nobody really knew (or knows) what “automate everything” means. (The managers were probably thinking about the visible aspects of testing: pressing keys, moving mice, and comparing output to specified, desired results. That’s something we call checking. Testing can’t be automated, although checking can.) And another part is that programming turns out to be time-consuming and tricky. Who knew?!

The result was a bunch of automated checking that was shallow, brittle, didn’t find many bugs, and that didn’t keep pace with development. Testers had lots of questions for the programmers, and investigating the problems detected by automated checks took time and effort. Many of those problems turned out to be non-problems because of errors in the check code. Programmers and managers perceived, not unreasonably, that all this was interrupting the programmers’ work.

The short story motivating the decision to get rid of the testers was: testing was slowing down development. The solution: get rid of the testers. Get the programmers to do the “testing” (that is, programmed checking), since the programmers are already good at programming.

It’s usually a very good idea for programmers to do checking, especially at the unit level. Low-level checking provides the programmers with very fast feedback, alerting them to coding errors and problems that might otherwise get buried. It helps with the discipline of building the product cleanly and simply, such that it can be built and changed safely. But checking is not all there is to testing.

In the Rapid Software Testing namespace, testing is evaluating a product by learning about it through exploration and experimentation. So let’s expand the short story above: evaluating the product by learning about it through exploration and experimentation was slowing down development.

Now, let’s make that a little more concise: learning about what they were building was slowing down development. Another way to put that is that development was going too fast for people to learn about what they were building. As a former program manager, I find that ominous.

One big error, apparently, was in believing that programmed checking is all there is to testing, which led to this interpretation: getting non-programmers to write programs as a means of learning about the product was slowing down development.

It seems to me that while writing programs might be helpful for some learning purposes, it’s a highly imperfect and incomplete way to learn about a product. Why was this not recognized?

It’s often the case, alas, that testers don’t have the skills to compose, edit, narrate, and justify the story of their testing work. It’s also common for people—even testers—to believe that testing is all about developing and following scripted procedures and checking output, rather than investigating risk.

When testing is reduced to demonstrating, by rote, that the product can work, it’s easy to believe that testing is simply a programming task. Create programs to do really fast typing and really fast comparison of desired and actual results. Simple! No testers required!

Mind you, by that line of reasoning, all programmers do is type instructions from business people into a computer, right? So, here’s a modest proposal for those managers: teach business people to program, and then we won’t need programmers OR testers. Simple! No technologists required! Yet managers rarely suggest this.

Why not? Perhaps it’s because managers are aware the programming is hard. Yet those same managers often seem unaware that investigating complex systems for deeply hidden, subtle, rare, emergent problems is also hard. Testers must take responsibility for making that clear.

That requires two things: doing excellent testing, and describing excellent testing. That won’t happen when we reduce testing to test cases; when we talk of “manual” testing and “automated” testing; when we forget that we’re here to find problems that matter to people.

Doing excellent testing requires testers to understand the context of the product, where and how it will be used, and the project and processes being used to develop it. Excellent testing requires testers to have some degree of technical skill, but also to have the social skills, the communication skills, and the domain knowledge to test effectively.

Doing excellent testing requires testers to be rapid learners. It requires testers to develop rich, detailed models of risk and of the product so that they can cover it with testing. It requires good oracles—ways to recognize problems when we encounter them—far more than comparing the product to an specified, desired, “expected” result or to some line in a requirements document. It requires testers to be aware that a product is not simply units of code; the product and its users represent a system of people and things in meaningful relationship. Such systems have interactions and properties that are emergent, not obvious from simply checking the parts.

Describing excellent testing requires testers to tell the testing story, including what isn’t being tested—and what’s slowing testing down, or making testing harder, or reducing the value of the work. It requires testers to be able to articulate all of the dimensions of excellent testing: context, risk, coverage, oracles, systems thinking—and why testing a product is so much more than writing code to check its output.

================================================

Irrespective of your role, do you want to learn more about deeper testing? Rapid Software Testing 4.0 is coming to Australia (Sydney and Brisbane) in February 2019. Come to the Sydney Testers’ Meetup on December 13, 2018 for a preview: Rapid Software Testing in Agile Contexts. A public class of RST will be presented in Iceland in March. See all upcoming events, including those with James Bach, here.

If We Do Sanity Testing Before Release, Do We Have To Do Regression Testing?

December 3rd, 2018

Here is an edition of the reply I offered to a question that someone asked on Quora. Bear in mind that it might be a good idea to follow the links for context.

If we do sanity testing before release, do we have to do regression testing?

What if I told you Yes? What if I told you No?

Some questions shouldn’t be answered. That is: some questions shouldn’t be answered with a Yes or a No without addressing the context first. No one can give you a good answer to your question unless they know you, your product, and your project’s context.

Even after that problem is addressed, people outside your context may not know what you mean by regression testing or sanity testing, and you can’t be sure of knowing what they mean. That applies to other terms in the conversation, too; maybe they’ll talk about “manual testing”; I don’t believe there’s such a thing as “manual testing”. Maybe you agree with them now; maybe you’ll agree with me after you’ve read the linked post. Or maybe after you read this one.

Some people will suggest that regression testing and sanity testing are fundamentally different somehow; I’d contend that a sanity test may be a shallow form of regression testing, when the sanity test is what I’ve talked about here, and when regression testing is testing focused on regression- or change-related risk. In order to sort that out, you’d have to talk it through to make sure that you’re not in shallow agreement.

Nonetheless, some people will try to answer your question. To prepare you for some of those answers: it’s probably not very helpful to think about needing to do one kind of testing or the other. It’s probably more helpful to think in terms of what you and your organization want to do, and choosing what to do based on what (you believe) you know about your product, and what (you believe) you want to know about it, given the situation.

While this is not an exhaustive list, here are a few factors to consider:

  • Do you and the developers already have a lot of experience with your product?
  • Is your product being developed in a careful, disciplined way?
  • Are the developers making small, simple, incremental changes that they comprehend the risks well?
  • Is the product relatively well insulated from dependencies on platforms (hardware, operating systems, middleware, browers…) that vary a lot?
  • Are there already plenty of unit-level checks in place, such that the developers are likely to be aware of low-level coding errors early and easily?
  • Is it unusual to do a shallow pass through the features of the product and find bugs that are sticking out like a sore thumb?
  • Do you and the developers feel like they’re working at a sustainable pace?

If the answer to all of those questions is Yes, then maybe your regression testing can afford to be more focused on deep, rare, hidden, subtle, emergent problems, which are unlikely to be revealed by a sanity test. Or maybe your product (or a given feature, or a given change, or whatever you’re focused on) entails relatively low risk, so deep regression testing isn’t necessary and a sanity test will do. Or maybe your product is poorly-understood and has changed a lot, so both sanity checking and deep regression testing could be important to you.

I can offer things for you to think about, but I don’t think it’s appropriate for me or anyone else to answer your question for you. The good news is that if you study testing seriously, practice testing, and learn to test, you’ll be able to make this determination in collaboration with your team, and answer the question for yourself.

James Bach and I teach Rapid Software Testing to help people to become smart, powerful, helpful, independent testers, with agency over their work. If you want help with learning about Rapid Software Testing for yourself or for your team, find out how you can attend a public class, live or on line, or request one in-house.

What Should I Automate?

November 22nd, 2018

I get this question a lot: a tester who has just learned to program, or who has just learned about a new framework or tool set asks “Now that I’ve learned this, what should I automate?”

Some people (mostly men, so it seems) go into hardware stores and see some fancy tool like a compound mitre saw.

Image:  Compound Mitre Saw

Unable to resist temptation, they imagine themselves building… something. So they buy the tool. Then they take it home, take it out of the box, look at it and wonder, “Hmmm… what can I use a compound mitre saw for?”

It’s good to have a compound mitre saw available when you need one, of course. But if you simply charge ahead and start cutting stuff up without a purpose because you can, you’ll end up with a bunch of wood scraps and sawdust everywhere, and maybe with your spouse asking you what happened to the kitchen table.

When you’ve got a new tool in your toolbox, remember The Lesson of The Compound Mitre Saw. Instead of asking “What can I automate?”, start testing your product. As you do so, keep a question in the back of your mind: could some tool—a tool that I can make myself!—help me to get this part of the job done?

Ask yourself if a tool that you could create would help to enable, extend, enhance, accelerate, intensify, amplify, clarify what you’re doing—at a reasonable cost. If the answer is No because it looks too big or too hard, resist the temptation until you’ve found something that feels more tractable. When the answer is Yes, you’ll know what to automate. Creating little tools, over time, will give you the experience you need to take on the bigger things.

Exploratory Testing on an API? (Part 4)

November 21st, 2018

As promised, (at last!) here are some follow-up notes on previous installments in the series that starts here.

Let’s revisit the original question:

Do you perform any exploratory testing on APIs? How do you do it?

To review: there’s a problem with the question. Asking about “exploratory testing” is a little like asking about “vegetarian cauliflower”, “carbon-based human beings”, or “metallic copper”. Testing is fundamentally exploratory. Testing is an attempt by a self-guided agent to discover something unknown; to learn something about the product, with a special focus on finding problems and revealing risk.

One way to learn about the product is to develop and run a set of automated checks that access the product via the API. That is, a person writes code the machine to operate and observe the product, apply decision rules, and report the outputs. This produces a script, but it’s not a scripted process. Learning about the product, designing the checks, and developing the code to implement them checks are all exploratory processes.

When we use a machine to perform automated checks, there’s no discovery or learning involved in the performance of the check itself. Checks are like the system on your car that monitors certain aspects of your engine and illuminates the “check engine” light. Checks are, in essence, tools by which people become aware of specific conditions in the product. The machine learns no more than the dashboard light learns. It’s the humans, aided by tools that might include checking, who learn.

People learn as they interpret outcomes of the checks and investigate problems that the checks seem to indicate. The machinery has no way of knowing whether a reported “failed” output represent a problem in the product or a problem in the check. Checks don’t know the difference between “changed” and “broken”, between “different” and “fixed”, between “good” and “bad”. The machinery also has no way of knowing whether a reported “passed” output really means that the product is trouble-free; a problem in the check could be masking a problem in the product.

Testing via an API is exploratory testing via an API, because exploratory testing is, simply, testing. (Exploratory) testing not simply “acting like a user”, “testing without tools”, or “a manual testing technique”.

Throughout this whole series, what I’ve been doing is not “manual testing”, and it’s not “automated testing” either. I use tools to get access to the product via the API, but that’s not automated testing. There is no automated testing. Testing is neither manual nor automated. No one talks about “automated” or “manual” experiments. No one talks about “manual” or “automated” research. Testing is done neither by the hands, nor by the machinery, but by minds.

Tools do play an important role in testing. We use tools to extend, enhance, enable, accelerate, and intensify the testing we do. Tools play an important role in programming too, but no one refers to “manual programming”. No one calls compiling “automated programming”. Compiling is something that a machine can do; it is a translation between on set of strings (source code) and another set of strings (machine code). This is not to dismiss the role of the compiler, or of compiler writers; indeed, writing a sophisticated compiler is a job for an advanced programmer.

Programming starts in the complex, imprecise, social world of humans. Designers and programmers repair messy human communication into a form that’s so orderly that a brainless machine can follow the necessary instructions and perform the intended tasks. Throughout the development process, testers explore and experiment with the product to find problems, to help the development team and the business to decide whether the product they’ve got is the product they want. Tools can help, but none these processes cannot be automated.

In the testing work that I’ve described in the previous posts, I haven’t been “testing like a user”. Who uses APIs? It might be tempting to answer “application programs” (it’s an application programming interface, after all), or “machines”. But the real users of an API are human beings. These include the direct users—the various developers who write code to take advantage of what a product offers—and the indirect users—the people who use the products that programmers develop. For sure, some of my testing has been informed by ideas about actions of users of an API. That’s part of testing like a tester.

In several important ways, there’s a lot of opportunity for testability through APIs. Very generally, components and services with APIs tend to be of a smaller scale than entire applications, so studying and understanding them can be much more tractable. An API is deterministic and machine-specific. That means means that certain kinds of risks due to human variability are of lower concern than they might be through a GUI, where all kinds of things can happen at any time.

The API is by definition a programming interface, so it’s natural to use that interface for automated checking. You can use validator checks to detect problems with the syntax of the output, or parallel algorithms to check the semantics of transactions through an API.

Once they’re written, it’s easy to repeat such checks, especially to detect regressions, but be careful. In Rapid Software Testing, regression testing isn’t simply repetition of checks. To us, regression testing means testing focused on risk related to change; “going backwards” (which is what “regress” means; the opposite of “progress”).

A good regression testing strategy is focused on what has changed, how it has changed, and what might be affected. That would involve understanding what has changed; testing the change itself; exploring around that; and a smattering of testing of stuff that should be unaffected by the change (to reveal hidden or misunderstood risk). This applies whether you are testing via the API or not; whether you have a set of automated checks or not; whether you run checks continuously or not.

If you are using automated checks, remember that they can help to detect unanticipated variations from specified results, but they don’t show that everything works, and they don’t show that nothing has broken. Instead, checks verify that output from given functions are consistent from one build to the next. Do not simply confirm that everything is OK; actively search for problems. Explore around. Are all the checks passing? Ask “What else could go wrong?”

Automated checks can take on special relevance when they’re in the form of contract testschecks. The idea here is to solicit checks from actual consumers of an API that represent specified, desired results, and to check the contract from both the supplier and consumer ends. Nonetheless, remember that such checks are heavily focused on confirmation, and not on discovery of problems and risks that aren’t covered by the contracts.

On the other hand, now that you’ve gone to the trouble of writing code to check for specific outputs, why stop there? I’ve used checks in an exploratory way by:

  • varying the input systematically to look for problems related to missing or malformed data, extreme values, messed-up character handling, and other foreseeable data-related bugs;
  • varying the input more randomly (“fuzzing” is one instance of this technique), to help discover surprising hidden boundaries or potential security vulnerabilities;
  • varying the order and sequences of input, to look for subtle state-related bugs;
  • writing routines to stress the product, pumping lots of transactions and lots of data through it, to find performance-related bugs;
  • capturing data (like particular values) or metadata (like transaction times) associated with the checks, visualizing it, and analyzing it, to see problems and understand risks in new ways.

A while back, Peter Houghton told me an elegant example of using checking in exploration. Given an API to a component, he produces a simple script that calls the same function thousands of times from a loop and benchmarks the time that the process took. Periodically he re-runs the script and compares the timing to the first run. If he sees a significant change in the timing, he investigates. About half the time, he says, he finds a bug.

So, to sum up: all testing is exploratory. Exploration is aided by tools, and automated checking is an approach to using tools. Investigation of the unknown and discover of new knowledge is of the essence of exploration. We must explore to find bugs. All testing on APIs is exploratory.

Exploratory Testing on an API? (Part 3)

July 30th, 2018

In the last two posts, I’ve been answering the question

Do you perform any exploratory testing on an API? How do you do it?

Last time I described the process of learning about factors in the project environment that affect my strategy as I go about the testing of an API. This time, I’ll go deeper, learn more, and begin to interact with the product. That learning and interaction feeds back into more learning and more interaction.

Along the way, I’ll develop ideas that are reflected in the Heuristic Test Strategy Model. I get ideas about potential problems and risks based on Quality Criteria (and things that threaten them). I identify Product Factors and get ideas about coverage—how to the cover the product with testing. I apply those ideas using approaches that come from families of Test Techniques.

Notice, though, that even though I’ve tagged them, these ideas don’t come to me from the HTSM as such. Ideas occur to me as I encounter elements of the product and its context. Each interaction with and observation of the product triggers ideas about risk and how to test for it. Some of these ideas come from my testing and development experience; others from stories told by my colleagues or reported in the news; still others from the product and its documentation; and those all lead to new ideas that are synthesized in the moment.

You may find that this post is long. That length reflects the volume of ideas that occur in an exploratory process.

Whether they are end users—getting access to the product through the GUI—or developers—getting access to the product through an API—people use software to make things happen. They have tasks that they want to get done. So what can I imagine the API’s user wanting to do? Product Factors/Operations; Test Techniques/Scenario Testing.

An API’s user wants to get some data, make something happen, or effect some change. Anything that happens or changes in the product is driven by specific functions in the code. What functions does the product perform? Product Factors/Functions I find out by looking at the product and API documentation if it’s available, or by looking at source code if that’s available. Project Environment/Information If I can’t get to either one, I might ask developers or designers for help. Project Environment/Developer Relations

I take notes about what I’m learning, perhaps in the form of a mind map; perhaps in point-form text; perhaps in a table. The process of taking notes helps to cement what I’m learning. In addition, I want to notice inconsistencies between my sources of information. Sometimes different people tell me the different things. Some people contradict what is written in the requirements documents or specifications. Whenever those things happen, bugs are breeding. I’ll collect lots of specific statements about the product should do, and look for controversy. Test Techniques/Claims Testing

I’ll review what I already know. To what functions does the API offer access? The user of the API wants to obtain some kind of value from interacting with the product, and those values are expressible in terms of quality criteria. Quality Criteria/Capability What could go wrong with those functions and threaten something do do with one of those quality criteria?

What are the protocols for calling those functions? Product Factors/Interfaces Is there anything, like setting up or logging in, that should happen before an API call? What if they don’t happen? I’ll make a note to try that later. Test Techniques/Flow Testing “Before” reminds me to ask myself about “after”, too: Are there there things to be cleaned up after an API call? Product Factors/Time Does data need to be saved or purged?

Given a set of available functions, I’ll try to use a small selection of the functions that I’m aware of. Test Techniques/Function Testing; Test Techniques/Claims Testing I’ll try some simple stuff using a tool that affords direct interaction; these days it would be Postman, or some quick calls from IRB (the Ruby interpreter). Does the product appear to do what it should? Can it work? Quality Criteria/Capability

I’ll start simple with representative inputs; then I’ll explore around the extents and limits that I’ve been told about or that are documented. Test Techniques/Domain Testing. I’ll put in some preposterously long strings, or large quantities, or deliberately malformed input. I’ll leave out mandatory fields in the call; put in extra fields that I’ve made up; provide letters where where only numbers are anticipated; feed in symbols or accented characters where only normal letters are anticipated. If it’s a Web app, I might focus on a smattering of characters that are associated with cross-site scripting, SQL injection, or other security risks. Quality Criteria/Security

If I do something that should trigger error handling, does error handling happen? Does the product return an error code? After that, does continue to do what it should do? Test Techniques/Claims Testing; Quality Criteria/Reliability/Robustness I pay special attention when the product appears to contradict the documentation or what someone has told me. Test Techniques/Claims Testing

At this point, if I find problems, I note them. Even if I don’t, I start to gather some ideas about how I might use tools and automated checks to vary the input and cover whole sets of possible inputs when it’s time for deep testing.

I pause and remember that that there’s a difference between the product doing what it should and appearing to do what it should. Every step of the way, I’d like to be able to observe as much as I can of the product’s state. Are there any special API function calls that I can use to examine or control the state of the product before, during, and after the usual calls? Quality Criteria/Development/Testability These may not be noted in the public documentation; I may have to ask developers about them Project Environment/Developer Relations or look at source code. Project Environment/Information

Thinking about testability like this reminds me to ask: Does the product offer logging? I have a look. Is the format of the logs consistent with other logs in the same system? Consistent log files will help to make my testing go more quickly and easily, which helps me to get to deeper testing. Quality Criteria/Development/Testability If there is no logging and there is resistance from the developers or from management to putting it in, I’ll keep that in mind and note it for the future, when I design automated checks; the logs for my checks may have to be more specific and detailed.

I’ve wandered from thinking about functions to testability, so I pull on the leash to bring my mind back to functions again. What services do the functions perform for the rest of the system? What are the bits and pieces of the system? Product Factors/Structure If I haven’t started one already, I’ll sketch a diagram of the product’s structure, and use that to help guide my understanding of the product. As my understanding develops, I’ll share the diagram with others to determine whether we’re all on the same page, and to help to maintain our awareness of the product’s structure. I might also post a copy up in a prominent place and invite people to let me know as soon as they see something wrong. Every now and again, I’ll deliberately review the diagram.

Thinking about the parts of the system reminds to to ask myself: Upon what parts of the whole system do those functions depend? In Rapid Software Testing, something that our product depends upon that is outside the control of our current project is called a platform. Product Factors/Platform.

Platforms might include the operating system, the browser, application frameworks, third-party libraries, stuff that was built in-house on another project. Platforms might also include hardware elements, too: displays, keyboards, mice, touchscreens, printers, or other peripherals. I’ll develop lists of platform dependencies, preparing myself for deeper coverage later.

I return to exercising the functions. Most functions do something to something: data. What data can I supply via the API? In what format? What responses will the API provide? How are responses structured? Product Factors/Data. Answers to these questions will influence my selection of tools for deeper testing. Project Environment/Equipment and Tools

Is the API providing access to a database? If so, and I’m focused on testing the functions in the API, I might be able use direct access to the database as an oracle. Test Techniques/Function Testing

Thinking about databases prompts me to think about stored data and its quality. I focus for a moment on the quality of the data, and about what data I might want to use or generate for automated checks. Test Techniques/Domain Testing; Test Techniques/Automated Checking If I want to examine risk related to data quality, going directly to the database might be more efficient than trying to get at the data through the API’s functions. Of course, I could do a smattering of both, for which I’ll need different tools. Project Environment/Equipment and Tools

Although I’m doing it in a somewhat more painstaking way, as I’m learning to use the API, I’m doing what one of its users—typically a developer—would do to learn about it. Test Techniques/User Testing Just like its intended user, I make mistakes—some intentionally, some quite unintentionally. How difficult might it be for a developer to use this API correctly? How easy might it be to use it incorrectly, or in a way that provides misleading, ambiguous, or confusing results? Quality Criteria/Usability Test Techniques/Risk-Based Testing

I imagine a set of programming tasks that I might want to accomplish using the API. At that same time, I imagine a person performing those tasks; a role. It’s quite possible that not all users of the API will be expert programmers, so it makes sense for me to create a “novice programmer” role. Test Techniques/User Testing What if I make the kinds of mistakes a novice programmer might make, and ask for something in a way that the API doesn’t support? What if I hand it pathological or malformed data; data structures with missing elements, dates in the wrong format, or characters that might be used in countries other than my own? Quality Criteria/Localizability Does the product report an error? Will the novice understand it? A potential misunderstanding might point to a problem in the product or in the documentation for it.

How well do things get cleaned up after an error condition? Are error conditions handled gracefully? Quality Criteria/Reliability/Error Handling How are error conditions relayed to the user of the API—the software calling it, and the human writing that software? Are there error codes to make troubleshooting easier? Quality Criteria/Development/Supportability? Human-readable strings or data structures? Are they consistent with the documentation? Quality Criteria/Usability. Are they reasonably elegant and efficient, or are they clunky, confusing, misspelled, or embarrassing in any way? Quality Criteria/Charisma

Some APIs are used for atomic interactions; single calls to a service followed by single responses from it. Other APIs work more like a conversation, retaining data or maintaining the product in a certain state from one call to the next. A conversation has its risks: what happens when I interrupt it? What happens if I leave out a particular call or response? Test Techniques/Flow Testing What if I varied the sequence of interactions in a conversation? Product Factors/Time Has anyone made any claims about timeouts, either informally or in the documentation? Test Techniques/Claims Testing

How long will the system let me leave a conversation dangling? Product Factors; Time What happens to the internal state of the system or to the data that’s being managed after a conversation has been abandoned for a while? What happens when I try to pick up a previously abandoned conversation? Quality Criteria; Reliability

Thinking about sequence leads me to think about time generally. How long does it take for the system to process an API call and return a result? Product Factors/Time Tools help make the round-trip time easily visible—but I don’t trust evaluating performance based on the first result. Because the tool makes it so easy, I often do several quick tests to observe results from the same call several times. If there’s substantial variation, I make a note of it; it might be a bug, or it might be a target for deeper testing.

Has anyone made explicit claims about performance? Project Environment/Information Even though I might refer to those claims, I don’t need them to do performance testing; I’ll operate the product, collect observations about performance, and include those observations in my report.

In addition to the API, there may also be human user interfaces: a GUI, or a command line interface. I’ll exercise those interfaces too, and look for inconsistencies between the way they behave and the way the API behaves. Product Factors/Interfaces Inconsistency within the product is a powerful oracle.

Considering the human interfaces reminds me of other interfaces. The API interacts with certain components in the product, and the product inevitably interacts with other products: the operating system, the file system, third-party frameworks and libraries, networks, printers, ports, Bluetooth… These interfaces afford access to the platforms upon which the product depends. Product Factors/Platform

What if one of those dependencies were missing? What if I deprived the of something it needs by preventing access to files or databases? Test Techniques; Stress Testing Would it stay up? Quality Criteria; Reliability

Thinking about reliability prompts me to think about how the product might behave when I place it under load. Quality Criteria; Performance Could I stress it out by overwhelming it with large amounts of data in specific fields, or with an extreme rate of calls? Test Techniques; Stress Testing I’ll definitely need tools for that. Project Environment/Equipment and Tools

I could use those tools to do the same checks over and over again, comparing the product’s output to a specified, hard-coded result. But I could broaden my coverage by varying the inputs for the checks, and using a parallel algorithm to compute the desired result. Exploring the product prepares me to identify those opportunities.

After a session or two of this kind of activity, I’ve learned a lot of useful stuff about the product. My learning allows me to make much better decisions about where I might want to apply automated checks to address particular risks; how I might want to induce variation in those checks; how I might want to extend ideas about checking.

Based on the length of this post, it may seem like this exercise is necessarily a long process. It doesn’t have to be. Code takes time to develop. Quick little informal experiments can fast and cheap. Ideas about risk, coverage, quality criteria, and oracles can be developed much more quickly and less expensively than code can. Writing about these things, as I’ve done here, takes far longer than thinking them through! Yet taking notes as I go often helps by inducing pauses in which more ideas occur to me.

This is testing! I’m not only testing to find bugs here; I’m testing to learn how to test the product. If I’m going to use automated checking as part of a strategy for learning specific things about the product quickly, I must test to learn how to develop those checks. This is an exploratory process! Notice how often each idea above is connected to the one before. Testing is fundamentally exploratory.

Do I do exploratory testing on an API?  Yes—because if I’m testing it, I’m doing exploratory work! If I’m doing anything other than that, it’s probably automated checking—and I can’t begin to do excellent checking until I’ve done that exploratory work.

I’ll have some follow-up notes in the final post in this series.

Exploratory Testing on an API? (Part 2)

July 17th, 2018

Summary:  Loops of exploration, experimentation, studying, modeling, and learning are the essence of testing, not an add-on to it. The intersection of activity and models (such as the Heuristic Test Strategy Model) help us to perform testing while continuously developing, refining, and reviewing it. Testing is much more than writing a bunch of automated checks to confirm that the product can do something; it’s an ongoing investigation in which we continuously develop our understanding of the product.

Last time out, I began the process of providing a deep answer to this question:

Do you perform any exploratory testing on APIs? How do you do it?

That started with reframing the first question

Do you perform any exploratory testing on APIs?

into a different question

Given a product with an API, do you do testing?

The answer was, of course, Yes. This time I’ll turn to addressing the question “How do you do it?” I’ll outline my thought process and the activities that I would perform, and how they feed back on each other.

Note that in Rapid Software Testing, a test is an action performed by a human; neither a specific check nor a scripted test procedure. A test is a burst of exploration and experiments that you perform. As part of that activity, a test include thousands of automated checks within it, or just one, or none at all. Part of the test may be written down, encoded as a specific procedure. Testing might be aided by tools, by documents or other artifacts, or by process models. But the most important part of testing is what testers think and what testers do.

(Note here that when I say “testers” here, I mean any person who is either permanently or temporarily in a testing role. “Tester” applies to a dedicated tester; a solo programmer switching from the building mindset to the tester mindset; or a programmer or DevOps person examining the product in a group without dedicated testers.)

It doesn’t much matter where I start, because neither learning nor testing happen in straight lines. They happen in loops, cycles, epicycles; some long and some short; nested inside each other; like a fractal. Testing and learning entail alternation between focusing and defocusing; some quick flashes of insight, some longer periods of reflection; smooth progress at some times, and frequent stumbling blocks at others. Testing, by nature, is an exploratory process involving conversation, study, experimentation, discovery, investigation that leads to more learning and more testing.

As for anything else I might test, when I’m testing a product through an API, I must develop a strategy. In the Rapid Software Testing namespace, your strategy is the set of ideas that guide the design, development, and selection of your tests.

Having the the Heuristic Test Strategy Model in my head and periodically revisiting it helps me to develop useful ideas about how to cover the product with testing. So as I continue to describe my process, I’ll annotate what I’m describing below with some of the guideword heuristics from the HTSM. The references will look like this.

A word of caution, though:  the HTSM isn’t a template or a script.  As I’m encountering the project and the product, test ideas are coming to me largely because I’ve internalized them through practice, introspection, review, and feedback.  I might use the HTSM generatively, to help ideas grow if I’m having a momentary drought; I might use it retrospectively as a checklist against which I review and evaluate my strategy and coverage ideas; or I might use it as a means of describing testing work and sharing ideas with other people, as I’m doing here.

Testing the RST way starts with evaluating my context. That starts with taking stock of my mission, and that starts with the person giving me my mission. Who is my client—that is, to whom am I directly answerable? What does my client want me to investigate?

I’m helping someone—my client, developers, or other stakeholders—to evaluate the quality of the product. Often when we think about value, we think about value to paying customers and to end users, but there are plenty of people who might get value from the product, or have that value threatened. Quality is value to some person who matters, so whose values do we know might matter? Who might have been overlooked? Project Environment/Mission

Before I do anything else, I’ll need to figure out—at least roughly—how much time I’ll have to accomplish the mission. While I’m at it, I’ll ask other time-related questions about the project: are there any deadlines approaching? How often do builds arrive? How much time should I dedicate to preparing reports or other artifacts? Project Environment/Schedule

Has anyone else tested this product? Who are they? Where are they? Can I talk to them? If not, did they produce results or artifacts that will help me? Am I on a team? What skills do we have? What skills do we need? Project Environment/Test Team

What does my client want to me to provide? A test report, almost certainly, and bug reports, probably—but in what form? Oral conversations or informally written summaries? I’m biased towards keeping things light, so that I can offer rapid feedback to clients and developers. Would the client prefer more formal appoaches, using particular reporting or management tools? As much as the client might like that, I’ll also note whenever I see costs of formalization.

What else might the client, developers, and other stakeholders want to see, now or later on? Input that I’ve generated for testing? Code for automated checks? Statistical test results? Visualizations of those results? Tools that I’ve crafted and documentation for them? A description of my perception of the product? Formal reports for regulators and auditors? Project Environment/Deliverables I’ll continue to review my mission and the desired deliverables throughout the project.

So what is this thing I’m about to test? Project Environment/Test Item Having checked on my mission, I proceed to simple stuff so that I can start the process of learning about the product. I can start with any one of these things, or with two or more of them in parallel.

I talk to the developers, if they’re available. Even better, I participate in design and planning sessions for the product, if I can. My job at such meetings is to learn, to advocate for testability, to bring ideas and ask questions about problems and risks. I ask about testing that the developers have done, and the checking that they’ve set up. Project Environment/Developer Relations

If I’ve been invited to the party late or not at all, I’ll make a note of it. I want to be as helpful as possible, but I also want to keep track of anything that makes my testing harder or slower, so that everyone can learn from that. Maybe I can point out that my testing will be better-informed the earlier and the more easily I can engage with the product, the project, and the team.

I examine the documentation for the API and for the rest of the product. Project Environment/Information I want to develop an understanding of the product: the services it offers, the means of controlling it, and its role in the systems that surround it. I annotate the documentation or take separate notes, so that I can remember and discuss my findings later on. As I do so, I pay special attention to things that seem inconsistent or confusing.

If I’m confused, I don’t worry about being confused. I know that some of my confusion will dissipate as I learn about the product. Some of my confusion might suggest that there are things that I need to learn. Some of my confusion might point to the risk that the users of the product will be confused too. Confusion can be a resource, a motivator, as long as I don’t mind being confused.

As I’m reading the documentation, I ask myself “What simple, ordinary, normal things can I do with the product?” If I have the product available, I’ll do sympathetic testing by trying a few basic requests, using a tool that provides direct interaction with the product through its API. Perhaps it’s a tool developed in-house; perhaps it’s a tool crafted for API testing like Postman or SOAPUI; or maybe I’ll use an interpreter like Ruby’s IRB along with some helpful libraries like HTTParty. Project Environment/Equipment and Tools

I might develop a handful of very simple scripts, or I might retain logs that the tool or the interpreter provides. I’m just as likely to throw this stuff away as I am to keep it. At this stage, my focus is on learning more than on developing formal, reusable checks. I’ll know better how to test and check the product after I’ve tried to test it.

If I find a bug—any kind of inconsistency or misbehaviour that threatens the value of the product—I’ll report it right away, but that’s not all I’ll report. If I have any problems with trying to do sympathetic testing, I’ll report them immediately. They may be usability problems, testability problems, or both at once. At this stage of the project, I’ll bias my choices towards the fastest, least expensive, and least formal reporting I can do.

My primary goal at this point, though, is not to find bugs, but to figure out how people might use the API to get access to the product, how they might get value from it, and how that value might be threatened. I’m developing my models of the product; how it’s intended to work, how to use it, and how to test it. Learning about the product in a comprehensive way prepares me to find better bugs—deeper, subtler, less frequent, more damaging.

To help the learning stick, I aspire to be a good researcher: taking notes; creating diagrams; building lists of features, functions, and risks; making mind maps; annotating existing documentation. Periodically I’ll review these artifacts with programmers, managers, or other colleagues, in order to test my learning.

Irrespective of where I’ve started, I’ll iterate and go deeper, testing the product and refining my models and strategies as I go. We’ll look at that in the next installment.