Blog Posts from June, 2010

Questions from Listeners (2): Is Unit Testing Automated?

Monday, June 28th, 2010

On April 19, 2010, I was interviewed by Gil Broza.  In preparation for that interview, we solicited questions from the listeners, and I promised to answer them either in the interview or in my blog.  Here’s the second one.

Unit testing is automated. When functional, integration, and system test cannot be automated, how to handle regression testing without exploding the manual test with each iteration?

This question provides a great opportunity to look at a number of points—so many that I’d like to address only the first sentence in the question this time around. I’ll look at the second part of the question later on.

Expansive Definitions

I find the most helpful definitions and descriptions to be those that are expansive and inclusive. While testing, one big risk is that I might have narrow ideas about certain risks or threats to the value of the product. Thinking expansively helps me to avoid tunnel vision that would lead to my missing important problems. In conversations, thinking expansively helps me to remain alert to the possibility that the other person and I might be talking at cross-purposes. That can happen when one of us uses a word that means different things to each of us. It can also happen when we’re thinking of the same thing, but using different words. In fact, as Jerry Weinberg once remarked to James Bach, “A tester is someone who knows that things can be different.” Here’s an example of that. The questioner says that “unit testing is automated”. I’d argue that this refers to one part of testing, test execution, the part we can automate. Well, to me, things can be different.

Testing Includes Many Activities

Testing includes not only test execution, but also test design, learning, and reporting, all performed in cycles or loops. What is test design? As we say in the Rapid Software Testing course notes, test design includes

  • modeling the test space (that is, considering questions of what we could test; what’s in scope);
  • determining oracles (that is, figuring out the principles or mechanisms by which we’d recognize a problem, and considering how those principles or mechanisms might fail to help us recognize a problem)
  • determining coverage (that is, how much testing we’re going to do, given the scope)
  • determining procedures (how we’re going to perform the tests; how we’ll go about the business of test execution)

Test execution includes

  • configuring the product (obtaining it, setting it up for the purposes of a given test)
  • operating the product (exercising the product in some way to obtain coverage)
  • observing the product (applying the oracles that we’ve determined in advance, but also recognizing behaviours that trigger us to recognize and apply new oracles)
  • evaluating the product (comparing its behaviour to our oracles)
  • applying a stopping heuristic (deciding when the test is done)
  • Test execution may or may not include reporting, but reporting happens at some point. And when testing is being done well, learning is happening pretty much all the time. This isn’t a strictly linear process, by the way. Depending on your approach to testing, and depending on what you’re these things may happen in the order that you see above, or they may happen all at once in an organic tangled ball, with lots of tight little loops. Sometimes all of the elements of testing are done by the same person, and the elements interact with each other very quickly. Sometimes one person designs a test and another person handles the execution, in which case the loops will be long or broken. If you separate test design and test execution (as happens in scripted testing), you separate the learning associated with each. Sometimes we’ll evaluate a result and stop a test; sometimes we’ll stop first and then interpret what we’ve seen. For a given test, some aspects may take much longer than others; some may be done more consciously or thoughtfully than others. But at some point in pretty much every test, each of the steps above happen.

    Unit Testing Includes Many Activities

    Like any other kind of testing, unit testing consists of cycles of design, execution, learning, and reporting. Like any other test, a unit test starts with some person having a test idea, a question that we want to ask about the program. A person designing a unit test typically frames that question in terms of a check—an observation linked to a decision rule such that both can be performed by a machine. The person writes program code to express that yes-or-no question, usually assisted by some kind of unit testing framework. Next, some person—or, more often, some process that a person has initiated—performs the checks. The check produces a result. Sometimes a person observes that result independently of other results; more often, some person (the author of the automation framework) has programmed a mechanism that provides a means of aggregating the results. Then some person interprets the aggregated results and figures out what needs to be done next—whether everything is okay, whether a test result suggests that the product should be revised, or whether the check is excellent or wanting or broken irrelevant. And then the development cycle continues, in a loop that includes some development of the actual product too.

    Most Parts of Unit Testing Are Sapient, Not Mechanical

    Notice how many times the word “person” appears in the above description of unit testing. None of the steps in the process (with the exception of the running of the checks) can be automated, since each step requires a thinking person, rather than a machine, to seek information, to make decisions, and to control the overall process. Parts of unit testing can be assisted by automation, but the automation isn’t doing anything particularly on its own; it remains an extension of the person’s ability to execute and to observe.

    What form might unit test automation take? Many people think in terms of a testing framework that sets up some conditions, executes some code from the product under test, makes some assertions about the output of some function or some aspect the state of the system. That’s cool, and quite powerful. But for years at Quarterdeck, I watched programmers doing unit testing (and did some myself) by stepping though code under various debuggers (DEBUG, SYMDEB, WDEB386, or Soft-ICE, a software-based simulacrum of an in-circuit emulator), watching the registers and the ports for each instruction. Sometimes I’m writing some stuff in Ruby, and I want to do a quick little test of a fairly trivial function that I know I’m going to throw away. In that case, I don’t bother with the testing framework; I run the code and inspect the variables in IRB, the Ruby interpreter, and get my information that way. Sometimes I write a function, and generate some data to test it using automation. Sometimes, while unit testing, I use tools to examine the contents of a database table or a file or the Windows registry. Are all these different things unit testing? Jerry Weinberg says that testing is “gathering information with the intention of informing a decision”. I’m testing a unit, and I’m using automation to assist that testing, even though (so it seems) people tend to hold a more narrow view of what unit testing is. Unit testing is testing done at the unit level.

    Is stepping through the code the way that we should always do unit testing? Of course not. For the purpose of creating easily-runnable change detectors, the unit test framework is the way to go. Yet different approaches, tools, and techniques that we employ allow us to observe in different ways, discover different problems, and learn different things about the unit under test.

    Finally, it’s important to note that the development of unit-level checks tends to reveal more problems than the running of them. Chip Groeder won a best paper award at the STAR conference in 1997, in which he claimed that 88% of the bugs that he found with automated tests were found during development of the tests (that is, the non-automated parts of the testing). (Thanks to Cem Kaner for pointing me to this.)  Anecdotally, everyone that I speak to who uses automation for the execution of tests—whether at the unit level or not—says exactly the same thing.  That’s not to say that automated checks are useless.  On the contrary; checks, as change detectors, are very useful.  Instead, my point is that unit testing is not automated; not the interesting parts. Unit checking is automated.

    In summary:

    • Unit testing is a highly exploratory process, in the that the loops are short, tightly integrated, and typically performed by the same person.
    • The most important parts of unit test are the sapient parts—the design, programming, design of reports, interpretation of results, and the evaluation of what to do next.
    • The scripted part of unit testing—the execution of the checks—is the least interesting part of unit testing. And yet…
    • Many people seem to be fascinated by the mechanical parts, dazzled by lines on the screen, blissful upon observation of the green bar. And the same people say things like “unit testing is automated”. Why is that?

    That’s a lot for now. I’ll answer the rest of the question in a future post.

    Doing Development Work vs. Doing Quality Assurance

    Saturday, June 5th, 2010

    Here’s a case where a comment and question were worthy of a post of their own.  In reference to my recent post, Testers:  Get Out of the Quality Assurance Business, Selim Mia writes:

    Hi Michael,

    I have started following your blog just from past few days and I like to thank you for all of your thoughtful posts by which reflects your craftsmanship.

    Thank you for reading, and thank you for thanking me.

    I have solely agreed all of your points/advice/discussions on this post. I had many confusion about the term QA and QC since the start of my testing career and still have many confusion, i think other testers have the same. i have been working in a department called “QA” in my organization but doing mostly testing tasks as like other companies in Bangladesh. But along with testing we have also doing some of the QA tasks (i think) and below i have mentioned some of these:

    • Check-in Review: we check, each developer at-least once in a day Check-in their source code into the svn repository (source code management system) with the comment what changes he made for this particular check-in and also reviewer name who pair reviewed the code before check-in.
    • Code review: we check, is the code reviewed by the technology expert in witch technology project is developing in the regular interval (at least for the new developer’s code, code of complex functionalities, etc) and also we ensure that actions has been taken for all the review comments.
    • Audit Process Framework: we check, are all the development processes are following by the all project members except their have enough justification and approval not to follow the particular process(es).
    • Audit Bug repository: we ensure all the reported bugs have been taken into action (not a bug, assigned, WIP, fix, won’t fix).
    • Audit Document Management System: we ensure that all the updated version of all documents of the particular project are stored on the DMS.

    Are not all above activities are part (of course, not all) of QA? Your kind words will be very much helpful to me.

    – Selim

    What a great question! Thank you for asking.

    The overarching mission for a tester, in my view, is to be of service to the project. Now, that’s not only the case for testers; I think it’s the overarching mission of anyone, everyone, on the project. We’re all in service to our paramount clients—the product owners, the business owners, the gold owners and the goal donors (as some Agile wags have said)—but we’re also in service to each other. When we’re thinking that way, the testers help the programmers by testing the product using a different skill set and mind set from the programmers; the programmers help the testers by providing a more testable product (log files, scriptable interfaces, and so on). Testers may help programmers to pinpoint the circumstances in which a bug happens; programmers help testers by providing explanations, test programs, hints on what to test. Testers learn to program; programmers learn to test. We support each other and learn from each other.

    The Agile people for years have been advocating the idea of the self-organizing team. I believe in that too. That means that, in principle, anyone on the team is empowered to do whatever work needs to be done. So if a programmer takes on the tasks of setting up and configuring test environments, or if the tester is recruited to review code or models or bugs—activities that help to assure quality as a part of collaborative process, I’d say that’s cool.

    The audit stuff gives me pause. Auditing, in my view, is a kind of testing role: gathering information with the intention of informing a decision. Auditors don’t set policy or enforce rules; they provide information to management. In many process-model-obsessed organizations (here in the West, at least) the role has taken on a different slant: auditors are a kind of process police. In such organizations, people rearrange and reprioritize their work not to optimize its value, but to keep the auditors happy. This is a form of goal displacement. To me, the priority should be on providing service and value to our clients, including each other.

    In my view, if auditors discover some deviation from a set policy or a process model, I’d argue that the first step is to question the reasons for the deviation. Maybe someone is being sloppy; maybe someone is cutting corners; maybe someone is adding risk. But maybe someone has discovered a faster, less expensive, more efficient, more informative, more productive way of handling a task. Models always leave out something. Process models often leave out means by which we can encourage beneficial variation and change. I’ve never heard of an auditor reporting on some fabulous new problem-solving approach that someone has discovered internally. Most often, in my experience, process models leave out adaptability and people, as this remarkable TED talk describes.

    It’s neither a tester’s job nor an auditor’s job, in my view, to set or enforce policy, and I think it’s politically dangerous for us to be perceived that way. As soon as we are perceived to be responsible for enforcement, we run the risk of being seen as tattletales, busybodies, quality police. In that kind of environment, information will soon start to be hidden, which undermines the task of investigating the product and identifying problems with it that threaten its value.

    So, to the extent that you’re doing development work that helps to assure quality; to the extent that your teammates themselves are asking you to assist them; to the extent that you’re providing a service to them; to the extent that they appreciate what you’re doing as a service to them; and to the extent that they thank you for it, I’d say “rock on”, and congratulations.

    In another forum, a correspondent suggested “Maybe it’s all down to the “overall” thing – be part of the process, not a megalomaniac who thinks he owns it.” I absolutely agree with that.  To the extent that you’re doing “quality assurance”; to the extent that your managers are requiring you to impose on your teammates (or even worse, to the extent that you’re imposing without being asked by anyone); to the extent that you’re slowing down the project or inflicting help; to the extent that the programmers see your work as enforcing the contents of a process model or policy document; to the extent that you are barely tolerated or outright resented—well, as always, that’s up to you and your organization. But it’s not the kind of work that I would condone or accept myself.

    Again, thanks for writing.

    Test Ideas for Documentation

    Friday, June 4th, 2010

    Most people who bother with the matter at all would admit that the English language is in a bad way, but it is generally assumed that we cannot by conscious action do anything about it. Our civilization is decadent and our language — so the argument runs — must inevitably share in the general collapse. It follows that any struggle against the abuse of language is a sentimental archaism, like preferring candles to electric light or hansom cabs to aeroplanes. Underneath this lies the half-conscious belief that language is a natural growth and not an instrument which we shape for our own purposes.

    Now, it is clear that the decline of a language must ultimately have political and economic causes: it is not due simply to the bad influence of this or that individual writer. But an effect can become a cause, reinforcing the original cause and producing the same effect in an intensified form, and so on indefinitely. A man may take to drink because he feels himself to be a failure, and then fail all the more completely because he drinks. It is rather the same thing that is happening to the English language. It becomes ugly and inaccurate because our thoughts are foolish, but the slovenliness of our language makes it easier for us to have foolish thoughts. The point is that the process is reversible. Modern English, especially written English, is full of bad habits which spread by imitation and which can be avoided if one is willing to take the necessary trouble. If one gets rid of these habits one can think more clearly, and to think clearly is a necessary first step toward political regeneration: so that the fight against bad English is not frivolous and is not the exclusive concern of professional writers.

    George Orwell, Politics and the English Language

    On those increasingly rare occasions when I’m at home, I’m in the city of Toronto, the province of Ontario, Canada.  At the beginning of July, Ontario will replace its provincial sales tax with the Harmonized Sales Tax (HST). The HST combines Canada’s federal Goods and Services Tax with something that looks a lot like the current provincial tax, but which, in essence, gets applied to more goods and services.  One positive aspect of the HST is that Ontario businesses won’t have to collect and submit two different taxes to two different governments.  On the negative side, we have to learn some new rules.  Here’s one: the amount of tax charged depends in part on where the service is delivered.  There are four rules associated with this, as documented in the Government of Canada’s GST Technical Bulletin “Place of supply rules for determining whether a supply is made in a province” I’ll show you just one of those rules, and its explanation, here:

    Rule 1

    Subject to the proposed place of supply rules for services that are explained in Parts II to XII of this section and in Sections 6 to 9, a supply of a service is made in a province if, in the ordinary course of business of the supplier, the supplier
    (a) obtains only one address that is a home or a business address in Canada of the recipient, the home or business address in Canada of the recipient in the province,
    (b) obtains more than one address described in paragraph (a), the address in the province described in that paragraph that is most closely connected with the supply, or
    (c) in any other case, obtains an address in Canada of the recipient that is most closely connected with the supply.

    That sounds a little confusing.  Let’s see what it means:

    Rule 1 is based on the location of the recipient. The rule generally accomplishes this by determining the place of supply based on a particular address in Canada of the recipient that is obtained by the supplier in the ordinary course of business.


    The rule does not require a supplier to obtain an address of the recipient that the supplier does not already obtain in the ordinary course of its business. The determination of the relevant address in respect of a supply under Rule 1 is based on the facts taking into account the ordinary business practice of each supplier with respect to each supply.


    It should be noted that an address of the recipient obtained by a supplier will only be relevant for purposes of this rule if it is obtained in the ordinary course of the supplier’s business practices in connection with the supply. On the other hand, any address of the recipient obtained in the ordinary course of business of the supplier should be taken into consideration in applying the rule. The relevant address of the recipient also does not have to be an address obtained in respect of every supply made to the recipient for it to be considered a relevant address obtained in the ordinary course of business. An address of the recipient obtained by a supplier in the ordinary course of business could therefore include: an address of the recipient from which the supplier is hired in connection with a supply pursuant to an agreement for the supply (the “contracting address”); an address of the recipient that the supplier deals with in connection with a supply; or a billing address of the recipient in connection with a supply.

    Oh, thank you.  That clears things up considerably. (And if you, dear reader, didn’t wade all the way through that mud, I promise I won’t be upset.)

    Here’s a direct quote from the Rapid Software Testing course:  “When people say ‘that should be documented’, what they really mean is ‘that should be documented if and how and when it serves our purposes.'”  And we add, “Who will read it? Will they understand it? Is there a better way to communicate that information? What does documentation cost you?”

    When people—managers, consultants (yes, like me), bureaucrats, process enthusiasts, methodologists, staff employees, executives, pundits, TV preachers—tell us do something, it behooves us to think of the value of (not) doing something, cost of (not) doing something, the risk of (not) doing something, and the quality of the work. Yet none of those things—especially quality, value to some person—are the same for different people. To me, the most important thing to start with is to ask the question, “Who are my clients? Who are my stakeholders?” Let’s call those people your client community.

    For any piece of writing, I’d like to suggest that there’s an important stakeholder who is often overlooked: you. Your writing is a medium, as Marshall McLuhan would say. That is, your writing is an extension of you and your human capabilities, in particular speech and presence. Writing extends your physical being and what you might say were you immediately, physically present. As such, your written work a stand-in for you. It represents you, literally re-presents you at a different time, in a different place, and in a different form.

    Another significant person in your client community is your direct client, the person you’re working for, the one who has commissioned the work. (That might be your employer, or it might be yourself.) Your writing is what you might say to your client, or on your client’s behalf, if you were actually in the room with your reader.

    As most authorities on writiing will agree, your reader is extremely important and the ongoing focus of your work. Yet media present risk: as much as a medium extends and accelerates and intensifies some human capabilities and the roles of some senses (McLuhan might have said “heats them up”), it also diminishes the roles of others. While writing extends speech and intensifies the role of vision, writing diminishes conversation and the interaction between the speaker and the listener. Your listener can’t ask questions, and as Northrop Frye remarked, a book always says the same thing. Vision becomes hot, and sound becomes cool, requiring the reader to involve himself by constructing the sound of your voice. That calls for caution: if your writing is really going to represent you, it should sound in your reader’s head as you would want it to sound. Trouble is, you can’t ever know that. But you can test your writing, and if you discover problems, you might reveal information that would inform a decision to fix it.

    So one really good way to test a piece of writing is to ask questions about it.

    • How would this sound if I were to utter it out loud?
    • Does it roll off the tongue naturally?
    • Would it sound like me as I’d like to be heard?
    • Would it sound like the sort of thing I would say?
    • Are these the words that I would choose?
    • Would I sound sloppy, or inflated, or confusing if I were to speak this sentence in a conversation?
    • Is there an obvious way to re-interpret or misinterpret this sentence?
    • What are the risks associated with misunderstanding?
    • What could I do to make my meaning more plain?

    And, most importantly,

    • What problem am I trying to help my reader to solve?
    • Have I helped my reader to solve that problem?

    Those questions, if we’re going to answer them well, require us to think carefully about our readers. We’re often writing for people that we haven’t met in person. We’re writing for people who have never heard us. Who might such people be? How do we imagine them? Might some readers be unfamiliar with the topic? Might some be unfamiliar with English? Might some be easily confused, or easily bored? What might we do to help them understand us? Should we provide examples? Stories? Make the message longer? Shorter?

    Here’s another interesting question: what does the finished piece of writing—and the effort that it took to produce it—tell us about the subject of our writing? Might that information tell us something important about the subject? Here’s a hint: the Technical Bulletin, designed to help us interpret where a service is being performed, runs to 53 pages.

    I suspect that no one—neither the writer of the passage I quoted above, nor his or her clients—asked any the questions that I’ve suggested here. Or if those questions were asked, no one cared enough about the information to do anything about it.  Still, as an optimist, I believe that the process is reversible.  If you’re feeling down, one antidote might be to read Orwell’s brilliant essay, and follow its advice.

    And at last, I leave you, dear reader, with a question:  How is all this like testing and software development?