DevelopsenseLogo

Testing: Difficult or Time-Consuming?

In my recent blog post, Testing Problems Are Test Results, I noted a question that we might ask about people’s perceptions of testing itself:

Does someone perceive testing to be difficult or time-consuming? Who? What’s the basis for that perception? What assumptions underlie it?

The answer to that question may provide important clues to the way people think about testing, which in turn influences the cost and value of testing.

As an example, an pseudonymous person (“PM Hut”) who is evidently associated with project management in some sense (s/he provides the URL http://www.pmhut.com) answered my questions above.

Just to answer your question “Does someone perceive testing to be difficult or time-consuming?” Yes, everyone, I can’t think of a single team member I have managed who doesn’t think that testing is time consuming, and they’d rather do something else.

This, alas, isn’t an unusual response. To someone like me who offers help in increasing the value and reducing the cost of testing, it triggers some questions that might prompt reframes or further questions.

  • What do the team members think testing is? Do they think that it’s something ancillary to the project, rather than an essential and integrated aspect of software development? To me, testing is about gathering information and raising awareness that’s essential for identifying product risks and steering the project. That’s incredibly important and valuable.

    So when the team members are driving a car, do they perceive looking out the windshield to be difficult or time-consuming? Do they perceive looking at the dashboard to be difficult or time-consuming? If so, why? What are the differences between the way they obtain awareness when they’re driving a car, versus the way they obtain awareness when they’re contributing to the development of a product or service?

  • Do the team members think testing is the mindless repetition of actions and observation of specific outputs, as prescribed by someone else? If so, I’d agree with them that testing is an unpalatable activity—except I don’t call that testing. I call it checking, and I’d rather let a machine do it. I’d also ask if checking is being done automatically by the programmers at lower levels where it tends to be fast, cheap, easy, useful and timely—or manually at higher levels, where it tends to be slower, more expensive, more difficult, less useful, and less timely—and tedious?
  • Is testing focused mostly on confirmation of things that we already know or hope to be true? Is it mostly focused on the functional aspects of the program (which are amenable to checking)? People tend to find this dull and tedious, and rightly so. Or is testing an active search for new information, problems, and risks? Does it include focus on parafunctional aspects of the product—the things that provide important perceptions of real value to real people? Are the testers given the freedom and responsibility to manage a good deal of their own investigation? Testers tend to find this kind of approach a lot more engaging and a lot more interesting, and the results are typically more wide-ranging, informative, and valuable to programmers and managers.
  • Is testing overburdened by meaningless and valueless paperwork, bureaucracy, and administrivia? How did that come to pass? Are team members aware that there are simple, lightweight, rapid, and highly effective ways of planning, recording, and reporting testing work and project status?
  • Are there political issues? Are testers (or people acting temporarily in a testing role) routinely blown off (as in this example)? Are the nuggets of information revealed by testing habitually dismissed? Is that because testing is revealing trivial information? If so, is there a problem with specific testing skills like modeling the test space, determining coverage, determining oracles, recording, or reporting?
  • Have people been trained on the basis of testing as a skilled, sophisticated thinking art? Or is testing something for which capability can be assessed by a trivial, 40-question multiple choice exam?
  • If testing is being done well (which given people’s attitudes expressed above would be a surprise), are programmers or managers afraid of having to deal with the information that testing reveals? Does that lead to recrimination and conflict?
  • If there’s a perception that testing is by its nature dull and slow, are the testers aware of the quick testing approaches in our Rapid Software Testing class (PDF, page 97-99) , in the Black Box Software Testing course offered by the Association for Software Testing, or in James Whittaker’s How to Break Software? Has anyone read and absorbed Lessons Learned in Software Testing?
  • If there’s a perception that technical reviews are slow, have the testers, programmers, or managers read Perfect Software and Other Illusions About Testing? Do they recognize the ways in which careful observation provides us with “instant reviews” (see Perfect Software, page 143)? Has anyone on the team read any other of Jerry Weinberg’s books on software management and measurement?
  • Have the testers, programmers, and managers recognized the extent to which exploratory testing is going on all the time? Do they recognize that issues revealed by testing might be even more important than bugs? Do they understand that every test result and every testing problem points to meta-information that can be extremely valuable in managing the project?

On PM Hut’s own Web site, there’s an article entitled “Why Project Managers Fail“. The author, Jim Benson, lists five common problems, each of which could be quickly revealed by looking at testing as a source of information, rather than by simply going through the motions. Take it from the former program manager of a product that, in its day, was the best-selling piece of commercial software in the world: testers, testing, and the information they reveal are a project manager’s best friends and most valuable assets—when you have the awareness to recognize them.

Testing need not be difficult, tedious or time-consuming. A perception that it is so, or that it must be so, suggests a problem with testing as practised or testing as perceived. Astute managers and teams will investigate that important and largely mistaken perception.

7 replies to “Testing: Difficult or Time-Consuming?”

  1. Thanks for the post Michael. I always find your thoughts on testing very useful.

    I found this post particularly interesting as like many testers I’ve been asked by other team members “Why do you like testing? It’s so boring”.

    To which I typically reply “Because I enjoy it. I left coding for testing because I found the job too repetitive and I prefer the variety I get out of testing.”

    But after reading this post I realize a big red flag should have appeared in front of my face. I should have replied: “If you don’t like testing, why are you in software development?”.

    Michael replies: Brilliant!

    Reply
  2. This reminds me of a test exercise I did a few years ago.

    The ostensible problem was to test a windows application that was running the triangle problem.

    In training, I split the group of twelve people into three groups. Group one I gave the specification, told them to go off into a corner, take all the time they needed, and come up with test cases. Oh, there was the usual complaints about the spec — it was short — about one page. Yet for the complexity of the app, however you want to attempt to measure that (function points?) the spec was very close to a good level of detail that you might /hope/ for on a real-world, complex project.

    For the second team, I demoed the GUI first, then I gave them the spec, told them to go into a different room and write test cases.

    I brought the first and second team back. Then I brought in the third team, that had been sequestered, and had no exposure to the app yet. I allowed the third team to explore, and not be bound any test cases, while the first two had to execute only the test cases they had created.

    After some reasonable timebox, I asked the team members to score the experience from 1 to 10, where one was “I would rather have dental surgery than do this again”, also 1 to 10 in the value of the testing they were providing, and i also asked them to count up the serious problems they found in the app, by what they believed to be root cause. (In other words, don’t count the same defect twice if possible. If 1,1,1 doesn’t count as a equilateral triangle, and neither does 2,2,2, that is one bug.)

    I conducted this exercise twice, and both times, the exploratory team fared the best and the scripted team the worst.

    We could poke holes with my simulation all day — it was an imperfect experiment in lots of ways. My goal, however, was to demonstrate that testing did not have to be boring, brain-dead and no-fun, and that, worse, boring, brain-dead, no-fun testing was often more expensive and less valuable.

    (to the reader) — It might be worth trying something similar at your shop; some people only learn by experience and evidence.

    Reply
  3. “Testing need not be difficult, tedious or time-consuming.” I agree that tedium shouldn’t be part of testing, but difficulty and time consumption could very well be good in testing.

    When I read the word “difficult”, I interpret it as “challenging”, which testing should be. That being said, I can see that, in a negative connotation, difficult could be interpreted as something that is excessively challenging to the point of frustration. As for time-consuming, again it depends on your definition. Good testing sometimes takes a lot of time, especially if the bugs you find are hard to reproduce. Then again, if the testing is repetitive, then time consumption feels like time wasted.

    Reply
  4. Very nice post. I agree with Matt ““difficult” can be interpreted as “challenging”. QA and Testing require an eye to detail and expertise. If you want to ensure quality of a product or application sometime you need to give more time. I think most Quality Assurance and Testing professional will agree that Testing is indeed challenging and exciting job.

    Reply
  5. Hi Michael,

    Thanks for replying to my comment as a whole new post. Let me tell you what, from my experience, is the developers’ problem with testing:

    They don’t mind testing the functionality of their code (although they hate it and some developers claim that it’s not their responsibility even to ensure that their code is actually working, can you believe that?) but they do hate it.

    But, what they really, really loathe is testing if their code meets the business requirements. In my opinion, they’re right in hating this part. The problem with this part is not the difficulty or the challenge, but quite often the ambiguity of this task.

    Reply
  6. Hi,

    This article replicates how broadly you think about a concept. You have kept almost all the possible thoughts and views about what testing is,came to know much from this post. Thanks for sharing it..

    Michael replies: You’re welcome.

    Reply

Leave a Comment