Tom Ball's Blog

The Problem with Unit Testing

When Kent Beck started promoting a test-first best practice as a part of eXtreme Programming, lots of testing experts weighed in on the importance of full-coverage and other metrics in unit tests, invasive strategies for testing private methods, and other heavyweight procedures to ensure test "compliance". My reaction, like most developers, was summed up elegantly by King Arthur in Monty Python and the Holy Grail: "It's the legendary black beast of Aaaaauuuugghhhh! Run away! Run AWAY!!!"

Remember how at the circus there are the trapeze artists that use a net, and the "daring" ones that don't? As a kid I noticed that the non-daring ones actually did much cooler, more difficult tricks, because they had the net! Without the net, the artist has to be more careful and circumspect in his work, or make a big (possibly final) splat in front of his fellow artists. Like most developers, I hate making a big splat (breaking the build, introducing showstopper bugs, etc.) in front of my peers. It's a good aversion to have when working on a team, but it can lead to paralysis if you lack confidence that your changes are correct.

As a development engineer, all I want from unit testing is a safety net, not a huge, QA-approved test monster (for awhile, JavaSoft only had the JCK to do automated testing, which required a very large disk partition and took hours to run). So over the years, I've built several safety nets, keeping them hidden and avoiding names with words like "test", "quality", "verification", etc. The problem with a single tool as a safety net is that like all software it grows over time and becomes its own maintenance headache (to my horror, management learned of one of my test tools and shipped it!).

The advantage of a safety net approach is that it allows me to focus on those areas where I'm most likely to "fall": difficult algorithms, past regressions, murky designs. Shining a unit test light on the latter has proven especially important, as writing tests for some feature I haven't given enough thought to tends to highlight the bad aspects of my design. It also shows testability problems which seem to me to be directly related the cost of maintaining that code. Testable code is maintainable code, and like most developers I would rather write new code than maintain old.

But a safety net approach differs from full testing in that it is much more lightweight practice, which takes less time for me (as a developer) to implement and enhance. Unlike traditional testing approaches which can hurt my individual productivity, having and enhancing my safety net improves my productivity, because like a trapeze artist, I am able to implement more impressive changes to my code without fear of breaking it. These days, I'm a happy JUnit user who finds JUnit's green bars to be a great mood and productivity enhancer. My safety net of unit tests is very incomplete from a QA perspective, but it continues to improve along with my current project.

So what's the problem with unit testing? Unit tests are not tests, but rather a developer's safety net. QA doesn't own unit testing, developers do, and it's an important tool to be shared with your fellow developers on any project. Rename this engineering best practice, and it will be perfect.

Comments

• you're right that part of the problem is with the perception that testing is a monolithic process.

  the test first approach doesn't really attack that either, making tests so all-intrusive that the tests often start to take more time than the coding that's being done.

  Programmer testing should be restricted to the most risky parts of the code, code you don't feel quite comfortable with.
  Those parts you write specific test cases against and write until the code passes those test cases.

  Of course no software should leave your hands that doesn't run and crashes in standard use.
  But IMO it's not the task of the programmer (initially) to take the job of the QA team and do exhaustive testing of every possible scenario. Or rather it's his job to make sure the software passes those tests the QA team can come up with rather than having to come up with the tests himself.

  When writing user interfaces for example I'm constantly surprised by the way non-technical users abuse them, inputting blatantly incorrect information where it's (to me at least) utterly incomprehensible that they'd do something like that.
  We as programmers are burdened by an intimate knowledge of what input our software expects, a knowledge which (subconsciously) hampers our ability to define test cases for all eventualities.

  When was the last time you forgot to test whether that numeric input field was save from entering alphanumeric data for example?
  Or when did you last test for a properly formatted date?
  And that's just some of the more obvious mistakes users will make.

  Posted by: jwenting on October 11, 2004 at 06:53 AM

• I enjoyed this blog, Tom. I'm probably in the camp of people who preaches about automated unit testing as too much of a silver bullet, but you've summed it up eloquently. Without refactoring, you can't have simplicity (your simple abstractions will come up short.) And without testing, you can't trust your refactoring.

  Nice blog.

  Posted by: batate on October 11, 2004 at 11:44 AM

• I think "Unit tests are not tests, but rather a developer's safety net" is the key here. Unfortunately, some QA departments are actually thinking this differently due to confused managers. Thousands of unit test cases are written by QA engineers in these organizations. For QA teams, JUnit’s green bar is the release criterion. Unfortunately, when you look into some of these test cases that turn JUnit’s bar to green, you will find out they cannot be QA’s safety net. Thanks for bringing this up.

  Posted by: twosuns on October 11, 2004 at 12:20 PM

This work is licensed under a Creative Commons License.