Skip to main content

Background Unit Testing: New evolutions in unit testing and IDE integration

Posted by johnsmart on February 2, 2009 at 12:45 PM PST

An emerging innovation in unit testing is the idea of Continuous Unit Testing, or having your unit tests run in the background whenever you modify your code. In this approach, whenever you save your code, the appropriate unit tests are executed in the background. This avoids the problem of committing changes with broken tests just because you forgot to run the appropriate tests before committing. The trick, of course, is knowing what the appropriate tests are - you don't want have to wait for all your your tests to run every time you save a change. You want to focus on the tests that are most likely to be affected by your code changes. I'm sure this sort of thing will be a standard IDE feature in a couple of years.

For now, however, it isn't, and you need to resort to third-party tools. I am aware of two emerging tools that try to achieve this. The first is JUnitMax, from Kent Beck, one of the pioneers of XP and TDD. The second is Infinitest. Infinitest is free and open source, whereas JUnitMax is available for a modest subscription fee ($2US per month at this stage). I don't think this is a big deal, though it may well put some people off from trying, which would be a shame. Both tools are pretty green, with little in the way of documentation (there is a decent introduction to JUnixMax here).

Infinitest is interesting in that it is supposed to work both with Eclipse and IntelliJ, whereas JUnitMax is purely an Eclipse plugin. Infinitest is actually a separate Java application, that you run from within Eclipse via a Java Application run configuration. The integration with the IDE is therefore far from seamless, though there is a step-by-step tutorial of what you are supposed to do in Eclipse. Unfortunately, when I ran it on a module in my multi-module Maven project, it didn't manage to find any unit tests to run.

JUnitMax comes as an Eclipse plugin, so installing it into Eclipse is more straight-forward. To try it out, I installed the plug-in and made a few trivial changes to one of my unit tests. When I saved the test case, sure enough, JUnitMax kicked off some unit tests in the background, and found an error that I wasn't expecting! JUnixMax runs discreetly in the background, and only makes a fuss if it discovers any failing tests. In this case, unit test errors appear like compilation errors, in the margin of the source code. This is very cool - unit test failures are considered on the same level as compilation errors in the IDE, which is a great visibility boost. It is also a great productivity booster - you don't need to remember to run your unit tests after each change, and you don't have to wait (and possibly be side-tracked) while your unit tests are running.

junitmax-error.png

As you would expect, the failing unit tests are also flagged in the project view, so you are less likely to miss test failures, even in unexpected places:

junitmax-error2.png

Ensuring that your unit tests stay up to scratch is great, but what is really useful is the ability to run the unit tests relating to a particular application class whenever you change the class. So I modified an application class, introducing an error. Sure enough, within a few seconds, a red marker appeared on the corresponding unit test class. As I mentioned earlier, the tricky thing here is knowing what tests to run, so that you can know about failed test as fast as possible.

JUnitMax uses some clever heuristics to guess which tests need to be run first, and to run the fastest tests first. At the end of the day, all of your unit tests are executed in the background whenever you make a change. This is an approximate process, but it seems to work OK. A more accurate technique would require code coverage metrics that back-track to figure out what unit tests run each line of code. This is much harder. Clover can do this, but not in real time. If you integrate Clover with Maven, for example, you can get Maven to run a sub-set of the unit tests based on what code has been changed and what tests execute that code. This is an excellent way to speed up the build process, but it isn't at the stage where you can integrate it smoothly into your IDE yet, as you do with JUnitMax. A perfect JUnitMax would combine these two technologies, so that only the relevant tests are run each time. But, for now, it's a promising start.

"Probably the best training course I've been on."..."Not just how to write Java code but the 'business end' - how to build, test, deploy, manage and monitor"..."One of the best and most useful courses I have attended. And they didn't even try to sell me anything!" - Get up to scratch with the latest in Java tools and best practices! Check out the new 2009 season of the Java Power Tools Bootcamps.

Comments

The Infinitest link is broken, should probably be: http://code.google.com/p/infinitest/

Along the same lines two of systems for Continuous Integration Testing are Hudson and TeamCity. TeamCity has great integration with Intellij's IDEA IDE and Hudson is nice as well. Both are free, though there is a commercial version of TeamCity for larger projects.

Oops, type - thanks, fixed now.

Kent Bent?