A Unit Test framework Wish List
Nowadays, Java developers have a choice of several unit testing frameworks used by Java developers these days:
- The old, venerable, widely-used JUnit 3.x.
- The cutting-edge TestNG, which tries to address issues found in JUnit 3.x, and take unit testing into the world of Java 5, annotations and IOC.
- The long overdue JUnit 4, which is basically JUnit's answer to TestNG.
Now I don't think I'm all that difficult to please, but none of the above options seem to do everything I want. Just for discussion purposes, here's the top few entries in my wish list for a decent unit testing framework:
- Distinguish unit tests from integration tests : I want to be able to run fast, light-weight unit tests (backed by an embedded memory database or mock objects, for example) on development machines, and proper integration (and performance, why not!) tests on the CI server. These would access the real (well, the test) database and any other servers the application needs to integrate with. These tests take longer to run and sometimes require more fiddling to set up, so they would slow down a developer and discourage him or her from running the unit tests regularly. So I want to be able to tag them to be run uniquely on the CI server. I can do this with the JUnit Ant task (use Ant 1.7 for best results with JUnit 4), but it's messy. I tried to do this with Maven and TestNG (which has some nice supporting features for this), but it doesn't work very well either.
- Work with Maven 2: you might not think this one would be a tall order, but out-of-the-box, neither JUnit 4 nor TestNG are picked up by Maven unit tests. (The fact that Maven doesn't work with JUnit 4 seems to be a known issue (http://jira.codehaus.org/browse/SUREFIRE-31), and there is even a work-around plugin (http://blog.unto.net/work/maven-2-and-junit-4/).)
- Work with IDEs: You should be able to run any unit tests directly in your IDE (as you can with JUnit 3.8 tests in most IDEs I know). JUnit 4 works OK with Eclipse, and there is a plugin for TestNG. NetBeans is another story, though...
What are people's experiences and thoughts in this area? Any other wish-list features?