John Ferguson Smart's Blog

Continuous Integration build strategies - stage your builds!

So you've got hundreds of tests, but they take ages to run. You have a Continuous Integration server, but it takes an hour to tell anyone when there's a failure. What can you do?

This is where staged builds can come in handy. I basically distinguish fast unit tests from slower integration tests. TestNG test groups are very cool for this, but you can also use simple naming conventions. For example, your unit tests run everything called *Test except classes called *IntegrationTest, whereas your integration tests only run *IntegrationTest.

Unit tests should be snappy, and give feedback within minutes.For integration tests, they should do the job they are intended to do, but time is not really of the essence.

Strictly speaking, unit tests are tests written in isolation, with no interaction with other components. I tend to be a bit more pragamtic here, so some interaction is allowed. However, you do need to be careful loading Spring contexts or Hibernate configurations, as this can be slow. If it slows down your fast unit tests, put it with the integration tests.

Heavy-weight tests, such as performance, integration, or GUI tests, will take longer to do. So these should always go in the integration-test category.

Actually, I often distingush several types of integration test. Here's how I often do it:

• First, a battery of lightning-fast unit and simple integration tests clears 90% of the terrain. Get in there quickly, test quickly, and get out fast. Give feedback on the whole code base within minutes. These tests should be done in a couple of minutes. For the developer, it's all about providing quick feedback. And a lot of errors actually do get caught at this level.
• Secondly, I run the slower tests, which are somewhere between fast unit tests and true integration tests. Things like testing Hibernate configuration files with an embedded database.
• Thirdly, I deploy to an integration server and run the database and GUI tests.
• Finally, I run the test coverage and code quality reports. This can take an hour, there's really no rush.

Each task is a separate build on the build server (say, Hudson). The first kicks off whenever the source code repository is updated. The others kick off once the fast unit tests have succeeded.

I'll be giving a 4 day Java Power Tools Bootcamp in San Francisco, from May 12 to May 15 just after JavaOne. So, if your in SF at this time, come along!

Comments

• I read an interesting post on multistage continuous integration at http://damonpoole.blogspot.com/2008/01/advanced-multi-stage-continous.html

  I think you would instantly get it.

  Posted by: scrumpuppet on March 05, 2008 at 04:53 PM

This work is licensed under a Creative Commons License.