John Ferguson Smart's Blog

John is a freelance consultant specialising in Enterprise Java, Web Development, and Open Source technologies, currently based in Wellington, New Zealand. Well known in the Java community for his many published articles, John helps organisations to optimize their Java development processes and infrastructures and provides training and mentoring in open source technologies, SDLC tools, and agile development processes. John is principal consultant at Wakaleo Consulting, a company that provides consulting, training and mentoring services in Enterprise Java and Agile Development.

Installing Sonar on a linux build server

Anyone who has read many of my blog entries or articles will know that I'm a great fan of code quality metrics. By code quality metrics, I am referring to coding standards, best practices, complexity, but also to other associated statistics such as the number of unit tests run and the level of code coverage. Code Quality management is an important part of any project, but sometimes it can be difficult to get a global picture. Any given metric, such as code coverage or code complexity, can be difficult to interpret in isolation. This is where Sonar comes into the picture.

If you are serious about Code Quality management, you should know about Sonar. Sonar is a one-stop shop for code quality metrics. If you are using Maven to generate code quality metrics with tools like Checkstyle, PMD, FindBugs, Cobertura, and/or Clover, forget the standard Maven reports - use Sonar instead! Sonar gives you a dynamic view of both snapshot and historical data coming from all of these tools.

In this article, I go through the basic steps involved in setting up your very own Sonar server in a Linux-type environment (it works for Mac OS too, of course), based on my own experience of installing Sonar on a new build server running on CentOS. It's not too hard. First, download Sonar from the Sonar web site:

$ wget http://dist.sonar.codehaus.org/sonar-1.9.2.zip
$ unzip sonar-1.9.2.zip
$ mv sonar-1.9.2 /usr/local

I like to create a symbolic link to applications like this to make upgrades easier:

$ ln -s /usr/local/sonar-1.9.2/ /usr/local/sonar

Sonar works with Maven, so you need Maven (2.0.7 is the minimum supported version - it also works fine with Maven 2.1.0). You will also need JDK 5 or higher.

The basic architecture behind Sonar works at two levels. First, your Maven project generates XML metrics, about test results, code quality, code coverage, and so forth. The Maven Sonar plugin takes this data and sends it to your local Sonar server, where it is processed and incorporated into the Sonar database. You can then go to the Sonar site to view your project's code quality metrics.

Sonar stores its data in a real database, so you will need one available. Although it comes with a built-in Derby database, you shouldn't really use this for anything else than testing. For production use, go to the trouble of setting up a dedicated database for Sonar. Sonar works fine with MySQL, Oracle and Postgres. I set up a local MySQL database for this purpose. Here's how they recommend creating the Sonar database in the documentation:

mysql> CREATE DATABASE sonar CHARACTER SET utf8 COLLATE utf8_general_ci;
mysql> grant all privileges on sonar.\* to 'sonar'@'localhost' identified by 't0ps3cr3t';
mysql> flush privileges;

You also need to modify the conf/sonar.properties file to use the correct database. For example, if you are using MySQL, you need to comment out the lines referring to Derby and uncomment (and modify as required) the following lines:

...
# MySql
# uncomment the 3 following lines to use MySQL
sonar.jdbc.url:                            jdbc:mysql://localhost:3306/sonar?useUnicode=true&characterEncoding=utf8
sonar.jdbc.driverClassName:                com.mysql.jdbc.Driver
sonar.jdbc.validationQuery:                select 1
sonar.jdbc.user                            sonar
sonar.jdbc.password                        t0ps3cr3t

You can start the Sonar server using the sonar.sh script in the OS-specific subdirectory of the bin directory in your Sonar installation, e.g.

# /usr/local/sonar/bin/linux-x86-64/sonar.sh start

# cp bin/linux-x86-64/sonar.sh /etc/init.d/sonar

Now modify the /etc/init.d/sonar file. Add a SONAR_HOME variable pointing to the Sonar installation directory, and a PLATFORM variable matching your platform-specific directory name.

SONAR_HOME=/usr/local/sonar
PLATFORM=linux-x86-64

Next, replace the WRAPPER_CMD and WRAPPER_CONF variables with the following values:

WRAPPER_CMD="${SONAR_HOME}/bin/${PLATFORM}/wrapper"
WRAPPER_CONF="${SONAR_HOME}/conf/wrapper.conf"

Finally, change the PIDDIR variable to "/var/run":

PIDDIR="/var/run"

Now you need to set this up as a service. On Ubuntu or Debian, for example, you might use update-rc.d. On Redhat, Fedora or CentOS, you would use chkconfig

# chkconfig --add sonar

You can now start up the Sonar server as you would any other service:

#/etc/init.d/sonar start

You can now go to the Sonar web page. By default, you will find it on http://localhost:9000/, though you can modify the port and web context in the sonar.properties file if required.

Once your server is up and running, you need to configure your Maven projects to send data to Sonar. This is done using the Maven Sonar plugin.

However, a better approach is to get your CI server to provide the data for you. The key to having good code quality metrics is to have produce them automatically and regularly. A Continuous Integration server is a perfect candidate for the job. For example, you can set up a build job to run a special code metrics build job every night, and send the data to Sonar.

Better still, if you are using Hudson, you can use the Hudson Sonar plugin. The beauty of this plugin is that it lets you integrate Sonar into your Maven projects without changing a line of code. All you need to do is set up the Sonar configuration in Manage Hudson screen. Hudson needs to know how to provide data to the Sonar database, so you need to provide the JDBC connection details.

Next, create a special Sonar build job for each of the projects you want to monitor. Since Sonar treats historical data in 24 hour slices, you only really need to run the code quality metrics on a nightly basis, at least for Sonar. This job just needs to generate the code quality metrics data - you don't need to invoke the Maven Sonar plugin directly. The Hudson Sonar plugin will do this for you - just tick the "Sonar" checkbox further down the screen.

Now you should have a Sonar site up and running, complete with (at least) one project. From now on, Sonar will start collecting metrics, and in no time (well, a few days) you will start to have historical data as well as a snapshot of your current project quality metrics. You might also check out the documentation to see what you can do in Sonar.

<blatent self-promotion>For anyone interested in learning more about integrating Sonar into a nice open source Java development stack build around tools such Maven, Hudson, Checkstyle, PMD, FindBugs, Cobertura, and so on, come along to one of the Java Power Tools Bootcamps! Starting from the London bootcamp, but also in the other sessions (see below), I will be including a brand-new module on Sonar and how to integrate it into your development stack.</blatent self-promotion>

"Probably the best training course I've been on."..."Not just how to write Java code but the 'business end' - how to build, test, deploy, manage and monitor"..."One of the best and most useful courses I have attended. And they didn't even try to sell me anything!" - Coming soon to London, Wellington, Auckland, Canberra and Sydney - Get up to speed with the latest and coolest in Java tools and best practices! Sign up on the 2009 season of the Java Power Tools Bootcamps.

Talking about Maven and Build Automation in London

Wakaleo Consulting is going to London to visit the Queen! Well, the London Java community, at least.

I'll be in London between the 6th and 10th of July to run the first London session of the Java Power Tools Bootcamp with the folks from Skills Matter. There are still a few places left for this session, but you probably should hurry ;-).

While I'm in London, I will be giving a talk at the SkillMatters "In-The-Brain" evening session of the 6th of June from 6:30pm-8:00pm. This will be a rerun of the talk I gave at JavaOne on "Getting Serious About Build Automation: Using Maven in the Real World":

Maven 2 is becoming increasingly popular in larger organizations looking to standardize and industrialize their build processes as well as in smaller shops simply trying to get more out of their builds. This session, for developers wanting to learn about Maven and Maven users wanting to get more out of their build tool, briefly covers the main features and benefits of Maven and then looks at some of the more advanced uses of Maven in the real world, including complex transitive dependency management, dependency conflicts, multimodule projects, and integration with other build systems. It also looks at how the m2eclipse plug-in can be used to improve the Maven user experience and how to use the Nexus repository manager with the Maven release process to publish your APIs and deploy your applications within your organization.

The talk will take place in the SkillsMatter buildings in Clerkenwell, London.

So, see you in London!

"Probably the best training course I've been on."..."Not just how to write Java code but the 'business end' - how to build, test, deploy, manage and monitor"..."One of the best and most useful courses I have attended. And they didn't even try to sell me anything!" - Coming soon to London, Wellington, Auckland, Canberra and Sydney - Get up to speed with the latest and coolest in Java tools and best practices! Sign up on the 2009 season of the Java Power Tools Bootcamps.

The Agile2009 Tools Roundup

The Agile2009 conference in Chicago, coming up in August, is shaping up to be one of the most compelling conferences of the year. I have been helping out organizing the Tools for Agility stage, and there really are some great talks on the geekier side of Agile. In this entry, I want to mention a few gems.

Alistair Cockburn himself will be presenting a not-to-be-missed session on Blitz Planning: "This is the shortest, fastest, easiest way I know to create a project plan...Thanks to Jens Coldewey for first showing this to me in 1998 - it instantly improved the way our teams developed project plans."

Ben Rady and Rod Coffin will be giving a demo on Continuous Testing Evolved: "Continuous Testing (CT) is a developer practice that shortens the feedback loop established by Test Driven Development. It gives you near instant feedback about the correctness of your code, and helps you find bugs as quickly as syntax errors." This idea of Continous Testing is one of the more interesting evolutions in recent development practices, allowing you to conceptually treating unit test failures in the same way as syntax errors in your IDE. Very habit-forming! Rod is

Paul King and Craig Smith will be showing How to make your testing more Groovy:" a swathe of great tools, all centered around how Groovy makes testing so much cooler.

And of course there is the CI vendor cage fight: "In this session, we invite CI tool vendors to give a short demonstration of the best features of their tool. Each vendor will be given 10min to show off the best features of their software, with a further 5min of questions."

There are also lots of talks on Continuous Integration: CITCON dude Eric Lefevre has also done a nice round up of all the Hudson-related presentations, as well as the presentations that discuss other CI tools.

There are heaps of other cool talks, but I don't have room to mention them all: check out the program! I will also be giving a few talks myself:

• Lasse Koskela and myself will be giving a talk on Executable requirements: BDD with easyb and JDave.
• I will be presenting some very practical techniques about managing releases and automating the deployment phase using tools like Maven, Nexus, JIRA, and Bamboo or Hudson in Automated deployment with Maven and friends - going the whole nine yards. The examples in this talk will be using both Bamboo and Hudson, though one of the interesting things is showing how Maven, Nexus, JIRA and Bamboo can be bound together to automated the release cycle.
• Finally, I will be giving a fairly detailed round-up of the latest in development life cycle tools and techniques in Java Power Tools - getting it all together.

So, if you can make it to Chicago in August, don't miss the Agile2009 Conference -see you there!

Java Power Tools Bootcamps in London, Auckland and Wellington

If anyone wants to catch me in London, I will be over that way between the 6th and 10th of July for the first London session of the Java Power Tools Bootcamp, that I am running with the folks from Skills Matter. There are still a few places left for this session, so if you want to come up to speed with the latest all-round development lifecycle tools and techniques, and learn how to set up a development infrastructure and build environment that kicks butt, come along!.

And for those in New Zealand, don't miss out on the sessions in Wellington and Auckland!

"Probably the best training course I've been on."..."Not just how to write Java code but the 'business end' - how to build, test, deploy, manage and monitor"..."One of the best and most useful courses I have attended. And they didn't even try to sell me anything!" - Coming soon to London, Wellington, Auckland, Canberra and Sydney - Get up to speed with the latest and coolest in Java tools and best practices! Sign up on the 2009 season of the Java Power Tools Bootcamps.

Software as a service for your development project infrastructure

For a small-to-medium size company, it makes a lot of sense to externalize infrastructure code such as source code management, issue tracking, project wikis, and so on. All of these tools need server space and admin time, which of course costs money.

Atlassian, for example is offering JIRA Studio, which comes with hosted instances of JIRA (bug tracking and project management), Confluence (wiki on steroids), Fisheye and Crucible (code browser and awesome code review tool) and a Subversion repository, with a very modest price tag (starting at around $25 per user at the time of writing). They are even throwing in GreenHopper, a very cool JIRA plugin that transforms JIRA into a graphical Agile project management tool. All that's missing is the Continuous Integration tool (Bamboo integrates very smoothly with this stack, but I guess they are still working on the best way to virtualise a CI instance, which is quite an interesting problem in itself). Hosting is done by Contegix, who are a very competant bunch indeed. They do great Grails hosting as well.

The marketplace would be no fun without a bit of competition, however. Collabnet, the company behind Subversion, have their own hosted "ALM" infrastructure offering called TeamForge. I'm not familiar with the details of this package, but it appears to contain a suite of packaged open source tools, including (naturally) a Subversion instance, Trac for issue management and for the project wiki, a Hudson instance preconfigued with heaps of plugins for Continuous Integration, and some sort of Eclipse integration. It also comes with support for Cloud-like deployment for testing, and release workflow support. The tools appear to be bundled together and managed through a single web interface. The cost is apparently comparable to the JIRA offering. The focus is a little different from the Atlassian offering, but this package too looks quite compelling, and worth investigating.

I find both these offerings, and this trend in general, very appealing, in a Tayorian sort of way. Specialised hosting companies will be able to administrate your software infrastructure more efficiently and at less cost than you, because that's their job - they do it all day. Why not let them worry about keeping your JIRA and Subversion instances up and just get on with the coding, which is, after all, your job!

Java Power Tools Bootcamps in London, Auckland and Wellington

If anyone wants to catch me in London, I will be over that way between the 6th and 10th of July for the first London session of the Java Power Tools Bootcamp, that I am running with the folks from Skills Matter. There are still a few places left for this session, so if you want to come up to speed with the latest all-round development lifecycle tools and techniques, and learn how to set up a development infrastructure and build environment that kicks butt, come along!.

And for those in New Zealand, don't miss out on the sessions in Wellington and Auckland!

"Probably the best training course I've been on."..."Not just how to write Java code but the 'business end' - how to build, test, deploy, manage and monitor"..."One of the best and most useful courses I have attended. And they didn't even try to sell me anything!" - Coming soon to London, Wellington, Auckland, Canberra and Sydney - Get up to speed with the latest and coolest in Java tools and best practices! Sign up on the 2009 season of the Java Power Tools Bootcamps.

JavaOne - my personal favorite sessions

As usual, JavaOne was a great networking opportunity, and I caught up with old friends, made new ones, and met up with people I had only ever known virtually. This year I was giving a session myself, so I didn't get to as many sessions as I would have liked to. However, I did attend a few gems. Here are some of them:

Neil Ford talked about a variety of unit testing tools in Unit Testing That Sucks Less: Small Things Make a Big Difference">, going from JUnit-based techniques such as Hamcrest asserts, MockRunner, and Groovy as a unit-testing language, through to more exotic tools such as Infinitest (which seems to have made a lot of progress since I last looked at it), Jester, and the intriguingly-named Spock (an interesting BDD Groovy-based framework with a very concise and expressive DSL).

Dave Klein spoke about Grails Integration Strategies, presenting an excellent case for the use of Grails in a broader enterprise context, and not just for small isolated and/or internal web applications. He showed a number of useful tricks such as how to configure a Grails app to talk to EJBs, to use JNDI resources, and to access legacy databases.

I also caught an interesting session from the guys at LinkedIn called JDBC? We Don’t Need No Stinkin' JDBC: How LinkedIn Scaled with memcached, SOA, and a Bit of SQL, which was about scaling apps using a combination of BASE and ACID. Although this mixture would usually produce only salty water, the folks at LinkedIn have managed to use a comibination of fast ACID updates when really required, and slower updates when you can get away with it to design a very highly scalable architecture.

And the Groovy/Grails BOF was a lot of fun.

Would have liked to have heard Kosuke on Hudson, but unfortunately I was giving my own presentation at that time - too bad!

Anyway, another great JavaOne - looking forward to catching up with everyone at some of the other upcoming conferences (I'm thinking Agile2009 and Groovy, in particular), or at JavaOne next year!

If anyone wants to catch me in London, I will be over that way between the 6th and 10th of July for the first London session of the Java Power Tools Bootcamp, that I am running with the folks from Skills Matter. There are still a few places left for this session, so if you want to come up to speed with the latest all-round development lifecycle tools and techniques, and learn how to set up a development infrastructure and build environment that kicks butt, come along!. On the 6th, I will also be speaking at the Skills Matter In-The-Brain evening session (6:30 - 8:00pm).

"Probably the best training course I've been on."..."Not just how to write Java code but the 'business end' - how to build, test, deploy, manage and monitor"..."One of the best and most useful courses I have attended. And they didn't even try to sell me anything!" - Coming soon to London, Wellington, Auckland, Canberra and Sydney - Get up to speed with the latest and coolest in Java tools and best practices! Sign up on the 2009 season of the Java Power Tools Bootcamps.

Slides from the Maven talk at JavaOne

This year at JavaOne, I gave a talk entitled 'Getting Serious About Build Automation: Using Maven in the Real World'. The talk summary is as follows:

Maven 2 is becoming increasingly popular in larger organizations looking to standardize and industrialize their build processes as well as in smaller shops simply trying to get more out of their builds. This session, for developers wanting to learn about Maven and Maven users wanting to get more out of their build tool, covers the main features and benefits of Maven and then looks at some of the more advanced uses of Maven in the real world, including complex transitive dependency management, dependency conflicts, multimodule projects, and integration with other build systems. It also looks at how the m2eclipse plug-in can be used to improve the Maven user experience and how to use the Nexus repository manager with the Maven release process to publish your APIs within your organization.

You can download the slides for this talk here

"Probably the best training course I've been on."..."Not just how to write Java code but the 'business end' - how to build, test, deploy, manage and monitor"..."One of the best and most useful courses I have attended. And they didn't even try to sell me anything!" - Coming soon to London, Wellington, Auckland, Canberra and Sydney - Get up to speed with the latest and coolest in Java tools and best practices! Sign up on the 2009 season of the Java Power Tools Bootcamps.

A quick primer on testing Selenium in Groovy

Selenium is a widely-used and very useful tool for automated web testing. This article is a very short primer designed to get you up and running Selenium Tests in Groovy in your Maven projects.

The examples assume you are running your Selenium tests against an existing web server (say an integration or staging server). In this case, I typically set up a dedicated Maven project to do the job. This project will start up a Selenium client, run some Selenium integration tests, and then shut down the Selenium client. An alternative approach is to start your application up in Jetty, which works fine as well, but involves a bit more configuration in the pom.xml file.

The test cases are written in Groovy, primarily because it is a real pleasure to write anything in Groovy :-). Easyb is also an excellent alternative that works well with Selenium. The Groovy tests by default go in the test/src/groovy directory. You can set up Groovy in your Maven project by adding the gmaven plugin, as shown here:

<plugin>
  <groupId>org.codehaus.groovy.maven</groupId>
  <artifactId>gmaven-plugin</artifactId>
  <version>1.0-rc-4</version>
  <executions>
    <execution>
      <goals>
        <goal>generateStubs</goal>
        <goal>compile</goal>
        <goal>generateTestStubs</goal>
        <goal>testCompile</goal>
      </goals>
    </execution>
  </executions>
</plugin>

Next, you need to set up Selenium in your Maven project. You can use the Selenium plugin for this:

<plugin>
  <groupId>org.codehaus.mojo</groupId>
  <artifactId>selenium-maven-plugin</artifactId>
  <version>1.0-rc-1</version>
  <executions>
    <execution>
      <id>xvfb</id>
      <phase>pre-integration-test</phase>
      <goals>
        <goal>xvfb</goal>
      </goals>
    </execution>
    <execution>
      <id>start</id>
      <phase>pre-integration-test</phase>
      <goals>
        <goal>start-server</goal>
      </goals>
      <configuration>
        <background>true</background>
        <logOutput>true</logOutput>
      </configuration>
    </execution>
    <execution>
      <id>stop</id>
      <phase>post-integration-test</phase>
      <goals>
        <goal>stop-server</goal>
      </goals>
    </execution>
  </executions>
</plugin>>

Note that we have set up the Selenium client to start just before the integration tests, and to close down once the integration tests are finished. We also need to configure the Surefire plugin to run the Selenium tests during the integration-test phase. You can use whatever convention you want - in this case, the Selenium tests all go in a package called 'selenium', placed somewhere in the project class structure. In the following configuration, we ensure that these tests are only run during the integration-test phase:

<plugin>
  <artifactId>maven-surefire-plugin</artifactId>
  <configuration>
    <excludes>
      <exclude>**/*IntegrationTest.*</exclude>
      <exclude>**/selenium/**</exclude>
    </excludes>
    <forkMode>never</forkMode>
  </configuration>
  <executions>
    <execution>
      <id>integration-tests</id>
      <phase>integration-test</phase>
      <goals>
        <goal>test</goal>
      </goals>
      <configuration>
        <excludes>
          <exclude>**/*Base.java</exclude>
          <exclude>**/*$*</exclude>
        </excludes>
        <includes>
          <include>**/*IntegrationTest.*</include>
          <include>**/selenium/**</include>
        </includes>
      </configuration>
    </execution>
  </executions>
</plugin>

We will also need a dependency on the Selenium Java client classes for our test code later on:

<dependencies>
    <dependency>
      <groupId>org.seleniumhq.selenium.client-drivers</groupId>
      <artifactId>selenium-java-client-driver</artifactId>
      <version>1.0-beta-2</version>
      <scope>test</scope>
    </dependency>
</dependencies>

The Groovy classes themselves are fairly simple. We start off with a base class, which will set up the Selenium client, and shut it down cleanly after the tests. We use a property called base.url to ensure that we can run the tests against different servers as required:

public class SeleniumTestBase {

    /**
     * Only create one instance of this server for all the tests.
     */
    private static Selenium seleniumInstance

    /**
     * Base URL defined by the ${base.url} property.
     * Child classes should use this variable to ensure that tests are run
     * against the right test server.
     */
    private static String baseUrl;

    protected static synchronized Selenium getSelenium() {
        
        if (!seleniumInstance) {
            seleniumInstance = new DefaultSelenium("localhost", 4444,
                                                   "*firefox", getBaseUrl())
            seleniumInstance.start()
        }
        return seleniumInstance
    }

    public static String getBaseUrl() {
        if (!baseUrl) {
            baseUrl = System.getProperty("base.url") ?: ""
        }
        return baseUrl
    }

    @AfterClass
    public static synchronized void shutdown() { 
    	seleniumInstance.close()
    	seleniumInstance = null
    }    
}

Finally, the tests themselves are derived from this class, and use the selenium and baseUrl public class VersionNumberTest extends SeleniumTestBase { @Test void homePageShouldHaveAVersionNumberInFooter() { selenium.with { open "${baseUrl}/index.html" waitForPageToLoad "5000" assert isTextPresent(/regexp:myapp version \d+\.\d+\.\d+/) } } ... }

That should be enough to get you started with Selenium tests written in Groovy for your Maven projects - have fun!

"Probably the best training course I've been on."..."Not just how to write Java code but the 'business end' - how to build, test, deploy, manage and monitor"..."One of the best and most useful courses I have attended. And they didn't even try to sell me anything!" - Coming soon to London, Wellington, Auckland, Canberra and Sydney - Get up to speed with the latest and coolest in Java tools and best practices! Sign up on the 2009 season of the Java Power Tools Bootcamps.

My JavaOne Agenda

In case you missed it, JavaOne is on next week. There are some great sessions as usual, and it should be a great opportunity for catching up with old and/or virtual friends, and networking in general. So, as some others have done, I've decided to publish my intended JavaOne agenda:

So if anyone wants to catch up while I'm in San Francisco, just give a yell!

CI Adoption Stories (7/8) Clever tools to optimize Continuous Integration

This case study is the seventh of an 8-part blog series about why so many developers adopt continuous integration, and originally published on the Atlassian blogs. It naturally has a bit of a Bamboo slant to it (if you look close enough, you can even see the odd panda), but the issues discussed apply to any team trying to adopt modern development practices, no matter what CI tool is used.

Clever tools to optimize Continuous Integration

Sandra works in a large government service in Toronto. She is in charge of a team of seasoned developers whose job is to provide technical support and mentoring to other teams, mostly made up of less experienced developers. Recently, her team has been called in to help out a team full of very inexperienced Java developers on an important external-facing web application. These developers, recently brought over from COBOL and the world of mainframes, have a basic Java training but very little experience in the way of best practices. Her team's job is to mentor these inexperienced developers, teaching them the tricks of the trade, so to speak.

Appealing plugins

Metrics play a key role in this task. Sandra set up a special "code quality" build plan, which runs a swathe of static analysis tests (including Checkstyle, PMD, and FindBugs), and also comprehensive code coverage reporting using Clover. The Bamboo plugins for each of these tools give her a high-level picture of how these metrics evolve over time. This is an important tool for her to keep tabs on how well the trainee developers are applying the techniques that they are being taught. For example, low or dropping code coverage on a class may indicate that one of the new developers is having trouble assimilating the test-driven development and testing practices that her team is trying to teach.

Although a high-level view is just what the doctor ordered when you want to keep tabs on the overall health of your development practices, there are still times when you would like to be able to drill down into the details, and see exactly what the violations were in a particular build, or what lines of code were not being tested. To this end, Sandra stores the HTML reports generated by each tool as build artifacts. This way, she can drill down into the detailed reports of any build whenever she wants.

Watertight reports

These reports are also an important part of Sandra's training strategy. She reviews both the high-level statistical views and the detailed reports in a weekly code quality meeting, where any new issues or trends are discussed. In one of the first meetings, for example, Sandra pointed out the increasing number of Checkstyle violations related to an absence of Javadocs. A short talk on the merits of commenting ones code was enough to put most of the developers back on the right track. The issues raised by FindBugs, for example, are particularly educational, and are often used to discuss some of the finer points of Java programming. In one instance, for example, the discussion over one FindBugs issue reveiled a fundamental misunderstanding about the immutability of Java Strings, which was rectified on the spot by an impropmtu tutorial on the topic of Java Strings.

"Probably the best training course I've been on."..."Not just how to write Java code but the 'business end' - how to build, test, deploy, manage and monitor"..."One of the best and most useful courses I have attended. And they didn't even try to sell me anything!" - Coming soon to London! Get up to scratch with the latest in Java tools and best practices! Sign up on the 2009 season of the Java Power Tools Bootcamps.

CI Adoption Stories (6/8): Automating the staging and production deployments

This case study is the sixth of an 8-part blog series about why so many developers adopt continuous integration, and originally published on the Atlassian blogs. It naturally has a bit of a Bamboo slant to it (if you look close enough, you can even see the odd panda), but the issues discussed apply to any team trying to adopt modern development practices, no matter what CI tool is used.

Automating the staging and production deployments

Testing intermediate builds is one thing, moving to General Availability and production deployments are a whole other subject. Steve and Simon, the developers in story 5, also had this issue to grapple with as prior to using Bamboo this was an entirely manual process. When they moved to using Bamboo for Continuous Integration they also upgraded their issue tracker to JIRA which made for a very pleasant surprise.

The pain of manual builds

Originally, this was a manual process, requiring a developer to run the mvn release:prepare and mvn release:perform commands from the command line. Maven would prompt for the next version number, which had to be entered by hand. If an error or mistake occured, or the process was interrupted half-way through, things had to be tidied up by hand. Many times, an incorrect version number was deployed, and the Subversion tags had to be removed by hand afterwards. In addition, an obscure Subversion bug meant that occasionally the release:prepare phase would fail for no reason. Although the work-around was relatively simple (just run an svn update and re-run the release:prepare command), this increased the complexity of the process and added one more thing that could go wrong. It also duplicated maintenance work between the version numbers in JIRA (where issues were managed) and in the Maven project itself.

Preparing Maven, JIRA and Bamboo

In Steve and Simon's project, Steve now uses JIRA and Bamboo to help automate the build promotion process. The version numbers of each release are maintained in JIRA. There is generally a maintenance release once a week, on Friday night, and a major new release every two months or so. Version numbers use the default three-digit Maven convention (e.g. 2.0.44). Each version released to staging is given a unique version number. Automated deployments to the integration server use snapshot versions (e.g. 2.0.45-SNAPSHOT).
Since their project uses Maven, he uses the Maven Release Plugin to help manage the build promotions. This powerful plugin checks that all the latest changes had been committed to Subversion, creates a new tag in the Subversion repository with a label matching the next release number, and updates the Maven project version number. In a second phase, the tagged code is checked out, built, tested and deployed to the Nexus repository, ready to be used during the staging deployment.

Using Bamboo and the JiraVersion plugin, Steve was able to entirely automate this phase. The Maven Release plugin can be run in batch mode, without any human interaction. By default, it will simply increment the current version number. However, you can also provide the next build number as a parameter. This makes it easy to integrate the JiraVersion plugin, which will pass the selected version number as a parameter to your build script. In Bamboo, Steve set up a build plan configured with the JiraVersions plugin. This plan proposes a list of available JIRA versions for deployment. Steve simply chooses the version that he wants to deploy to kick off the build promotion.

Practice makes perfect

With a little work, Steve was able to automate the process and create a series of Bamboo build plans that authorized users can now use to deploy to the right environment with the click of a button. In Bamboo, there are now two special "release" build plans, one for staging and one for production. The staging and production build plans are run on remote agents, directly on the staging and production servers. Each build plan can only be executed by people authorized to deploy to staging and production respectively. These build plans propose a list of available JIRA versions for deployment, and the user simply needs to choose the right version number to kick off a full-blown deployment.

To get around the Subversion bug mentioned above, Steve wrote a Groovy script that automates the whole process, including the work-around if the Subversion issue occurs.

Now, whenever a release is ready to go onto staging or production, the build promotion process is as simple as clicking on the corresponding link in the list of available version numbers.

"Probably the best training course I've been on."..."Not just how to write Java code but the 'business end' - how to build, test, deploy, manage and monitor"..."One of the best and most useful courses I have attended. And they didn't even try to sell me anything!" - Coming soon to London! Get up to scratch with the latest in Java tools and best practices! Sign up on the 2009 season of the Java Power Tools Bootcamps.

JavaOne: Getting Serious About Build Automation: Using Maven in the Real World

This year, I'll be talking at JavaOne about Getting Serious About Build Automation: Using Maven in the Real World (TS-3941):

Maven 2 is becoming increasingly popular in larger organizations looking to standardize and industrialize their build processes as well as in smaller shops simply trying to get more out of their builds. This session, for developers wanting to learn about Maven and Maven users wanting to get more out of their build tool, covers the main features and benefits of Maven and then looks at some of the more advanced uses of Maven in the real world, including complex transitive dependency management, dependency conflicts, multimodule projects, and integration with other build systems. It also looks at how the m2eclipse plug-in can be used to improve the Maven user experience and how to use the Nexus repository manager with the Maven release process to publish your APIs within your organization.

In this talk, I'll be discussing some of the finer points of Maven development, including:

• Some more advanced uses of multi-module projects, project inheritance, and dependency management
• Revving up the release process with automated releases, Hudson and Nexus
• Doing up your deployment process - towards one-click deployments

The talk is on Wednesday, June 3, 9:45-10:45 (Esplanade 300). Last time I looked, there were still about 20 places left for this talk.

I'll be in San Francisco all week, so if anyone wants to catch up or hang out, just give a yell!

"Probably the best training course I've been on."..."Not just how to write Java code but the 'business end' - how to build, test, deploy, manage and monitor"..."One of the best and most useful courses I have attended. And they didn't even try to sell me anything!" - Get in quick - there are still some places available on the London course! Get up to scratch with the latest in Java tools and best practices!

CI, Branching and Release strategy for a medium-size project - a case study

Someone recently asked me about how I organised releases in our Subversion repository. So I decided to write up the response as a blog entry. This article describes how we have set up Continuous Integration and release management for a medium size web project for one of our clients. The approaches described here represent some typical practices that are (I think) fairly commonnly used for this size of project.

This project has a large legacy code base, apparently from the dark times before Struts, when it was fasionnable to roll your own MVC frameworks. There are some unit tests, but the code coverage is only around 20%. The original developers have moved on, and the new developers tread with caution when modifying the code, for fear of inadvertently breaking things elsewhere. Development is done by developers from several different companies, on different sites.

For this project, we use a central Subversion repository. However, local development work can be done using Git, as is the case on my machine. This allows fast local branching and merging, and intermediate commits without confusing other developers. We use FishEye to keep track of changes, and Crucible for online code reviews. For this reason in particular, Subversion is likely to remain as the central repository for the time being.

Development work is done on the main trunk. A release branch is used for release preparation. During the release preparation stage, change sets are merged from the trunk into the release branch. This requires more work than releasing directly from the trunk, but gives more control over what is actually released. It also allows certain high-priority changes to be "fast-tracked" directly into the release branch before other features have been completed or stabilized. However, it introduces the risk of errors if the features are not correctly migrated to the release branch. So regression testing is vital.

A Hudson Continuous Integration server runs unit, integration and web tests whenever the code on the trunk is modified. If the unit and integration tests succeed, the application is automatically built and deployed to a test server.

There are several test servers. First, every change on the trunk results in a new automatic deployment onto the integration server. Selenium web tests are run against this server as part of the Continuous Integration process. The Selenium tests are slow, but the application makes heavy use of Google Maps and other Javascript and Ajax-based technologies, and I have yet to test frameworks like HTMLUnit in this regard.

This sort of application is a good example of where web tests can be really useful. Starting from scratch, we are progressively building up a suite of Selenium regression tests that are run against every new release. These tests are set up in their own separate Maven module, and can be configured to run against any server via a command line parameter.

The release branch has its own set of Continuous Integration build jobs, which run the full tests against every new version, and automatically deploys the application to a "pre-release" integration server. The release branch contains "release candidate" versions. When we are happy with the version on the release branch, we use the Maven Release plugin (run as a manual Hudson build job) to release an official version that is deployed to the Nexus server.

This released version can then be deployed to staging and production. This is done using parameterized Hudson build jobs - when you run the build job, Hudson prompts you for the release version to be deployed. Under the hood, this version number is passed as a command-line property to a special deployment project, which fetches the appropriate WAR file from Nexus and patches the configuration files for the target environment.

"Probably the best training course I've been on."..."Not just how to write Java code but the 'business end' - how to build, test, deploy, manage and monitor"..."One of the best and most useful courses I have attended. And they didn't even try to sell me anything!" - Get in quick - there are still some places available on the London course! Get up to scratch with the latest in Java tools and best practices! Sign up on the 2009 season of the Java Power Tools Bootcamps.

A whimsical ode to JavaOne

Now look at them coders, that's the way you do it
You code in Groovy in that IDE
That ain't workin' that's the way you do it
Get your lines of code for nothin' get your screens for free
Now that ain't workin' that's the way you do it
Lemme tell ya them dudes are cool
Maybe write twenty lines for your domain classes
And a closure for a business rule 

We gotta code in Java 1-dot-4
No annotations, EJB 2
We gotta deploy to IBM Websphere
We gotta talk to COBOL and DB2

See that dude ahacking with the iPhone and the macbook
Yeah buddy did you see his car
That dude just rewrote ebay on his laptop
That dude ahacking he's a real rockstar

We gotta follow enterprise standards
We gotta fill out those change requests
We gotta wait for the specifications
In scope, in budget, and on schedule

I shoulda learned to code in Groovy
I shoulda learned to code in Grails
Look at that dude, he just rewrote Twitter in a demo
Man he makes my code look pale
And he's up there, what's that? database persistence?
Just typing in your field names in an IDE
That ain't workin' that's the way you do it
Get your lines of code for nothin' get your screens for free

We gotta code in Java 1-dot-4
No annotations, no scripting here
We gotta deploy to IBM Websphere
We gotta talk to mainframe tiers

Now that ain't workin' that's the way you do it
You play with Groovy in your IDE
That ain't workin' that's the way you do it
Lines of code for nothin' get your screens for free
Lines of code for nothin' get your screens for free

"Probably the best training course I've been on."..."Not just how to write Java code but the 'business end' - how to build, test, deploy, manage and monitor"..."One of the best and most useful courses I have attended. And they didn't even try to sell me anything!" - Coming soon to London! Get up to scratch with the latest in Java tools and best practices! Sign up on the 2009 season of the Java Power Tools Bootcamps.

CI Adoption Stories (5/8): Optimizing Continuous Integration to push build artifacts

This case study is the fifth of an 8-part blog series about why so many developers adopt continuous integration, and originally published on the Atlassian blogs. It naturally has a bit of a Bamboo slant to it (if you look close enough, you can even see the odd panda), but the issues discussed apply to any team trying to adopt modern development practices, no matter what CI tool is used.

Optimizing Continuous Integration to push build artifacts

Simon Smith and Steve Peters are Java developers working on a large web application alongside 5 other developers, spread over two geographical sites. The application is designed using a clean modular architecture. The project uses Maven 2, which helps define and enforce the modular architecture. It consists of ten distinct Maven modules representing different parts of the system, with well-defined dependencies between the module. For example, there is a Maven module for the core domain logic, and another for the web layer. Steve has also written a specialized library encapsulating some customized localization features. Source code is stored in a Subversion repository. Build artifacts (JAR and WAR files) are deployed to a Nexus Maven repository, to be made available to other developers.

The Maven repository distinguishes between snapshot versions, which typically correspond to a nightly build or to the latest code changes made to the version control system, and release versions, which are more stable, numbered releases. Developers in the same team will typically use snapshots reflecting the latest code changes, whereas release versions are usually for external use where more stability is required. This approach does an excellent job of enforcing modularity and keeping dependencies clean.

Life before Continuous Integration

However, before CI was introduced, their approach to build artifacts was a source of frequent integration headaches. For example, if Steve corrected a bug in the localization module, he needed to first commit his code changes to Subversion, and then to manually deploy a new snapshot onto the Maven repository. Only then would his corrections be available to Simon. The deployment phase was time-consuming, as it involved executing a complete build and test cycle, and then transferring the file to the Nexus server on the other geographical site - depending on the size of the module, it could take between 5 and 20 minutes, several times a day, for the whole process to complete. Steve would occasionally forget this second step, which resulted in his corrections not being visible to Simon, even though they had been committed to Subversion. This also happened occasionally in the nine other modules.

The main impact of this sort of incident was lost time. The problem was relatively insidious, as in theory at least there was a simple work-around - Simon just needed to check out Steve's module and do the deployment himself. However this would interrupt Simon in his thought-stream, and oblige him to work on something else for 15 to 30 minutes. The problem was also hard to isolate: one time, it took 4 hours of debugging before Simon finally tracked a problem down to an undeployed snapshot. And if something went wrong, Simon was hard-put to trouble-shoot the issue.

Switching to Bamboo

This sort of wasted time prompted the team to look for a more automated solution. So, to make the overall integration process smoother, the team decided to install a Bamboo server. There is now a dedicated set of build plans for each Maven module, that compile and test the code whenever changes are made, run code quality and code coverage metrics, and notifies developers if something went wrong. In other words, from a CI perspective, each module can be considered as a separate CI project.

Since the development efforts are now closely integrated, modules use snapshot dependencies to ensure that they always have the latest code changes. Each module also has a second build plan, which automatically deploys the build artifact to the enterprise repository whenever the first build succeeds. Since the initial build runs a battery of unit and integration tests, a minimum of stability is ensured. For Steve and Simon, the result has been a massive time saver. Now, every developer can now access the latest stable versions of the libraries with no manual intervention required. The deployment still takes 5 to 20 minutes, several times a day, but it happens automatically, with no human intervention, and with no interruption in the developer's concentration.

"Probably the best training course I've been on."..."Not just how to write Java code but the 'business end' - how to build, test, deploy, manage and monitor"..."One of the best and most useful courses I have attended. And they didn't even try to sell me anything!" - Coming soon to London! Get up to scratch with the latest in Java tools and best practices! Sign up on the 2009 season of the Java Power Tools Bootcamps.

Where To Now With Build Automation

Just posted a new article about Continuous Integration and Build Automation on InfoQ, that I wrote in collaboration with my mates at Atlassian:

An effective Continuous Integration environment can save your team time, money and even existential angst. It can enable bugs to be discovered earlier, their cause identified more easily, and ultimately get them resolved more efficiently. It can encourage better source code management practices, help you leverage automated analysis tools, encourage better testing, track your progress and remove bottlenecks from your developers lives. It can facilitate the deployment process, and make your releases go more smoothly and more reliably. Managers will have more charts than they know what to do with and developers will be happier. To put it another way, not using Continuous Integration is like developing software using Notepad, it's possible but it's horribly inefficient.

Check out the full article!

"Probably the best training course I've been on."..."Not just how to write Java code but the 'business end' - how to build, test, deploy, manage and monitor"..."One of the best and most useful courses I have attended. And they didn't even try to sell me anything!" - Coming soon to London! Get up to scratch with the latest in Java tools and best practices! Sign up on the 2009 season of the Java Power Tools Bootcamps.

CI Adoption Customer Stories (4/8): Driving quality with hyper complex distributed builds

Sophie is a technical project manager in a large insurance firm. She manages the development of a web-based calculator for car insurance premiums. The calculations are done dynamically, using a sophisticated AJAX-based web interface. Car insurance premiums use complex algorithms, and it is vital that the web calculator results match those coming from the mainframe back-end to the cent. As a result, the application has a very large battery of unit tests - some 6000 unit tests are run against the calculator alone. The application also has web-based functional tests, using Selenium, and load tests using JMeter. Each build of the application needs to be tested against four different browsers (IE6, IE7, Firefox, and Safari). The team has also written a comprehensive test of integration tests which run against a production-scale Oracle test database and a CICS mainframe back-end.

All together, the full build can take up to an hour to run. Far too long for an effective Continuous Integration set up.

There is also another problem: although there are effectively Selenium test scripts that run smoke tests and functional tests against the deployed application, they need to be run in several different browsers, and on different operating systems. All of the targeted browsers can run on a Windows platform, but the department build server is set up on Linux box, and the QA department want the automated Safari tests to be run on an OS X machine.

The case for Continuous Integration

Distributed builds provided a good answer to both of Sophie's major problems. A distributed build architecture lets you run build jobs across several machines, providing potentially huge performance gains. It also enables you to run specific builds on dedicated machines, which makes it possible to run OS or environment-specific tests or build steps.
To resolve these problems, Sophie used a distributed Continuous Integration environment based on Bamboo. The CI setup consists of a main build plan, which compiles and runs the unit tests, along with build plans for functional tests running against different browsers (IE6, IE7, Firefox, and Safari), as well as other plans for the load and integration tests. Finally, there is a separate build plan for code coverage and code quality metrics. Seven build plans in all.

In Bamboo, it is a straightforward task to set up remote agents for your distributed builds. The main Bamboo server coordinates the builds, and runs build jobs either locally or on remote agents, depending on their availability. You can also use "capabilities" to help decide where a particular build job should be run. For example, Sophie has set up build agents on 5 machines: 2 recent and powerful Linux machines, 2 older Windows XP boxes, and a brand new iMac. The Firefox tests can be run anywhere, but the IE tests need to be run on the Windows machines, and the Safari tests are to be run on the Mac. She has added a custom capability called "operating.system" to help Bamboo know where to run each of the functional test build plans. Then, all she needs to do is to add an extra Requirement to each build plan to indicate what operating system that particular plan needs.

Load tests are done by deploying the application to a dedicated test server and then running a JMeter script on the build agent. Load tests are particularly processor-intensive, and for best results need to be run on fast machines. Sophie only wants load tests running on either one of the fast Linux machines, or the iMac. To do this, she has added another custom capability called "high.performance" to identify the machines with enough power to run the load tests correctly.

Syncing the build artifacts

One other issue that Sophie had to resolve was distributing the build artifacts. The normal operating procedure in most CI setups is to check out the latest copy of the source code, compile and build the application, and then run unit tests, integration tests, and so forth. In Sophie's case, this would mean that the application would be compiled, unit tested, and bundled up into a WAR file seven types for each build!

Sophie's project uses Maven 2, so she was able to use Maven's support for snapshot releases and dependency management to optimize things in this area. The initial build plan compiles, runs unit tests, generates a WAR file, and deploys the WAR file to the Enterprise Maven snapshot repository, making it available for the other build jobs. This initial build is the only build to be triggered off by changes in the source code repository - all of the other builds are dependent on this one. The integration, functional and performance tests are set up as separate Maven projects, thus avoiding the need to rebuild and rerun the unit tests each time. These build jobs automatically download the latest application snapshot and run the appropriate tests against this version. For the functional tests, Maven profiles are used to determine which browser to use in each case.

Making Agile Real: the Agile2009 conference

The official Agile2009 is out, and it looks fantastic! The Agile2009 conference, in Chicago, August 24 to 28, is chock-a-block with interesting and useful sessions with a swathe of Agile rock stars. I was involved with organizing the Tools stage this year, and I can confirm that the selection process was hard. There are way too many great sessions to fairly highlight just a few, but sessions like the CI vendor cage-fight should prove very popular, and don't miss on any account the "Programming with the stars" sessions at lunchtime!

I will be giving three talks at Agile2009 this year:

• Lasse Koskela and myself will be giving a talk on Executable requirements: BDD with easyb and JDave.
• I will be presenting some very practical techniques about managing releases and automating the deployment phase using tools like Maven, Nexus, JIRA, and Bamboo or Hudson in Automated deployment with Maven and friends - going the whole nine yards.
• Finally, I will be giving a fairly detailed round-up of the latest in development life cycle tools and techniques in Java Power Tools - getting it all together

It will be my first time in Chicago, and should be an excellent conference, and a great opportunity to meet up with some very cool people - I'm really looking forward to it. See you there!

Changing ports with Grails

When you run a Grails application from the command line (using grails run-app, for example), it will run on port 8080 by default. This also applies when Grails runs web tests such as Selenium, Canoo webtest or HTMLUnit. If you have web tests in your application, Grails will automatically start up the application before running the tests, which is just what you want.

However, the port 8080 is not always what you want. You may need to run your Grails app on a different port if port 8080 has already been taken by another application. This is particularly the case on build servers, for example. The simplest solution is to set the server.port property on the command line, as shown here:

$ grails run-app -Dserver.port=8888

This is good as far as it goes, but has some limitations. Firstly, what if you always want to use the port 8888 for your Grails apps? Secondly, if you are using Hudson with the Grails plugin, this plugin doesn't currently let you provide command-line options like this. A work-around is to invoke grails from the command line in Hudson, as shown here:

$ grails -Dserver.port=8888 test-app

Note that Grails is finicky about the position of command-line parameters - they need to go after the "grails" command and before the target.

Another solution is to change the default port used by Grails itself. In Grails 1.1 at least, the trick is to modify the $GRAILS_HOME/scripts/_GrailsSettings.groovy file. In it, you will find a reference to the default port (8080). Just change it to whatever you require, and Grails will always run on this new port. The modified entry might look like this:

serverPort = getPropertyValue("server.port", 8888).toInteger()

Once this is done, Grails will always run on this port.

"Probably the best training course I've been on."..."Not just how to write Java code but the 'business end' - how to build, test, deploy, manage and monitor"..."One of the best and most useful courses I have attended. And they didn't even try to sell me anything!" - Coming soon to London! Get up to scratch with the latest in Java tools and best practices! Sign up on the 2009 season of the Java Power Tools Bootcamps.

Micro-iterative development practices

Iterative development is all about state transitions. In a nutshell, iterative development is about moving from one stable state to another in very small steps. This obviously applies to team iterations or sprints, but it also applies to daily development practices. Think of it as "micro-iterations" - it's the idea of "code a little, test a little". I would express it as "code a little (implement a small feature or fix a bug), test a little (does it work?), save your work (commit your changes)".

Now you need to be able to know when you are in a stable state. This is where a comprehensive set of unit and integration tests is vital. Techniques like TDD and BDD certainly help with design, but they also have the purpose of building up a hefty set of regression tests. I also usually run though the application and do some quick smoke tests to make sure everything looks OK. If you got bogged down in an unstable state for too long, ditch it and return to the previous stable state. I time-box my work - I set a maximum time to successfully implement a feature. If at the end of this time, it still doesn't work and I can't see the end of the tunnel, I start considering the ditching option.

Each step has a precise goal, a specific bit of functionality to implement. It might be "Add the Location domain object", "Add a set of region/location drop-down lists with AJAX updates" or "Add a captcha to the review form", but it is small, precise and clearly limited in scope. As soon as it works, I commit my changes.

That's one reason why I like git. I commit every time I get a new unit test (or set of related unit tests working), which can be every half-hour or more at times. Some commits might be experimental or provisory - I don't want to bother other developers with them just yet. I may only commit to the central repository a couple of times a day, when more significant milestones are reached.

I am actually very reluctant to leave my workstation with the code in an unstable state - it can take a lot of time to get back into it the following morning. On the other hand, sometimes you really do get bogged down, and it is unwise to burn yourself out on a particular piece of code. From this point of view, the XP approach of ditching unfinished code and starting anew really does have a lot going for it at times.

Anyway, just my 2¢.

"Probably the best training course I've been on."..."Not just how to write Java code but the 'business end' - how to build, test, deploy, manage and monitor"..."One of the best and most useful courses I have attended. And they didn't even try to sell me anything!" - Coming soon to London! Get up to scratch with the latest in Java tools and best practices! Sign up on the 2009 season of the Java Power Tools Bootcamps.

An introduction to easyb at Devoxx

Most people would agree that testing your code is a good thing. And there is little doubt that modern Test-Driven development practices, when applied well, can produce high quality software that is reliable, flexible and easy to maintain. However, one of the big problems that often happens when developers adopt Test-Driven Development is that they concentrate so much on the tests that they lose focus on what they are actually meant to be testing. This results in writing large numbers of superficial and inefficient tests, without testing the underlying user requirements in any real depth.

Behaviour-Driven development is one way to counter this. In Behaviour-Driven development, you no longer focus on writing tests. Instead, you concentrate on describing what it is you want your class to do before you start writing tests. In other words, you describe the behaviour of your class. This is very much in the original spirit of test-driven development, where the real aim is to design good software - writing tests is very much a means, not an objective in itself.

Easyb is a fairly new BDD framework based on Groovy. It makes it very easy to write your tests using a clear, structured and easy-to-understand BDD-style approach. These test scripts become a very clear way of describing your business requirements, in a way that the end-user can actually understand. In addition, using Groovy to write your unit tests has many benefits. In Groovy, you get all the advantages of a scripting language while still being able to access all your Java classes. But possibly most importantly of all, the highly expressive Groovy language features such as list manipulation and closures make it a real pleasure to write thorough and comprehensive business-level tests.

In November, I gave a talk on Easyb at the Devoxx conference in Belgium. This is now available online - enjoy!

"Best development course I have been on in a very long time...Greatly enjoyed the course...A 'must' course for serious Java developers..." - Read what people are saying about the Java Power Tools Bootcamps.

CI Adoption Customer Stories (3/8): Building in the dark

This case study is the third of an 8-part blog series about why so many developers adopt continuous integration, and originally published on the Atlassian blogs.

Ragnar is principle consultant at an Icelandic internet development firm based in Reykjavík. He and his team develop web applications for a variety of clients, primarily using PHP. Application source code is stored in a Subversion repository. Applications are typically deployed to three different servers: a local development server, a staging server that the client can use to preview the application during development, and finally the live server. Before adopting a Continuous Integration system, people struggled to keep the code repositories synchronized with each server.

The deployment process was largely manual - they had to manually commit to the repository and then manually deploy the code to the various servers at the appropriate times. This manual system was slow and error prone. It was easy to accidentally deploy code to the wrong server, or to deploy the wrong code. Indeed, before the CI system was installed, developers sometimes fell into the bad habit of deploying new code directly to the live server before testing it. This tended to happen most in situations where time was a critical factor and the client was putting the team under pressure to deliver. However, this was a false economy and very often, the code would have a few minor issues that normally the client would never have seen.

Making the change

Ragnar wanted a system to automatically deploy the right code to the right server at the right time on each commit to the Subversion repository. So Ragnar set up a CI server to monitor all of the various code repositories. He configured it to deploy any newly committed code to the development and staging servers automatically on each new commit. On each build, an Ant script uses SCP to securely transfer the code to the servers. But he didn't want untested code going live, so he kept deployment to the live servers as a manual task.

Clever Notification Strategy

Once this was set up, he needed a notification system for the developers to keep them up to date on all the new deployments for their various projects. All staff members use instant messaging (IM) extensively for internal communication and collaboration, so it seemed an obvious choice for their primary notification strategy. Ragnar set up a public IM account specifically for the CI server. This way, after every build, engineers are automatically notified of it immediately via their IM client. With one of their current projects, for example, they are collaborating with another software company on the development. The IM notifications keep them informed of any new builds that they have triggered, allowing them to talk back to the CI server and collaborate with the team. Without this notification system they would have to manually check the comments in the repository to see what they had changed. Now, it's all automatically sent to them via IM.

Ragnar improvised his own "information radiator", a concept frequently found in Agile circles. According to Agile guru Alistair Cockburn, an information radiator "is a display posted in a place where people can see it as they work or walk by. It shows readers information they care about without having to ask anyone a question. This means more communication with fewer interruptions." In many Agile projects, information radiators take the form of graphs, charts or diagrams, drawn up on very large sheets of paper and stuck on a prominent wall in the team's working area. For a Continuous Integration server, Lava Lamps are sometimes used for this purpose - the lamp bubbles green if the builds are successful, or red if there are build failures. However, you don't always need to go to such lengths to get an effective CI information radiator: a strategically placed LCD screen can be just as good. The bigger the better, obviously, but even an old 17 inch flat screen display is better than nothing.

Check out how one of the Atlassian teams does it..

"Probably the best training course I've been on."..."Not just how to write Java code but the 'business end' - how to build, test, deploy, manage and monitor"..."One of the best and most useful courses I have attended. And they didn't even try to sell me anything!" - Coming soon to London! Get up to scratch with the latest in Java tools and best practices! Sign up on the 2009 season of the Java Power Tools Bootcamps.

CI Adoption Customer Stories (2/8): Aligning work habits

This case study is the second of an 8-part blog series about why so many developers adopt continuous integration, and originally published on the Atlassian blogs. It naturally has a bit of a Bamboo slant to it (if you look close enough, you can even see the odd panda), but the issues discussed apply to any team trying to adopt modern development practices, no matter what CI tool is used.

Claudia and Hans work together on a development team in Frankfort writing internal web applications for a large multi-national. While they were working on the same module, their work habits varied dramatically. Getting aligned on how they do their work was critical from transforming from a mediocre team to a high octane development team.

Claudia adopted the habit of developing in small units of work, ensuring that they are fully working within their limited scope, and committing her changes. She began using a 'Test-Driven Development' development approach, so the code changes she makes are always accompanied by a set of new or updated test cases. Before committing her changes, she would integrate any changes that have been made by other developers since her last commit. Since she commits her changes several times a day, this integration work is usually limited to a small number of changes, and if there is a conflict, it is not difficult to figure out who made the change, and to discuss the conflict with them if required.

Hans, on the other hand, was working as a solitary developer, who preferred to work on his latest brilliantly-designed feature in isolation, for several days or weeks, until it was as complete as possible. When Hans finally integrated his changes, he would have to integrate all of his changes with weeks worth of updates from other team members - a sizeable task indeed. It was also an inefficient one, as any integration issues that were discovered were late on in the piece, and required major rework on the part of both Hans and the other developers. For example, over the last week, Claudia had made some significant changes to a core database access module that Hans happened to need for his new feature. Now Hans also updated this feature on his side, but in a totally incompatible way. Someone would have to rethink their solution, and since several other developers were already using Claudia's implementation, chances are it will be Hans.

Diagnosing the root cause

This is very much a problem of communication, rather than a technical one. If Hans had been aware of the changes that Claudia was making to the database access module, he could have taken this into account for his own changes. Since Claudia was committing regularly, all Hans really needed to do was to synchronize his code changes with the source code repository on a more regular basis. Once developers have learnt to write in small, well-tested units and commit their changes often, you need to ensure that everyone who needs to know about code changes is informed.

Making the switch to Continuous Integration

After implementing Bamboo, Hans began to get notifications every time Claudia commits some changes. Seeing the benefit, he quickly figured that it is in his interest to commit working changes as frequently as often and integrating his brilliant code with the rest of the team's was no longer a hassle. The whole team began to feel that they were coding more and refactoring less. The fast, relevant feedback on build failures helped each developer from getting blocked by a broken build. Once Hans began using Bamboo, he began to realize his build errors within minutes.

"Probably the best training course I've been on."..."Not just how to write Java code but the 'business end' - how to build, test, deploy, manage and monitor"..."One of the best and most useful courses I have attended. And they didn't even try to sell me anything!" - Coming soon to London! Get up to scratch with the latest in Java tools and best practices! Sign up on the 2009 season of the Java Power Tools Bootcamps.

CI Adoption stories part 1 - Johan's Outsourcing Blues

This case study is the first of an 8-part blog series about why so many developers adopt continuous integration, and originally published on the Atlassian blogs.

Johan is a project manager who works on internal corporate web applications for an IT shop in Frankfurt, Germany. He has a team of 10 developers, testers and BAs based in Germany, as well as an offshore team of similar size based in Bangalore, India. The project is not without hitches - co-ordinating work between teams in different parts of the world requires a much bigger overhead in time and effort than Johan's organization had initially thought. The offshore approach was imposed on the German development team from above, and is unpopular with many of the local developers who view it as a threat to their own jobs. Johan knows that this is not insurmountable, but that the two teams need to gel better before they can work together effectively.

The language barriers also make things difficult. Although all the team members can get by with written English, many are not confident with spoken English, making communication by telephone or teleconference difficult. The two teams work in different time zones, which doesn't make things any easier. In several cases, developers on one team have committed end-of-day changes that broke the build process. One time for example, Hans, a developer in the Frankfurt team, committed his latest changes at 5 PM Frankfurt time, but forgot to include a key new class. When the Indian team got to their office 8 hours later and checked out the updates, they were stuck for most of the day with a project that would not compile. This sort of incident does little to foster good will and harmony between the teams.

One Step Forward... Two steps back

This sort of thing is symptomatic of a broader communication problem - the developers on both teams are largely unaware of work done outside their own IDE until it is too late. There is little communication between team members on why particular changes are being made, and little opportunity to give feedback on these changes. This inevitably makes integrating the work done by each time more difficult, and can also lead to hard-to-find issues caused by misunderstood APIs or architectures.

Driving for team improvement

Adopting Continuous Integration

The move to Continuous Integration made a significant difference because developers would get back notified of the success of their build changes within 15 minutes of each commit. It has also increased awareness about what changes other developers are making to the code, and Johan has noticed an increased number of spontaneous IM discussions about coding issues between the team members. The overall effect has been to reduce the amount of time wasted due to cascading problems with broken builds.

"Probably the best training course I've been on."..."Not just how to write Java code but the 'business end' - how to build, test, deploy, manage and monitor"..."One of the best and most useful courses I have attended. And they didn't even try to sell me anything!" - Coming soon to London! Get up to scratch with the latest in Java tools and best practices! Sign up on the 2009 season of the Java Power Tools Bootcamps.

June 2009

Search this blog:

Categories

Archives

June 2009
May 2009
April 2009
March 2009
February 2009
January 2009
December 2008
November 2008
October 2008
September 2008
August 2008
July 2008
June 2008
May 2008
April 2008
March 2008
February 2008
January 2008
October 2007
July 2007
June 2007
May 2007
April 2007
March 2007
February 2007
November 2006

Recent Entries

Installing Sonar on a linux build server

Talking about Maven and Build Automation in London

The Agile2009 Tools Roundup

Articles

Grails and Continuous Integration: An Essential Combo
Grails is an excellent, highly productive development framework that positively encourages good development and testing practices. This article shows how to set up a Continuous Integration build job to compile and test your Grails application in Hudson, for automated continuous integration. Jun. 23, 2009

Integrating Maps into Your Java Web Application with Google Maps and Ajax
Want to provide maps in your web application? The Google Maps API is straightforward to call from Java, and with an Ajax-ian approach, you can make it extra user-friendly. John Ferguson Smart shows you how to combine these approaches. Oct. 26, 2006

Instant Messaging in Java Made Easy: The Smack API
Jabber is a popular and widely supported XML-based API for exchanging instant messages. You could compose the messages by hand, but there's an alternative. John Ferguson Smart introduces the Smack API, which makes it easy to use Jabber services from Java. Oct. 5, 2006

All articles by John Ferguson Smart »

This work is licensed under a Creative Commons License.