John Ferguson Smart's Blog

John is a freelance consultant specialising in Enterprise Java, Web Development, and Open Source technologies, currently based in Wellington, New Zealand. Well known in the Java community for his many published articles, John helps organisations to optimize their Java development processes and infrastructures and provides training and mentoring in open source technologies, SDLC tools, and agile development processes. John is principal consultant at Wakaleo Consulting, a company that provides consulting, training and mentoring services in Enterprise Java and Agile Development.

JavaOne 2008 - The new NetBeans Javascript editor

Personally, I hate Javascript with a passion. This is largely my fault - I just haven't taken the time to learn it properly. Unfortunately, there are still times when you just have to work with it. And the lack of syntax checking and code completion has always driven me crazy.

This morning, I saw a most impressive demo - the new Javascript editor in NetBeans. Now, when you work with Javascript in NetBeans, you get proper code completion, complete with lists of available methods and API documentation. The API documentation also includes information about browser compatibility for each function. Sweet!

Technically, all this is no mean feat. Since Javascript is typeless, code completion is inferred dynamically from the way you use the objects in the code. So if you create a class dynamically in Javascript, you get code completion for this as well! If you document your Javascript using Javadoc-style comments, you can also have the documentation appearing with the code completion

There is also real-time syntax checking, and also best-practice rules, such as confusing '=' and '=='. And you get a proper debugger, via a Firefox plugin, where you can step through the javascript, insert breakpoints, view variables, and so on.

A very nice piece of work. A longtime Eclipse user, I am really impressed by the progress NetBeans has made over the last few versions.

JavaOne 2008 - geeky gadgets galore!

This morning, James Gosling gave his traditional keynotes speech, with a series of impressive and entertaining demonstrations. What came across was the vast range of in which Java is being used. From online gaming to 3D graphics on mobile phones, from Smart Cards to particle accelerators at the CERN, from intelligent pens to robotic cars and power stations, Java is being used absolutely everywhere.

A few examples at random. A demonstration of TOMMY Junior, a robotic car designed to be able to drive 100km across a busy city all by itself. Call Of The Kings, a 3D online game written using the Project Dark Star Java Gaming libraries. An instrumentation gadget called the Sentilla motes, which are used to instrument anyting from cargo (to keep track of where they are), bridges (for stress), planes (to know when they need maintenance) and beer coasters (to know when the beer glass is empty!

But one of the coolest geeky gadgets of the show would have to be the Pulse SmartPen from LiveScribe. This is a pen (Java-powered, of course) that not only records what you write, but can also record sound as you write, and match up the sound with the writing later on. So, if your listening to a presentation, you take some notes and record what the speaker is saying. Later, you go back and tap somewhere in your notes, and the pen plays what the speaker was saying at this time! Very cool for presentations and meetings! You can also load the data onto your PC, and consult both the written notes and the sound, and even share your notes on a web server in Flash. You can even write your own Java applications to run on the pen. This is one seriously cool pen! And yes, I got myself one ;-).

On the subject of running application on other plaforms: on the plane coming over, the crew seemed to be having some trouble with the inflight entertainment system. For some reason, it kept freezing and they had to reboot it. As it rebooted, the crew made an announcement, asking people not to press the buttons too quickly, as this tended to make the system crash. As it rebooted, I couldn't help but noticing on the screens: Windows CE... ;-).

JavaOne 2008 - some photos

This being my first JavaOne, I wanted to share a few photos to give readers some general impressions.

JavaOne takes place in the Moscone center in downtown San Francisco.

Coming into JavaOne through the main hall...

The first thing to say is that there are lots of people. Hmmm, I'm not sure I made that clear: there are lots of people.

A typical session - this is Josh Bloch taking about Effective Java. This was a good talk, with some useful tips on how to use generics properly and some best practices for enums. Low level and very technical.(My phone may have Java on it, but it still takes crappy photos in a dim conference room):

I don't know if I mentioned it, but at JavaOne, there are lots of people. I take my hat off to the people who hold the little round signs with the session numbers on them, to tell you where to line up for each session. Honestly I don't know how they do it. They just yell things like "Session 104 - just go down the corridor and snake around like an S, then back this way..." - it might not sound much, but I think it must take a lot of visual-spacial skill to get it right with multiple lines of several hundred people criss-crossing in a confined space.

And the lines go on, and on, and on...

And on...

Oops, how did that photo slip in here ;-)? Seriously, this photo is actually the first time I've seen Java Power Tools on the shelves (I'm not counting Amazon, they were selling "New and Used copies" before I'd even got my copy!)

What doesn't come across in the photos is the networking side of things. I've met up with heaps of people that I've only previously known through email, and, despite all the high tech stuff, nothing really does beat face-to-face communication.

JavaOne 2008 - FindBugs is a great little tool, and it just keeps getting better!

There are a lot of static analysis tools out there, but Findbugs is unique. Where Checkstyle will raise 500 issues, and PMD 100, FindBugs will only raise 10 - but you damn well better look at them carefully!

That is a slight over-simplification, but it does reflect the philosophy of FindBugs. FingBugs uses more sophisticated analysis techniques than tools like PMD and Checkstyle, working at the bytecode level rather than with the source code, and is more focused on finding the most high priority and potentially dangerous issues.

At JavaOne this year, I've listened to both of William Pugh's excellent presentations on FindBugs. He ran through some interesting bugs that FindBugs found in some real-world code, and shared some useful tips on using FindBugs for very large real-world projects. For example, he suggested ignoring the low-priority issues (but not the medium-prioruty ones, which contain some very useful rules), and concentrating only on the issues that have appeared since the last stable release. Indeed, FindBugs has the ability to compare two sets of bugs and figure out which ones are new.

One interest area that he mentioned was that of annotating issues. When you review an issue in the FindBugs GUI tool, you can add an annotation such as "Mostly harmless" or "Should fix", along with a comment. This feature will soon also be available in the Eclipse plugin. This is cool. However, at the moment, it's hard to share this information, as it is stored in a local XML file. One interesting evolution that is apparently in the pipelines is to allow you to store these annotations in an external location, such as in a database. This would mean that developers could safely share annotations and comments on the same bug, which would make bug reviewing and correcting that much easier.

Another interesting evolution is in the area of reviewing and closing FindBugs issues. In PMD, for example, you can tell the analyser to ignore a particular issue using annotations or comments. This is a useful feature, more convenient than the current FindBugs approach which involves configuring XML configuration files. FindBugs works on bytecode, not on source code, so comments are obviously not an option. However, William indicated that using annotations to suppress Findbugs issues is definitely on the cards for a future release.

Note that FindBugs is not really a competitor with tools like PMD and Checkstyle - the tools are really at opposite ends of the static analysis tools spectrum. In many places where I've worked, there is a strong drive for imposing coding standards, and this is where a tool like Checkstyle excels. With a correctly configured Eclipse environment, 90% of formatting errors will often dissapear with an automatic reformat. FindBugs, on the other hand, is strongly focused on potential programming errors. The tools are really quite complementary.

And now for the shameless plug - Java Power Tools has a full chapter on FindBugs, as it is a very cool tool.

JavaOne 2008 - towards user interfaces that rock!

Enhanced user experience is a key theme coming out of this year's JavaOne. There are countless sessions on innovative technologies such as JavaFX, the Google Web Toolkit, Comet, and Android. All of these have in common that they are Java-based technologies that allow you to create enhanced user experiences without all the hard work. Well, with less hard work than if you had to code it all yourself :-).

JavaFX is a high-level scripting language that lets you create extremely visual browser-based user interfaces. The Google Web Toolkit (in which you build an AJAX-enabled Javascript web interface using Java classes that any old Swing developer will feel at home with. Comet is a technology that enables the server to update the user's web page automatically whenever required, with no intervention on the part of the user and little network overhead.

If you believe sun, this is clearly the way of the future. It's a shame that some government organisations still refuse themselves these technologies, on the basis that they (or at least, the AJAX-based ones) require Javascript, and, in theory at least, not all users have Javascript activated on their browsers. I guess they'll learn with time...

JavaOne 2008 - some brand new monitoring and profiling tools for your Java apps

Performance monitoring and profiling applications is one of my favorite pet topics - indeed, there's a big chapter on tools in this area in Java Power Tools. So I was keen to here what Jaroslav Bachirik and Gregg Sporar had to say in their talk today, entitled "Improving Application Performance with Monitoring and Profiling Tools".

It was worth it. Naturally, they talked about the JDK tools such as jps, jinfo, jhat, and JConsole, and the Solaris-specific tool DTrace. JHat got a special mention as being the only tool to date capable of diagnosing permheap issues.

The NetBeans profiling tools, which are very good, were also discussed. NetBeans has powerful and smoothly integrated profiling features built right in to the IDE, which makes it very pleasant tool to use for profiling and performance tuning.

But there's more. They also discussed BTrace, a new tool for dynamically tracing a running Java application. It uses a fairly AOP-style approach where you write a script to indicate which classes and methods you are interested in monitoring. The script is actually a simple annotated Java class. For example, in the following code (shamelessly stolen from the BTrace documentation ;-) ), you set up a call-back method (func()) that will be called whenever the Thread.start() method is called:

@BTrace
public class HelloWorld {
 
    // @OnMethod annotation tells where to probe.
    // In this example, we are interested in entry 
    // into the Thread.start() method. 
    @OnMethod(
        clazz="java.lang.Thread",
        method="start"
    )
    public static void func() {
        // println is defined in BTraceUtils
        // you can only call the static methods of BTraceUtils
        println("about to start a thread!");
    }
}

Then you simply run BTrace against your running Java process. Easy!

The other new and interesting tool they mentioned is VisualVM, a very cool new profiling tool being developped at Sun. Visual VM (just out as an RC) gives you a high level, graphical overview of profiling data and statistics, and letting you drill down for more details about memory consumption or thread use, or to display graphs indicating the methods that your application is spending the most time executing. You can't drill down to the line level the way you can in the NetBeans profiler, but, still, it's a pretty neat tool.

JavaOne 2008 - Hudson wins the 2008 Duke's Choice Award

I like the Dukes Choice Awards. They have to be the most unashamably geeky awards in the industry. Indeed, the Dukes Choice Awards recognize "cool tools, technologies, and products". This appeals to my inner geek enormously, and so I find it very cool.

Anyway, this year, the Dukes Choice Awards has chosen one of my personal favorites for the "Developer Solutions" category, the Hudson Continuous Integration tool. Hudson is a relatively young tool, but it has been progressing at a phenomenal rate over the last year or two. Installation and configuration is a breeze, and there are an impressive number of plugins for extra features, including some very cool plugins for reporting on test results, code coverage, and code metrics.

The only downsides in the current version is that security is a bit limited (but getting better), and you don't get the sort of fancy pre-emtive builds that TeamCity offers. But it's very easy to get up and running quickly. If you're choosing a CI server, you definately should take a look at Hudson.

JavaOne 2008 - Subversion 1.5 is coming!

This morning, I went to a talk by some of the CollabNet guys on the now-imminent release of Subversion 1.5. I've talked about some of the main new features, notably the new merge tracking capabilities, elsewhere, so I won't rehash them here. However, there were a few other interesting new features that are worth mentioning.

To have a truly robust enterprise Subversion archtecture, you need some kind of replication. Backup servers in case your main repository dies a violent death.

Another big reason to have some kind of replicated architecture is for performance. If some of your users are geographically a long way from the main repository, access times can be much slower than for a server that is physically closer.

In Subversion 1.5, if you are running Subversion through Apache, you can at least address the second issue very easily. By using a feature called "Write-through proxying", you can set up a master Apache repository in one location, and a series of Apache slave servers distributed across your sites. The slave servers are basically read-only copies of the master repository. Any read-only requests (such as "svn update") will stop at the nearest slave server. Read-write requests (such as "svn commit") are transparently passed on to the master server. The master server takes care of updating the slave servers automatically using the svn_sync tool.

Setting this up isn't too complicated. You need to configure the Apache slaves using a special directive to point to the master server, and also configure them to accept updates from (and only from) the master server. Then you set up replication from the master to the slaves using svn_sync.

Note that for this to work, you need to be running Subversion 1.5 with Apache 2.2 with mod_proxy - you can't do it with svnserve.

This is a good solution for improving performance, and a partial solution for server redundency. For example, if the master server fails for some reason, no one will be able to commit their changes, but updates still will be possible.

CommunityOne 2008 - Slides avaliable for "Open Source Tools for Optimizing Your Development Process"

Today, I gave a talk at CommunityOne on "Open Source Tools for Optimizing Your Development Process". The abstract for the talk is as follows:

The talk, naturally, discusses some of the topics covered in the Java Power Tools book. You can get the slides here. Because of the scope and the limited presentation time, the slides cover the material at a fairly high level. On the other hand, now that I've covered the basics, I'll be able to do some more detailed talks on specific areas. Stay tuned!

One person made an interesting comment, along the following lines: When you commit your broken code to the server, you've already broken the build and placed a broken revision in version control - why not prevent the broken revision from being committed in the first place? This is a good point, and in my opinion the next must-have feature for any CI tool that respects itself. Intercept the commit before it gets to your version control system, run a pre-emptive build on the merged code, and refuse to commit if anything goes wrong.

Nice idea. Probably quite tricky to implement in practice. Currently, as far as I know, TeamCity is the only tool that does anything like this, though I believe the Hudson developers are thinking about how to do this too. TeamCity basically acts as a proxy between you and your version control system, and tests the build before commiting your changes.

An ode to JavaOne

On a warm summers evenin', on a plane bound for nowhere,
I met up with the coder; we were both too tired to sleep
So we took turns a surfin' techie website on our laptops
til boredom overtook us, and he began to speak

He said, son, I've made a life out of Java Server Faces
Of Hibernate and POJOs, of webapps portalized
So if you don't mind my sayin', you've made a mess of those use cases
If you let me use your keyboard, Ill give you some advice

So I handed him my keyboard, and he asked about my project
Then he flicked through my classes, and asked for some insight
And the night got deathly quiet, and his face lost all expression
Said, if you're gonna cut the code, boy, ya gotta learn to cut it right

You got to know when to mock 'em, know when to code 'em
Know when to unit test, and know when to scrum
You never count your stories till you've done your iteration
There'll be time enough for counting when the sprint is done

Now every coder knows that the secret to survivin'
Is keepin' your code agile, and keepin' your code clean
Cause requierments are a changin', and those change requests are coming
And the best that you can hope for is to keep the users keen

So when he'd finished speakin', he passed me back the keyboard
Taught me to refactor, JUnit and TDD
Continuous Integration, testing annotations,
And how to measure progress by results that you can see.

You got to know when to mock em, know when to code em
Know when to unit test, and know when to scrum
Don't you go believing that requirements are frozen
Even the users, they won't know 'em till the sprint is done

You got to know when to mock em, know when to code em
Know when to unit test, and know when to scrum
Don't get caught up read Dilbert when your doing pair programming
There'll be time enough for Dilbert when the sprint is done

Vote for your favorite New Zealand IT Rockstar

To my surprise, I've been nominated in the top 10 finalists for the New Zealand IT Rockstar competition. Some come along and vote!

By the way, watch this space, there'll be a Selenium slideshow coming up for JavaWorld pretty soon, as well as some serious blogging for JavaOne next week. Keep tuned!

Java Power Tools is now available!

After a bit of a wait, Java Power Tools has been finally released! Java Power Tools delivers 30 open source tools designed to improve the development practices of Java developers in any size team or organization. Each chapter includes a series of short articles about one particular tool - whether it's for build systems, version control, or other aspects of the development process - giving you the equivalent of 30 short reference books in one package. Actually, there are more than 30 tools, as some chapters talk about more than one tool. But who's counting :-).

Amazon promise to have it in stock on the 3rd of May. The source code will be on the book website real soon. Thanks again to everyone who contributed articles or who helped out in many other ways!

Slides from my Software Quality NZ talk on " Java Software Quality - Tools and Techniques" are now online

Recently I did a talk for Software Quality NZ on "Java Software Quality - Tools and Techniques". The slides for this talk are now online: check them out here.

This talk is a fairly high-level coverage of a wide range of tools, from a QA perspective. Here's the summary:

The slides from my recent JUG talk about JUnit 4.4 are now online

Last week, I gave a talk to the Wellington Java Users Group on JUnit 4.4. Unit testing is one area where everyone can (and generally does) share their own experiences, and this was no exception, so the discussion after the talk was particularly interesting.

You can download the slides here

Java Power Tools now available on Safari

Java Power Tools is now available on Safari! You can also check out a very detailed Table of Contents on the O'Reilly site, including many extracts from the book. It's a pleasure to see it finally out, as it was a lot of work! Thanks to everyone who contributed!

The print version should be out in the next few days (the 22nd, I believe).

PMD optimisation rules put to the test - the AvoidEmptyStrings rule

PMD is an excellent static code analysis tool, with a rich set of rules regarding coding best practices and potential errors. The trick is working out which rules apply for your code.

Out of curiosity, I ran some benchmarks on the Optimization PMD rules, to see how they measure up to the latest JDKs. The results were, interesting...

Consider the AvoidEmptyStrings rule. PMD Rule: AvoidEmptyStrings "Finds empty string literals which are being added. This is an inefficient way to convert any type to a String."

Today, it would appear not... Look at the following code, which illustrates the said bad practice:

        for(int i=1; i < 10000000; i++) {
        	String s = "" + 123; 
        }

This takes around 7 ms to run on my laptop.

Now look at the following "improved" version:

        for(int i=1; i < 10000000; i++) {
        	String s = Integer.toString(123);		
	}

This took 425 ms to run, or approximately 60 times slower. Hmmm.

There may be other reasons to use Integer.toString(), but it would appear that performance is not one of them.

That said, this sort of optimization is often very much an intellectual exercise with today's JDKs. Objectively, half a second for 10 million operations is still probably plenty fast for most situations, as other, slower operations like database calls will form a bottleneck long before String operations will. The rule of thumb is, as always, benchmark before you optimize, measure before and after, and know why you are optimising.

Still, if you need to squeeze every last CPU cycle, this sort of thing is good to know ;-).

Update

Later on, I reran some tests which seem to confirm the role played by the constant value pointed out by some of the comments below. Using variable values, the results are more what you would expect:

    for (int i = 1; i < 10000000; i++){
        String s = Integer.toString(i);
    }

    for (int i = 1; i < 10000000; i++) {
        String s = Integer.toString(i); 
    }

So Integer.toString() is faster in situations using variables. When constant values are used, the compiler optimises things a little better when simple String concatanation is used.

There's still time to register for the Java Power Tools bootcamp

The Java Power Tools Bootcamp is coming up fast - May 12th for San Francisco, June 17th for London, August 11 for Auckland, and September-October for Melbourne, Sydney and Brisbane. The response so far has been enthusiastic, but it's still not too late to register.

This training course is based on the Java Power Tools book, which should hit the shelves in the next few days. It's packed full of very useful tips and tricks on all those techniques you know you should be applying at work, but just haven't managed to get around to yet. This is a great way to kick-start your Continuous Integration service or streamline your development process. Come along, it will be a lot of fun!

Nexus - my next Maven repository manager

The lads at Sonatype have just released a new Maven Repository Manager, called Nexus. According to the Nexus web site, this is a rewrite of the deceased Proximity repository manager. I was never too fussed with Proximity, but Nexus is very cool.

I am very impressed by Nexus. In fact, I liked it so much I'll be including it in the upcoming Java Power Tools bootcamps, and the next edition of Java Power Tools ;-).

For more details, Tim O'Brien has written an excellent introduction to Nexus here.

A good reference for Agile SCM

Henrik Kniberg, author of the very useful Scrum and XP from the Trenches, has written an interesting article on SCM in an Agile context in Version Control for Multiple Agile Teams. He describes, in practical terms, his implementation of the so-called "stable trunk" pattern for Agile development methodologies.

In this approach, your trunk contains a release-ready version of your code, and contains only stable, fully-tested features. Each development team has their own development branch, which they use to progressively implement application features (user stories). Whenever a feature (or story) is finished, tested and deemed production-worthy, it is committed to the main branch. So, at the end of the iteration (sprint), the release is whatever is in the main branch at that time.

This approach is well-suited to Agile environments, though it needs a little discipline to ensure that developers only commit code to the various branches in an orderly manner (Henrik refers to this as the "branch policy" - one branch might accept unit tested code, whereas the main branch will only accept fully tested production-ready features).

Using this strategy, your Continuous Integration setup will be a bit more complex, with a CI job (or several CI jobs) for each active branch.

This approach, incidentally, can also be applied to other more traditional projects, especially in a maintenance phase. Your trunk is your stable production version, and branches are used by small teams working on new features or bug fixes.

He also shows how this strategy can scale upwards, and work for multiple teams working on the same project.

Nice work, Henrik!

Cleaning up spurious SpringIDE warnings

I've been using SpringIDE for my Spring applications for a while now, and it is a very nice tool. It makes visualising your Spring configuration files, navigating through beans, and spotting configuration errors, much easier all round. Not to mention the nice Spring Webflow editor.

However, one thing has been irritating me for some time. Consider the following screenshot:

Eclipse (or, more precisely, SpringIDE) is complaining that it can't find the sessionFactory bean. Now, you can take my word for it, the sessionFactory bean does exist, it's just defined in another Spring configuration file. It is my habit to split my Spring bean configuration between several configuration files, which makes things more flexible and easier to understand.

But these warnings were bugging me. I'd declared all the Spring configuration files in the SpringIDE section of the project properties (see below), but Eclipse still didn't seem to be able to find the references to beans in other files.

As it turns out, the solution was right in front of me. You just have to go to the Config Sets tab and create a Config set containing all of your configuration files.

Once you've done this, everything works fine, and no more spurious warnings!

The TestEarly blog recommends the Java Power Tools bootcamp!

Java Software Quality - Tools and Techniques

I will be talking for Software Quality NZ in Wellington on the 17th of April, at 4pm at Equinox house. The topic of the talk is as follows:

Java Power Tools - out soon!

The much-awaited Java Power Tools will be out soon, really! It's now looking like the book will be on the shelfs in the second half of April. If you're in a hurry, you can pre-order you copy on the O'Reilly web site or at Amazon, where it has been doing pretty well so far.

I've been reviewing the final drafts, and I must say they look good! Some nice work by the O'Reilly production team!

The good news is, due to popular demand, we are extending the Early Bird discount for the San Francisco Java Power Tools Bootcamp until the end of April. This training course will cover all of the up-to-the-minute product info and lots of real-world tips and tricks. Come along and join the fun - it should be a great workshop. Places are limited, so book quickly!

On the subtle uses of Hamcrest tests

I came across an interesting issue with the Hamcrest asserts today. I have a method that returns a list of domain objects, as shown here:

    List<Stakeholder> stakeholders = stakeholderManager.findByName("Telecom");

This is a good example of how Hamcrest can make testing with collections easier. For example, to test the findByName() method, you should be able to something like this:

    @Test
    public void should_find_embedded_search_term_at_start() {
        List<Stakeholder> stakeholders  = stakeholderManager.findByName("Health");
        assertThat(stakeholders, hasItem(hasProperty("name",is("Health Associates"))));
    }

This Hamcrest assert checks that the list contains at least one Stakeholder object whose name is "Health Associates". This is clearly much nicer than having to iterate through the result list and test each Stakeholder object individually.

Unfortunatly, this doesn't work: you get a compilation error along the following lines:

The method assertThat(T, Matcher) in the type MatcherAssert is not applicable for the arguments (List<Stakeholder>, Matcher>)

Indeed, Hamcrest seems to be having trouble with the idea that a List of Stakeholders is also a List of Objects.

So can't we just cast the result (a List<Stakeholder> object) to a List<Object>? The short answer is no. For type-safety reasons that are well-documented elsewhere, you can't cast a List<Stakeholder> to a List<Object>. If you could do this, you could add objects that weren't Stakeholders to the list. However, you can find a work-around fairly easily. But this is where the details get rather interesting. For example, the following seems to compile in Eclipse, but nowhere else:

        List stakeholders  = stakeholderManager.findByName("Health");
        assertThat(stakeholders, hasItem(hasProperty("name",is("Health Associates"))));

StakeholderManagerTest.java:[62,8] cannot find symbol 
symbol  : method assertThat(java.util.List,org.hamcrest.Matcher>)

My friend Eduard Letifov pointed out the following solution, which will work both within Eclipse and using a normal JDK:

        List stakeholders  = stakeholderManager.findByName("Health");
        List<Object> listOfStakeholders  = stakeholderManager.findByName("Health");
        assertThat(listOfStakeholders, hasItem(hasProperty("name",is("Health Associates"))));

        List stakeholders  = stakeholderManager.findByName("Health");
        assertThat((List<Object>) stakeholders, hasItem(hasProperty("name",is("Health Associates"))));

That's right, there are two casts here: one from List<Stakeholder> to List, and a second from List to List<Object>. There's sure to be a more elegant way, but it probably involves modifying the signature of the Hamcrest methods. So this will do for now.

Until next time...

Registration is now open for the Java Power Tools bootcamps in Auckland and Australia

The Java Power Tools Bootcamp training sessions are a great way to get a grip on some very cool open source technology. Maven 2, Continuous Integration with that very cool tool Hudson, the latest in TDD and BDD testing techniques with JUnit 4.4, Selenium, DBUnit, Cobertura, and more, not to mention Subversion, Checkstyle, PMD, FindBugs, Mylyn and lots of other goodies. And see how to use all of these hip tools together in one smooth agile process. These really are techniques that can boost the productivity of your development team.

The course is based on the Java Power Tools book, and every student gets a complimentary copy.

You can still book places on the courses in Wellington, San Francisco and London, in April, May and June this year. And now, registration is open for the other Australia/New Zealand sessions, in Auckland (August 11-14), Melbourne (September 23-26), Sydney (September 29 - October 2) and sunny Brisbane (October 6-9).

This is going to be one great course, so don't miss out! You can view the schedule and book online here, or contact me directly. Come along! It's going to be a lot of fun!

Open Agility - Tools and techniques for productive Java development

I will be giving a lunchtime talk in Wellington on the 8th of April on how Java development best practices can boost your productivity. In this talk, I will present some practical techniques for effective development of Java applications. The presentation will cover a number of key Java development practices and how these can improve productivity. For each practice I discuss the problems the practice is trying to solve, issues with implementing the practice, and the tools and techniques available to support the practice. In the time available practices will be covered at a high-level and will include:

• Testing best practices
• Continuous integration
• Dependency management
• Code quality
• Integrated issue management
• Automatically-generated technical documentation.

See here for details.

Praise for Java Power Tools

People are saying lots of nice things about Java Power Tools already!

• "John’s upcoming book is a true masterpiece– I can’t recommend a better book for the Java community, baby!" - Andrew Glover, Stelligent
• "I can't wait to get my own copy!" - Alex Ruiz, FEST guru

Andrew has reviewed the draft version of Java Power Tools and was kind enough to write the foreword, and Alex contributed a chapter on FEST-Swing. Thanks, guys!

Learn how to code smarter at the Java Power Tools Bootcamp!

The Java Power Tools book is coming out real soon. In conjunction with this event, I will be giving some special training sessions called the Java Power Tools Bootcamps from May 2008 onwards. The first courses will be in Wellington, San Francisco and London, with other cities in New Zealand, Australia, the USA and Europe planned for later on in the year.

The Java Power Tools Bootcamp is an intense 4-day hands-on workshop covering some of the best open source tools for Java development on the market.

This one-of-a-kind course takes you on a in-depth guided tour of some of the best open source Java tools around, showing how you can use them individually and together to write code better, faster and smarter. We cover virtually all aspects of software development, from Build Scripts, SCM Best Practices, Unit and Integration Testing, code quality, and of course Continuous Integration. And we place a particular emphasis on how to make the tools work together.

Some of the principal topics covered include:

• Building a Java application using Maven 2 and Archiva
• Using Maven's dependency management in Ant
• Better Unit and Integration testing with JUnit 4.4, DBUnit and Selenium
• Automatically enforcing code quality with CheckStyle, PMD and FindBugs
• Automatically generating useful technical documentation with Maven 2 and UMLGraph
• SCM best practices with Subversion
• Automating your build process using Continuous Integration with Hudson
• Automated testing
• Automated code quality audits and metrics
• Automated reporting on project status
• Automated deployment

This is a hands-on course that will give you techniques that you can take away and immediately apply in your daily development work.

Places are strictly limited, so book quickly! You can view the schedule and book online here, or contact me directly. Come along! It should be a lot of fun!

Cool ways to use Hudson - voice control

Paul Duvall, from Stelligent, has been experimenting with using voice commands to control a build server. A neat idea!

The basic idea is to use Jott, which is a service that converts your voice messages into email messages. He is running fetchmail on Cygwin to download the mail messages, though, as he remarks, on a *nix build server, native mail tools would do the job with less mucking around. Then he uses an Ant script to read and parse the downloaded messages and kick off a build if an appropriate message is received. Finally, Hudson runs this process at regular intervals.

Hudson is good at monitoring external tasks, so I could see this working well.

Another idea would be to configure the Ant script to accept a command-line parameter indicating the text you are looking for (eg "Build QA"). The script would return 1 or 0 depending on whether it found the specified text in the latest messages. Of course, under Unix, you could do that easily with an ordinary shell script as well, but it would be less portable.

In Hudson, you can define "Post-build actions", things that need to be done once the build finishes. These actions can include running other build jobs. So, you could define one build job, as described above, to monitor your Jott mail messages, and to trigger another build job if any corresponding build orders are found. This other build job would run the actual build. This approach would let you decouple your real build process from the way it is launched.

It's an interesting idea, in any case. Now what I'm wondering is, when the build fails, does Hudson answer with "I'm sorry Dave, I'm afraid I can't do that."

Using Hudson environment variables to identify your buildsUsing Hudson environment variables to identify your builds

So your CI server now automatically deploys your application to an integration server. You've even configured it so that you can manually deploy to the QA server using the same process. Great! But wouldn't it be nice to know exactly what build you are looking at at any point in time? Well, Hudson lets you do just that.

When Hudson runs a build, it passes in a few useful environment variables, that you can use in your build script. This is a great way to inject information about the build into your deployable application. For example, Hudson build has a unique number, which you can reference in your build scripts using something like "${BUILD_NUMBER}". This is the list of variables (taken from an obscure corner of the Hudson documentation :-)):

• BUILD_NUMBER: The current build number, such as "153"
• BUILD_ID: The current build id, such as "2005-08-22_23-59-59" (YYYY-MM-DD_hh-mm-ss)
• JOB_NAME: Name of the project of this build, such as "foo"
• BUILD_TAG: String of "hudson-${JOBNAME}-${BUILD_NUMBER}". Convenient to put into a resource file, a jar file, etc for easier identification.
• EXECUTOR_NUMBER: The unique number that identifies the current executor (among executors of the same machine) that's carrying out this build. This is the number you see in the "build executor status", except that the number starts from 0, not 1.
• JAVA_HOME: If your job is configured to use a specific JDK, this variable is set to the JAVA_HOME of the specified JDK. When this variable is set, PATH is also updated to have $JAVA_HOME/bin.
• WORKSPACE: The absolute path of the workspace.
• HUDSON_URL: Full URL of Hudson, like http://server:port/hudson/
• SVN_REVISION: For Subversion-based projects, this variable contains the revision number of the module.
• CVS_BRANCH: For CVS-based projects, this variable contains the branch of the module. If CVS is configured to check out the trunk, this environment variable will not be set.

Sweet, you say. But how can you inject this into your deployable application? Easy. Just use these variables as you would any other properties, and let your imagination do the rest! In a Maven project, for example, you can use the maven-war-plugin plugin to inject data into the MANIFEST.MF file as follows:

<project>

    ...

    <build>

        ...

        <plugins>

            <plugin>

                <artifactId>maven-war-plugin</artifactId>

                <configuration>

                    <manifest>

                        <addDefaultImplementationEntries>true</addDefaultImplementationEntries>

                    </manifest>

                    <archive>

                        <manifestEntries>

                            <Specification-Title>${project.name}</Specification-Title>

                            <Specification-Version>${project.version}</Specification-Version>

                            <Implementation-Version>${BUILD_TAG}</Implementation-Version>

                        </manifestEntries>

                    </archive>

                </configuration>

            </plugin>

            ...

        </plugins>

    </build>

    ...

</project>

Now, the WAR file generated by Hudson will contain the build number:

Manifest-Version: 1.0
Archiver-Version: Plexus Archiver
Created-By: Apache Maven
Built-By: hudson
Build-Jdk: 1.5.0_14
Implementation-Version: hudson-mywebapp-integration-tests-60
Specification-Title: MyWebApp
Specification-Version: 1.0-SNAPSHOT

It is then a simple matter to extract this field from the MANIFEST.MF file of your deployed application at runtime. The java.util.jar.Manifest comes in handy here. I use this to display the build version discretly at the bottom of the screen so that testers can identify exactly what version they are dealing with.

I'll be giving a 4 day Java Power Tools Bootcamp in San Francisco, from May 12 to May 15 just after JavaOne. So, if your in SF at this time, come along!

Continuous Integration build strategies - stage your builds!

So you've got hundreds of tests, but they take ages to run. You have a Continuous Integration server, but it takes an hour to tell anyone when there's a failure. What can you do?

This is where staged builds can come in handy. I basically distinguish fast unit tests from slower integration tests. TestNG test groups are very cool for this, but you can also use simple naming conventions. For example, your unit tests run everything called *Test except classes called *IntegrationTest, whereas your integration tests only run *IntegrationTest.

Unit tests should be snappy, and give feedback within minutes.For integration tests, they should do the job they are intended to do, but time is not really of the essence.

Strictly speaking, unit tests are tests written in isolation, with no interaction with other components. I tend to be a bit more pragamtic here, so some interaction is allowed. However, you do need to be careful loading Spring contexts or Hibernate configurations, as this can be slow. If it slows down your fast unit tests, put it with the integration tests.

Heavy-weight tests, such as performance, integration, or GUI tests, will take longer to do. So these should always go in the integration-test category.

Actually, I often distingush several types of integration test. Here's how I often do it:

• First, a battery of lightning-fast unit and simple integration tests clears 90% of the terrain. Get in there quickly, test quickly, and get out fast. Give feedback on the whole code base within minutes. These tests should be done in a couple of minutes. For the developer, it's all about providing quick feedback. And a lot of errors actually do get caught at this level.
• Secondly, I run the slower tests, which are somewhere between fast unit tests and true integration tests. Things like testing Hibernate configuration files with an embedded database.
• Thirdly, I deploy to an integration server and run the database and GUI tests.
• Finally, I run the test coverage and code quality reports. This can take an hour, there's really no rush.

Each task is a separate build on the build server (say, Hudson). The first kicks off whenever the source code repository is updated. The others kick off once the fast unit tests have succeeded.

I'll be giving a 4 day Java Power Tools Bootcamp in San Francisco, from May 12 to May 15 just after JavaOne. So, if your in SF at this time, come along!

Don't miss CommunityOne and the Java Power Tools Bootcamp in San Francisco in May this year

For anyone who's interested, I'll be giving a session at CommunityOne in May entitled "Open source tools to optimize your development process". It should be fun! CommunityOne is a free event on the Monday before JavaOne start, all about open source project and tools.

While I'm in the groove, I'll also be giving a special public Java Power Tools Bootcamp training session the following week. The Java Power Tools Bootcamp is a course on open source Java development tools, based on on-site training that I've been giving to clients. Public registration is now open. So, if you want to get ahead of the queue, you can register here.

Java Power Tools podcast on JavaWorld

A little while back I had a ball of a time doing an interview with my good mate Andy Glover about the upcoming Java Power Tools book. It was a fun, off-the-hip, and largely improvised talk, and my thanks go once again to Andy for organizing it.

Anyway, the podcast has finally been published, so anyone interested can have a listen here.

A bird's-eye survey of the world of Continuous Integration Tools in 2008

The original CI tool (if you don't count ye old cron job) came in first with a wopping 35% for CruiseControl. Hudson and Continuum where neck-and-neck, with 14% for Hudson and 13% for Continuum. IntelliJ's TeamCity performed well for a commercial product, with a score of around 9%.

But times change, and technologies evolve. A year is a long time in the Java world. So lets take a look at the lay of the land today. And, while you're at it, why not vote on the new 2008 Continuous Integration Poll.

I haven't used CruiseControl for a wee while, so I can't vouch for it's latest features. In my experience, it's powerful, and flexible, but a right pain in the nether regions to configure and maintain. The reporting and web interfaces are pretty so-so as well. But, still, lots of people out there are using it.

Hudson, the new kid on the block, is constantly evolving, and adding feature after feature. The latest releases have added role-based security, plus an every-increasing list of very cool plugins. And the user interface is still as groovy as ever! Of all the open source CI tools, Hudson is what I generally use, given the choice.

Continuum 1.1 has arrived at last, with some cool new features like project groups and role-based security. Unfortunatly, though, the online documentation still leaves a lot to be desired.

On the commercial front, JetBrains is now offering a free "Professional Edition" of TeamCity, the Continuous Integration tool from the makers of IntelliJ. This edition is designed for small organisations with a limited number of user accounts, and no support for complex user authentication schemes such as project-based roles and LDAP support.

Indeed, there are also a number of commercial Continuous Build servers. There are plenty of good open source Continuous Build tools out there, so why might you choose a CI commercial tool over an open source one? Let's play the devil's advocate for a moment, and consider the options.

If your shop works with both Java and Microsoft technologies, you might appreciate TeamCity's support of both Java and .NET builds.

TeamCity also has a feature called a "personal build", which is basically a sandbox on the build server where code is automatically compiled and tested before being committed to version control. This is basically equivalent to the developer compiling and running the full set of unit tests before each commit, but with the advantage of being automated. Parabuild seems to proposes something simmilar with their "unbreakable builds".

Both TeamCity and Parabuild provide innovative features around distributed builds, though Hudson also provides some basic support for this as well.

Atlassian also do a nice Continuous Build server, called Bamboo, which, unsuprisingly, integrates very smoothly with JIRA.

Documentation might be another factor in your decision. Documentation for a commercial product is often of higher quality than the open source equivalent. Continuum is a flagrant example of this. The Hudson online documentation is better, though it still has a rather Wiki-ish feel to it. TeamCity documentation is excellent.

However, introducing Continuous Integration into an organisation is as much, if not more, about changing mindsets than it is about choosing a particular tool. And there is little risk of vendor lock-in, as it is fairly easy to replace one CI server with another. So, if your new to CI, you might just want to download an easy-to-use open source CI tool (Hudson comes to mind) and start out with that. Once you get familiar with what you can do with a CI server, you can always rethink your requirements later on.

To get a better idea of what tools people are using, I've published a new Continuous Integration Poll - come along and vote for your favorite tool!

Behavior Driven Development - putting testing into perspective

The ultimate aim of writing software is to produce a product that satisfies the end user and the project sponsor (sometimes they are the same, sometimes they are different). How can we make sure testing helps us obtain these goals in a cost-efficient manner?

To satisfy the end user (the person who ends up relying on your software to make his or her work easier), you need to provide the optimal feature set. The main challenge here is that the optimal feature set is not always what the users ask for, nor is it always what the BA comes up with at the start. So you need to keep on your toes, and be able to change direction quickly as the users discover what they really need. But that's the realm of Agile Development, and not really what I wanted to discuss here...

To satisfy the project sponsor (the person who has to fork out the cash), you need to satisfy your users, but you also need to write your application as efficiently as possible. Efficiency means writing code quickly, but it also means avoiding having to come back later on to fix silly mistakes. For example, I cn wriet ths tezt REASLDY QUIFKLY but if I don't keep an eye on the quality, the end user (you, the reader, in this case) will suffer. Your code needs to be reliable (not too many bugs) and maintainable (easy enought to understand so that the poor bastard who comes after you can work on the code with minimum hair loss).

So what has all this got to do with testing? Writing a test takes time and effort, so, ideally, you need to balance the cost of writing a test against the cost of _not_ writing_ the test. Does the test you are writing directly contribute to delivering a feature for the user? Will it lower long-term costs by making it more flexible and reliable? If your tests are to contribute positively to the global outcome of your project, you need to think about this, and design your tests so that they will provide the most benefit for the project as a whole.

It is fairly well-established that, in all but the most trivial of applications, unit testing will help to make your code more reliable. The cost of writing unit tests is the time it takes to write (and maintain) them. The cost of not writing them is the time it takes to fix the bugs that they miss.

Techniques such as Test-Driven Development (TDD) help to do this by incorporating testing as a first-class part of the design process. When you code using Test-Driven Development, you begin by writing unit tests to exercise your code, and then write the code to make the tests work. Writing the unit test helps (in fact, forces) you to think about the optimal design of your class _from the point of view of the rest of the application_. This is a subtle but significant shift in how you write your code.

However, when it comes to testing, developers are often at a loss as to what exactly should be tested. In addition, they tend to focus on the low-level mechanics of their unit tests, rather than

Behavior-Driven Development, or BDD, can provide some interesting strategies here. If you're not familiar with BDD, Andy Glover has written an excellent introduction here. Behavior-Driven Development takes Test-Driven Development (TDD) a step further. It is actually more a crystallization of good TDD-based practices, rather than a revolutionary new way of thought. Indeed, you may well be doing it already without realizing it. Using BDD, your tests help define how the system is supposed to behave as a whole. Using BDD, developers are encouraged to "What is the next most important thing the system doesn't yet do?" (see http://behaviour-driven.org/PowerfulQuestions).

This, in turn, leads to meaningful unit tests, with meaningful (albeit verbose) names, such as shouldReturnZeroForIncomeBelowMinimumThreshold. For example, suppose you need to write a tax calculator. In the country in question, no tax is payable for incomes under a certain threshold, the exact value of which is determined by the laws of the land, and which, today, happens to be $5000.

Using a basic TDD approach, you might simply test directly against the $5000 value, as shown here:

@Test
public class TaxCalulatorTest {
    @Test
    public void testTaxCalculation() {

        double taxDueOnLowIncome = taxCalculator.calculateTax(4999);
	assertThat(taxDueOnLowIncome, is(0.0));

        double taxDueOnHighIncome = taxCalculator.calculateTax(5000);
	assertThat(taxDueOnHighIncome, greaterThan(0.0));
    }
}

However, at a more abstract level, this gets us thinking - where does this value come from? From a configuration file? A database? A web service? In any case, this is the sort of thing that can change at the whim of a politician, so it's probably not a good idea to hard-code it. So we should add a property to our TaxCalculator class to handle this parameter.

While we're at it, we rename the test to better reflect what behaviour we are trying to model. So, instead of talking about "testTaxCalculation" (where the emphasis is on what we are testing), we would use a name like "shouldReturnZeroForIncomeBelowMinimumThreshold". The use of the word "should" is deliberate - we are describing how the class should behave. Note how our intentions suddenly becomes clearer. The tests might now look like this:

@Test
public class TaxCalulatorTest {
    @Test
    public void shouldReturnZeroForIncomeBelowMinimumThreshold() {

	taxCalculator.setMinimumThreshold(5000);
        double taxDueOnLowIncome = taxCalculator.calculateTax(4999);
	assertThat(taxDueOnLowIncome, is(0.0));

        double taxDueOnHighIncome = taxCalculator.calculateTax(5000);
	assertThat(taxDueOnHighIncome, greaterThan(0.0));
    }
}

Tests written this way are a much clearer way of expressing your intent than with tests with names like "testCalculation1", "testCalulation2" and so on. In addition to the usual JUnit and TestNG, there are also some frameworks such as JBehave which make BDD even more natural. And, with tests like this, tools like TestDox (http://agiledox.sourceforge.net/) can be used to extract documentation describing the _intent_ of the classes. For example, for the test class above, TestDox would generate something like the following:

TaxCalulator
- should return zero for income below minimum threshold
...

Java Power Tools: where it's at

It's been a while since I've given any updates on the status of the Java Power Tools book. So, here goes. The actual writing is done. Over the last couple of months, Java Power Tools has been proofread and typeset, getting it ready to go to print. Estimated release date is mid March. I've seen and reviewed the individual chapters in the final form - now I'm just waiting for the final cut.

The book is finally much bigger than I thought. Current estimates are around 880 pages - I don't know exactly, as I wrote the book in Docbook (I used XMLMind to do the actual writing). My original estimate was around 475 pages. So underestimating work is not just something that happens in the software industry! I basically managed to more-or-less stick to the original schedule through sheer stubbornness, hard work and lots of late nights.

I'm planning to give a talk in Wellington sometime in March - more details to come. I'm also busy preparing some training material based on some of the topics covered by the book, for some courses to be given later on in the year. Again, more details later.

Reflections on SCM Branching strategies

Traditionally, in both CVS and Subversion, if you want to merge some changes from a branch back into the trunk, you need to specify the changes you want to apply. As in "I want to merge the changes made between revision 157 to and revision 189 on branch B back into the trunk". In Subversion 1.5 (which isn't out yet), you just say "Merge the changes from branch B back into the trunk". Subversion will figure out what has already been merged, and only apply the new stuff. This is very cool, and makes it a lot less disuasive to consider merging between branches. I discussed this in some detail in http://www.javaworld.com/javaworld/jw-01-2008/jw-01-svnmerging.html.

Anyway, this got me thinking about branching strategies. From what I've seen, there are two common strategies.

In the first approach, your development work goes in the main trunk. Branches are for releases or possibly for isolated changks of work. You create a new branch whenever you release a version into a new environment (UAT, production, whatever). Bug fixes made in the release branches can be merged back into the development trunk as required. This strategy has the great merit of simplicity.

In the second approach, your trunk contains the production-ready code. Branches are for milestone releases. This means you can have separate teams working on different releases, and theoretically work more efficiently in parallel, but this sounds a bit complicated to manage to me.

There are also other approaches, though. One interesting one is to adopt a much more agile stragegy. Create a new branch for each new user story/feature/whatever. Then merge it back into the trunk when it's ready to be integrated. You would also need branches for production releases as well, I suppose. This strategy would let individuals or small groups work on specific features/user stories, and merge them into the main trunk when they are ready.

One interesting question for this approach is how would you set up Continuous Integration with this approach. For example, you might have separate CI jobs hooked into each user story branch, as well as another job running against the main branch.

May 2008

Search this blog:

Categories

Archives

May 2008
April 2008
March 2008
February 2008
January 2008
October 2007
July 2007
June 2007
May 2007
April 2007
March 2007
February 2007
November 2006

Recent Entries

JavaOne 2008 - The new NetBeans Javascript editor

JavaOne 2008 - geeky gadgets galore!

JavaOne 2008 - some photos

Articles

Integrating Maps into Your Java Web Application with Google Maps and Ajax
Want to provide maps in your web application? The Google Maps API is straightforward to call from Java, and with an Ajax-ian approach, you can make it extra user-friendly. John Ferguson Smart shows you how to combine these approaches. Oct. 26, 2006

Instant Messaging in Java Made Easy: The Smack API
Jabber is a popular and widely supported XML-based API for exchanging instant messages. You could compose the messages by hand, but there's an alternative. John Ferguson Smart introduces the Smack API, which makes it easy to use Jabber services from Java. Oct. 5, 2006

Web Services Made Easy with JAX-WS 2.0
Standards are so much easier to adhere to when your tools do it for you. Thanks to JAX-WS and its implementation in application servers like GlassFish, you can write web services as plain ol' Java objects, just by adding a few annotations. John Ferguson Smart shows how it's done. Jun. 13, 2006

All articles by John Ferguson Smart »

This work is licensed under a Creative Commons License.