So why is this interesting? Well, look at the trends in seek time and transfer rate. Seek time has grown at about 5% a year, whereas transfer rate at about 20% [1]. Seek time is growing more slowly than transfer rate - so it pays to use a model that operates at the transfer rate. Which is what MapReduce does. I first saw this observation in Doug Cutting's talk, with Eric Baldeschwieler, at OSCON last year, where he worked through the numbers for updating a 1 terabyte database using the two paradigms B-Tree (seek-limited) and Sort/Merge (transfer-limited). (See the slides and video for more detail.)

The general point was well summed up by Jim Gray in an interview in ACM Queue from 2003:

... programmers have to start thinking of the disk as a sequential device rather than a random access device.

But even the growth of transfer rate is dwarfed by another measure of disk drives - capacity, which is growing at about 50% a year. David DeWitt argues that since the effective transfer rate of drives is falling we need database systems that work with this trend - such as column-store databases and wider use of compression (since this effectively increases the transfer rate of a disk). Of existing databases he says:

Already we see transaction processing systems running on farms of mostly empty disk drives to obtain enough seeks/second to satisfy their transaction processing rates.

[1] See Trends in Disk Technology by Michael D. Dahlin for changes between 1987-1994. For the period since then these figures still hold - as it's relatively easy to check using manufacturer's data sheets, although with seek time it's harder to tell since the definitions seem to change from year to year and from manufacturer to manufacturer. Still, 5% is generous.

(Cross-posted at my other blog.)

The need for consistent hashing arose from limitations experienced while running collections of caching machines - web caches, for example. If you have a collection of n cache machines then a common way of load balancing across them is to put object o in cache machine number hash(o) mod n. This works well until you add or remove cache machines (for whatever reason), for then n changes and every object is hashed to a new location. This can be catastrophic since the originating content servers are swamped with requests from the cache machines. It's as if the cache suddenly disappeared. Which it has, in a sense. (This is why you should care - consistent hashing is needed to avoid swamping your servers!)

It would be nice if, when a cache machine was added, it took its fair share of objects from all the other cache machines. Equally, when a cache machine was removed, it would be nice if its objects were shared between the remaining machines. This is exactly what consistent hashing does - consistently maps objects to the same cache machine, as far as is possible, at least.

The basic idea behind the consistent hashing algorithm is to hash both objects and caches using the same hash function. The reason to do this is to map the cache to an interval, which will contain a number of object hashes. If the cache is removed then its interval is taken over by a cache with an adjacent interval. All the other caches remain unchanged.

Let's look at this in more detail. The hash function actually maps objects and caches to a number range. This should be familiar to every Java programmer - the hashCode method on Object returns an int, which lies in the range -2³¹ to 2³¹-1. Imagine mapping this range into a circle so the values wrap around. Here's a picture of the circle with a number of objects (1, 2, 3, 4) and caches (A, B, C) marked at the points that they hash to (based on a diagram from Web Caching with Consistent Hashing by David Karger et al):

To find which cache an object goes in, we move clockwise round the circle until we find a cache point. So in the diagram above, we see object 1 and 4 belong in cache A, object 2 belongs in cache B and object 3 belongs in cache C. Consider what happens if cache C is removed: object 3 now belongs in cache A, and all the other object mappings are unchanged. If then another cache D is added in the position marked it will take objects 3 and 4, leaving only object 1 belonging to A.

This works well, except the size of the intervals assigned to each cache is pretty hit and miss. Since it is essentially random it is possible to have a very non-uniform distribution of objects between caches. The solution to this problem is to introduce the idea of "virtual nodes", which are replicas of cache points in the circle. So whenever we add a cache we create a number of points in the circle for it.

You can see the effect of this in the following plot which I produced by simulating storing 10,000 objects in 10 caches using the code described below. On the x-axis is the number of replicas of cache points (with a logarithmic scale). When it is small, we see that the distribution of objects across caches is unbalanced, since the standard deviation as a percentage of the mean number of objects per cache (on the y-axis, also logarithmic) is high. As the number of replicas increases the distribution of objects becomes more balanced. This experiment shows that a figure of one or two hundred replicas achieves an acceptable balance (a standard deviation that is roughly between 5% and 10% of the mean).

Implementation

For completeness here is a simple implementation in Java. In order for consistent hashing to be effective it is important to have a hash function that mixes well. Most implementations of Object's hashCode do not mix well - for example, they typically produce a restricted number of small integer values - so we have a HashFunction interface to allow a custom hash function to be used. MD5 hashes are recommended here.

import java.util.Collection;
import java.util.SortedMap;
import java.util.TreeMap;

public class ConsistentHash<T> {

 private final HashFunction hashFunction;
 private final int numberOfReplicas;
 private final SortedMap<Integer, T> circle = new TreeMap<Integer, T>();

 public ConsistentHash(HashFunction hashFunction, int numberOfReplicas,
     Collection<T> nodes) {
   this.hashFunction = hashFunction;
   this.numberOfReplicas = numberOfReplicas;

   for (T node : nodes) {
     add(node);
   }
 }

 public void add(T node) {
   for (int i = 0; i < numberOfReplicas; i++) {
     circle.put(hashFunction.hash(node.toString() + i), node);
   }
 }

 public void remove(T node) {
   for (int i = 0; i < numberOfReplicas; i++) {
     circle.remove(hashFunction.hash(node.toString() + i));
   }
 }

 public T get(Object key) {
   if (circle.isEmpty()) {
     return null;
   }
   int hash = hashFunction.hash(key);
   if (!circle.containsKey(hash)) {
     SortedMap<Integer, T> tailMap = circle.tailMap(hash);
     hash = tailMap.isEmpty() ? circle.firstKey() : tailMap.firstKey();
   }
   return circle.get(hash);
 }

}

The circle is represented as a sorted map of integers, which represent the hash values, to caches (of type T here). When a ConsistentHash object is created each node is added to the circle map a number of times (controlled by numberOfReplicas). The location of each replica is chosen by hashing the node's name along with a numerical suffix, and the node is stored at each of these points in the map.

To find a node for an object (the get method), the hash value of the object is used to look in the map. Most of the time there will not be a node stored at this hash value (since the hash value space is typically much larger than the number of nodes, even with replicas), so the next node is found by looking for the first key in the tail map. If the tail map is empty then we wrap around the circle by getting the first key in the circle.

Usage

(Cross-posted at Problems worthy of attack.)

The nice thing is that you can fire up a fair sized Hadoop cluster (20 nodes is the current limit on EC2) in minutes and run it just for as long as you need to run your job - you pay by the hour. EC2 is still in limited beta and has had long waiting lists to get on it, but recently they cleared the backlog, so if you're interested in trying it out, now might be a good time.

Now, can we have one of these for the Java EE stack please?

Cat cat = mock(Cat.class);

and then set expectations on the returned cat object itself:

checking(new Expectations() {{
    one(cat).miaow();
}});

(This means that the miaow method is expected to be called exactly once on the cat object.)

This change means that you can refactor your method names without breaking your tests, unlike jMock 1 where the method names were strings:

Mock mockCat = mock(Cat.class);
mockCat.expects(once()).method("miaow")

You may have noticed that the new syntax is a little weird (check out the double braces, and how e.g. will modifiers need new statements), and it may take a little getting used to, but I think it's worth it for the IDE completion and refactoring support. For example, in test driven mode I can write the name of the method that doesn't yet exist then get my IDE to create the method for me.

The other nice thing is the integration with Hamcrest (a library of matchers that I've written about before). My toolkit for writing new unit tests now includes:

1. Hamcrest for making flexible assertions.
2. jMock 2 for defining expected behaviour.
3. Responsible Naming for understanding what the test does.

After the sixth or so occurrence in few years, I wrote a class to detect the problem. It's a simple implementation of SecurityManager that throws an exception when an attempt is made to connect to a site that is not on the list of approved hosts. The code appears below.

To use it we set the -Djava.security.manager command line argument (to the fully-qualified classname of RestrictedNetworkAccessSecurityManager) when running our test suites. Tests that access hosts that we aren't expecting will then fail with an error.

It's not just for applications that use XML catalogs. It's useful for almost any code, as a way to audit - and regression test - the network resources your application depends on. (Like a database you thought wasn't being used any more.) It's also a simple way to discover when third-party libraries connect to the internet unannounced.

The code is rough and ready - it's good enough for testing, but not really suitable for anything else, since it is totally permissive except for the checks on outbound connections. Feel free to use it, and suggest improvements or ways you've tackled the same problem in the comments section.

import java.net.SocketPermission;
import java.security.AccessControlException;
import java.security.Permission;
import java.util.Arrays;
import java.util.List;

public class RestrictedNetworkAccessSecurityManager extends SecurityManager {
  private static final List<String> ALLOWED_HOSTNAMES = Arrays.asList(new String[]{
    "localhost", "127.0.0.1",
  });

  @Override
  public void checkConnect(String host, int port, Object context) {
    checkConnect(host, port);
  }

  @Override
  public void checkConnect(String host, int port) {
    if (host == null) {
      throw new NullPointerException("host can't be null");
    }
    if (!host.startsWith("[") && host.indexOf(':') != -1) {
      host = "[" + host + "]";
    }
    if (ALLOWED_HOSTNAMES.contains(host)) {
      return;
    }
    String hostPort;
    if (port == -1) {
      hostPort = host;
    } else {
      hostPort = host + ":" + port;
    }
    String message = "Opening a socket connection to " + hostPort + " is restricted.";
    throw new AccessControlException(message, new SocketPermission(hostPort, "connect"));
  }

  @Override
  public void checkPermission(Permission perm) {
    // Allow all other actions
  }

  @Override
  public void checkPermission(Permission perm, Object context) {
    // Allow all other actions
  }
}

Hamcrest matchers (what were called constraints in jMock) are actually useful for more than just writing unit tests, but it is their application in writing assertions where they really shine and will probably see most use.

So now I can write

assertThat(a, equalTo(b))

or even

assertThat(a, is(equalTo(b)))

rather than JUnit's

assertEquals(b, a)

Apart from being more readable, assertThat takes any matcher as the second argument, so I can combine matchers or even write my own rather than creating an ever growing listof assertXxx methods. For example, I can say such things as

assertThat(collection, hasItem(anyOf(is(item1), is(item2))));

And since the matchers (and the assertThat method) are accessed using static imports, I can use them in any test framework I like.

So why not read the tutorial and give Hamcrest a go?

Here I'd like to give a flavour of LiFT and why you might want to use it.

When you set out to test your web application you need to think about the mechanics of how you interact with the application. Typically this involves fetching pages, finding elements on the page that you want to read (so you can make an assertion about them) and elements you want to change (so you can simulate user input). Often, the code required to handle these aspects obscures the intent of the test. Consider this HttpUnit test to do a Google search for an image of our ill-fated company summer party of two years ago:

import junit.framework.TestCase;

import com.meterware.httpunit.WebConversation;
import com.meterware.httpunit.WebForm;
import com.meterware.httpunit.WebImage;
import com.meterware.httpunit.WebResponse;

public class HttpUnitGoogleTest extends TestCase {

    public void testGoogleImageSearch() throws Exception {
        WebConversation conversation = new WebConversation();
        WebResponse page = conversation.getResponse("http://www.google.com/");
        assertEquals(page.getTitle(), "Google");
        page = page.getLinkWith("Images").click();
        WebForm form = page.getForms()[0];
        form.setParameter("q", "kizoom");
        page = form.submit();
        assertTrue(page.getText().contains("<b>Kizoom</b> summer party"));
        boolean foundImage = false;
        for (WebImage i : page.getImages()) {
            if (i.getSource().endsWith("summer04.jpg")) {
                foundImage = true;
                break;
            }
        }
        assertTrue(foundImage);
    }

}

Hmm, not too clear what it's testing. I'm not trying to single out HttpUnit here - it's a great tool for interacting with web applications, but it's pretty low-level. Of course, I could abstract some of the operations into helper methods to make the test clearer. For instance, the bit at the end where I'm looking for an image is ripe for extracting as a method. In fact, this is the approach we took for years - building libraries of helper functions that made using HttpUnit a bit less of an effort. However, it's not a structured approach. We can do better.

import com.kizoom.lift.NavigatingLiftTestCase;

import static com.kizoom.lift.constraints.Constraints.*;
import static com.kizoom.lift.matcher.Matchers.*;

public class GoogleTest extends NavigatingLiftTestCase {

    public void testGoogleImageSearch() throws Exception {
        goTo("http://www.google.com/");
        assertThat(page, has(title("Google")));
        clickOn(link("Images"));
        enter("kizoom", into(textField()));
        clickOn(button("Search Images"));
        assertThat(page, has(text("Kizoom summer party")));
        assertThat(it, has(image().withUrlThat(endsWith("summer04.jpg"))));
    }
}

This is much more concise. But more importantly it is much more readable. Each line can be read as a sentence, either as a jMock-style flexible assertion starting with "assert that", or as a command, such as "click on".

Page navigation is handled for us - we are provided with a page variable which always refers to the current page we are on.

Perhaps the most powerful idea is in the use of matchers. Matchers allow us to refer to bits of the page - so we can assert something about them or do something to them. For example, title("Google") creates a matcher that matches HTML title elements with text content "Google". We then use the has constraint to assert that there is (at least) one such match. Or take the next line: link("Images") matches a link with text "Images" which we use to navigate to the next page by means of the clickOn action.

Matchers allow for extensibility - it is easy to write your own if you need to - and help avoid the creation of ad hoc helper method libraries. Often you don't even need to go this far, you can refine an existing matcher using any jMock constraint. This is what is done in the last line. The withUrlThat refinement on the image matcher limits the matches to those that meet the given constraint. In this case we use the jMock endsWith constraint to match the image filename.

LiFT still has some rough edges - for example, its error messages are not always as clear as they could be - but it is definitely ready for testing real web applications. There's much more I could say, but I won't here. If you want to learn more try LiFT out by following the introduction, or read the slides from the talk Robert and I gave at the Google London Test Automation Conference in September, or even watch the video.

Dependency injection is pretty well established these days, with plenty of Inversion of Control containers available to manage your beans. I'm currently reading Java Concurrency in Practice by Brian Goetz et al, which got me thinking about the thread-safety of large object graphs managed by IoC containers.

In most applications I've seen, the common usage pattern is to use dependency injection to wire up your object graph in one go when the application starts. After that the application uses the object graph, and effectively treats it as being immutable. For example, in an application for buying books we might have a BookStore object that is given a BookFinder so it can find whether a particular book is available. The BookFinder is created at the beginning of the application and never changes. It is common to code this using setter injection:

public class BookFinder {

    // BookFinder's business methods...

}

public class BookStore {
    private BookFinder bookFinder;

    public void setBookFinder(BookFinder bookFinder) {
        this.bookFinder = bookFinder;
    }

    // BookStore's business methods which use bookFinder...

}

The problem is that it's not clear that this code is thread-safe. Unless the container publishes the BookStore bean safely then there is no guarantee that other threads will see the value of the BookFinder bean. If this seems weird then that's because it is. To quote Java Concurrency in Practice (p33): "In general, there is no guarantee that the reading thread will see a value written by another thread on a timely basis, or even at all."

Thankfully, this is easy to fix. We can mark the bookFinder instance as volatile to ensure its visibility to other threads:

public class BookStore {
    private volatile BookFinder bookFinder;

    public void setBookFinder(BookFinder bookFinder) {
        this.bookFinder = bookFinder;
    }

    // BookStore's business methods which use bookFinder...

}

Alternatively, we can use constructor injection and final fields. This works because the Java Memory Model makes special guarantees about the safety of final fields.

public class BookStore {
    private final BookFinder bookFinder;

    public BookStore(BookFinder bookFinder) {
        this.bookFinder = bookFinder;
    }

    // BookStore's business methods which use bookFinder...

}

The Future

So could the IoC container providers take steps to ensure that application developers don't have to worry about thread safety in wiring up object graphs? Possibly, although at the time of writing it is less than clear whether the popular containers do or not. Tim Peierls, one of the authors of Java Concurrency in Practice, wrote in a email to me that "Until it's proven safe, I have adopted a rule that all such setter-injected effectively immutable fields must be volatile." He also suggested marking the field with an annotation, such as @WriteOnce, so that when your container's behaviour is clarified it is easy to go through code and remove the volatile modifier.

I would love an API that exposes Google's MapReduce, a simple programming model for crunching on large datasets. You can write and run MapReduce programs today, using Hadoop, but it's only really useful if you have enough machines at your disposal. The pay-as-you-go model of S3 (and Sun Grid) would be very attractive to developers who want to run ad hoc computations, or can't afford the upfront investment in hardware.

Well, now it's possible with the beta launch of Amazon EC2 (picked up from the O'Reilly Radar). EC2 (apart from coincidentally being the postcode of my company's new London office) stands for Elastic Compute Cloud and allows you to commission compute resources on an on-demand basis using simple web-service-based tools. The unit of compute capacity is an Amazon Machine Image (AMI) - a Linux image - which you can configure to have any software you like on it. You can run any number of instances, paying for the number of instance hours you use (and the data you transfer).

This goes beyond what I wished for in March as it allows you to run anything on the image! Going back to Hadoop and MapReduce, I can imagine a generic Hadoop AMI that you configure your job on, before commissioning a number of EC2 server instances to run it. Press go, wait for your job to complete and then decommission the server instances.

Definitely one to watch.

It struck me that you could treat S3 as a big hashtable, so I tried writing an implementation of java.util.Map (built on the Amazon S3 Library for REST in Java) that uses S3 as its backing store. Here's some code to put some arbitrary objects into it:

Map<String, Fruit> map = new S3Map<Fruit>(bucket, awsAccessKeyId, awsSecretAccessKey);
map.put("breakfast", new Fruit("banana"));
map.put("lunch", new Fruit("apple"));

for (Map.Entry<String, Fruit> entry : map.entrySet()) {
  System.out.println(entry.getKey() + ": " + entry.getValue().getName());
}

This code prints:

breakfast: banana
lunch: apple

Notice that keys are strings, but the objects in the map can be of any type, as long as they are Serializable (as Fruit is), although even this restriction can be dropped by providing a custom serialization strategy when constructing the map. (Something like XStream would be a good choice here.)

Also, the map needs the connection parameters for an S3 account, and a bucket name. One bucket corresponds to one map. The map is persistent, so I can run the code above again, but without putting anything into the map, and the values put in previously will be retrieved and printed out exactly as before.

Map<String, Fruit> map = new S3Map<Fruit>(bucket, awsAccessKeyId, awsSecretAccessKey);

for (Map.Entry<String, Fruit> entry : map.entrySet()) {
  System.out.println(entry.getKey() + ": " + entry.getValue().getName());
}

Writing an implementation of java.util.Map is interesting. It's actually straightforward to do, since you start by subclassing java.util.AbstractMap and implementing a handful of methods. There's a line in the javadoc which says "Each of these methods may be overridden if the map being implemented admits a more efficient implementation." Normally we strive for an implementation that is efficient in terms of speed, however in this case I was very aware of the monetary cost of each method. After all, I'm paying $0.20 per GB to get my data in and out of S3. There's a table in the javadoc that sets out the number of S3 operations performed by each method so you can get a feel for their cost. As it turned out, I ended up overriding almost all of AbstractMap's methods.

So, what is it useful for? Well, I haven't actually had a use for it yet, but S3Map is really a non-transactional persistent datastore, so it could be used in many scenarios, from applets (if the applet is hosted on S3 too), to desktop apps (to share user data between machines?), to server-side apps. If you're intrigued, you can try it out yourself by signing up for an Amazon S3 account, and downloading S3Map (it's hosted on S3 of course).

Singulars and plurals are so different, bless my soul.
Has it ever occurred to you that the plural of "half" is "whole"?

Allan Sherman, One Hippopotami

It's well known that Perl has been heavily influenced by ideas from natural language, which probably explains why it has so many libraries for handling natural language text. Java is less well served, but as our programming tools become more powerful, it is inevitable that they will provide more linguistic support. It would be handy for example if your IDE told you in a not-too-obtrusive way that you had mispelled an identifier.

One linguistic feature that is already in today's tools is pluralization - forming plurals of words. The most famous example is Ruby on Rails, which automatically creates names in the plural for collections of a singular entity. The JAXB 2.0 Reference Implementation also uses pluralization in generating bindings of repeatable XML elements by pluralizing the corresponding Java property name.

Tool builders can now easily add pluralization to their applications using Inflector, a new Java library hosted on java.net. Using it can be as simple as calling:

import static org.jvnet.inflector.Noun.pluralOf;
...
pluralOf("hippopotamus");

The library is also useful for producing natural language messages. For instance, the following code will print the message I bought 10 loaves.

int n = 10;
System.out.printf("I bought %d %s.", n, pluralOf("loaf", n));

(See the documentation for more examples.)

In principle this would be easy to internationalize as Inflector does have multi-language support. However at present Inflector's implementations of pluralization algorithms for different languages are a little thin on the ground. English is practically complete (thanks to Damian Conway's excellent contribution to the subject). There is an implementation for Italian too, but it is incomplete. So, if you have (grammatical) expertise in a language for which there is no pluralization algorithm, and you would like to get involved, then consider joining the project.

This got me thinking: is it possible to use anaphora more widely at the language level? Would such constructs be useful? Before trying to do this in Java I looked at more dynamic languages, starting with a very quick look at Lisp, where I first came across anaphora in programming languages.

Paul Graham, in his wonderful book ANSI Common Lisp (which is well worth a read, by the way, even if you never intend to write a line of Lisp), introduces an anaphoric if that captures the test condition in a variable called it. His example is:

(aif (calculate-something)
     (1+ it)
     0)

This can be read as "if the variable calculate-something is non-zero, then return one plus its value, otherwise return zero". The aif construct is actually a macro, a very powerful feature that effectively allows you to re-write the language. I won't go into the details here as it's very clearly explained in the book. In a nutshell, the Lisp compiler transforms the expression into another chunk of code (defined by the macro) that it can understand - in this case a regular if with a bit of variable capture.

Let's try another language - Ruby.

Ruby

Here's how aif would be used in Ruby:

hash = { 'a' => 'peach', 'b' => 'pear', 'c' => 'plum'}
aif hash['a'] do
  puts @it
end

This code snippet prints peach to the console

Ruby doesn't have macros, so the way this is implemented is not like Lisp, it's actually closer to the jMock approach. We basically extend Object, and add an aif method that takes a conditional expression and a block. If the expression is true, then the instance variable @it (instance variables always start with @ in Ruby) is set to the value of the expression and the block is executed. Here's the definition:

class Object
  def aif(expression)
    if expression
      @it = expression
      yield
    end
  end
end

The Ruby implementation is inferior to the Lisp one in a few ways (although some of this could be down to my inexperience in Ruby - I invite seasoned Rubyists to improve the code!). Firstly, there is no way that I can see of supporting an else clause for aif (the Lisp version does this easily). Secondly, there are concurrency issues - storing state in an instance variable is dangerous if multiple threads are using the object. However, this is fairly straightforward to solve with synchronization or by using thread-local variables. Thirdly, there is a minor, but potentially irritating, syntactic difference between aif and if: aif takes a block and hence has an extra do. Compare:

if hash['a']
  puts hash['a']
end

and

aif hash['a'] do
  puts @it
end

Java

Can you do the same thing in Java? Let's try to replicate the Ruby approach since Java doesn't have Lisp macros. We can't add methods to Object in the way we can in Ruby, so instead create a class called AnaphoricObject with a static method called aif. We can then either extend AnaphoricObject or statically import aif when we want to use anaphoric if.

Now we hit a problem: what does the definition of aif look like? It would take an Object as the test expression, but then we would need to cast the it variable to the type we wanted. Doable, but not very pleasant. (Aren't we trying to improve readability?) Worse, how do we supply a block of code? We can't do it. There is probably a way to do it using anonymous inner classes, but I don't want to go down that alley - we'll lose all improvements in syntax which is why we were trying to do this in the first place.

Conclusion

Obviously, I like Java a lot, but I know its limits. Anaphora work well for an API like jMock, but not at the Java language level. Even Ruby, touted for its ease of meta-programming, struggles to provide a nice implementation of anaphoric if (although, again, I'd be happy to be proved wrong on this). Lisp manages it, if only because it has macro support. (It's probably possible in your favourite language too.)

But, do we really need anaphoric if - after all, you can probably argue that you've never needed it. This is actually a weak argument. For example, I use Java 5 static imports all the time now and wouldn't like to give them up, but I didn't rage about not having them before I was introduced to them. I didn't know what I was missing. Similarly, once you've used constructs like anaphoric if, you get to know when it is useful, then start missing it in languages where it's not available. Here's Paul Graham again (from ANSI Common Lisp):

Is it worth writing a macro just to save typing? Very much so. Saving typing is what programming languages are all about; the purpose of the compiler is to save you from typing your programs in machine language. And macros allow you to bring to your specific applications the same kinds of advantages that high-level languages bring to programming in general. By the careful use of macros, you may be able to make your programs significantly shorter than they would be otherwise, and proportionately easier to read, write, and maintain.

Although he's talking about macros, I think the point is more general: it's worth striving for higher-level abstractions to make our programs more literate.

But ambiguity and programming languages don't go very well together - so why would anyone want to mix the two? Actually, there are circumstances in programming languages where there is no real ambiguity, and an anaphor can have a use in eliminating repetition. Think of it as applying the DRY (Don't Repeat Yourself) principle at the syntax level.

We introduced a limited form of anaphora in the functional test framework I talked about in my previous blog entry, Literate Programming with jMock. Consider some more sample code:

public void testMainMenu() throws Exception {

    goToTheIndexPage();
    assertThat(theCurrentPage, has(title("Home Menu")));
    assertThat(it, has(menuOptions().named("Departures and Arrivals",
                                           "Travel News")));
}

The variable it refers to the object that was the target of the assertThat call on the previous line; in this case the current page referred to by the theCurrentPage variable. We improve readability and remove multiple references to theCurrentPage, without compromising ambiguity (and therefore without the risk of introducing bugs).

So where does the the variable it come from? We created a subclass of org.jmock.MockObjectTestCase, and added a protected instance variable called it of type Object, then overrode the assertThat method to stash away the target object:

@Override
public void assertThat(Object target, Constraint constraint) {
    it = target;
    super.assertThat(target, constraint);
}

That's it.

Temporary Local Variables

The biggest gain is the case where you can avoid having to create a temporary local variable. You often want to assert several things about an object, so you end up creating a local variable with a short name to refer to the target object:

public void test() {
    Fruit fruit = lunchbox.getFruit();
    assertThat(fruit, isNotNull());
    assertThat(fruit, hasProperty("colour", eq("yellow")));
    assertThat(fruit, isIn(lunchbox.getItems()));
}

Alternatively, you can use the and constraint to make a single assertion, but this is less readable and is hard to format well:

public void test() {
    assertThat(lunchbox.getFruit(),
        and(isNotNull(),
            hasProperty("colour", eq("yellow")),
            isIn(lunchbox.getItems())));
}

Using it produces the most readable test:

public void test() {
    assertThat(lunchbox.getFruit(), isNotNull());
    assertThat(it, hasProperty("colour", eq("yellow")));
    assertThat(it, isIn(lunchbox.getItems()));
}

Our experience has been that the little bit of syntactic sugar that it provides is convenient, and not at all ambiguous.

assertThat(a, eq("3"));

is more readable and understandable than

assertEquals("3", a);

You can read it as "assert that a eq(uals) 3". (The only reason it is not equals is that it conflicts with Object's equals method.) This form makes the code more literate, that is, easier for humans to read, which is the main goal of Donald Knuth's idea of literate programming.

The assertThat form is very extensible too - you just write new constraints. We've grown a small library of constraints - a sort of DSL for unit testing. The library contains constraints ranging from the simple conveniences isNull and isNotNull, to more complex collections constraints such as includes which takes a variable number of objects (varargs) that the collection must contain:

assertThat(list, includes("peach", "pear", "plum"));

They are all easy to write.

Consequences

One thing we missed from JUnit assertions was the ability to control the failure message. This isn't often needed as the describeTo method of the constraint does a good job in most cases. There are occasions however when you need more context, such as when you have several boolean assertions in a single test and you want to distinguish them. For example, the following test

public void test() {
    boolean a = true;
    boolean b = false;
    assertThat(a, eq(true));
    assertThat(b, eq(true));
}

fails with a message that doesn't tell you if it is a or b that is false:

junit.framework.AssertionFailedError:
Expected: eq()
    got : false

We extended jMock's base test class org.jmock.MockObjectTestCase to take an overloaded version of assertThat that takes a new parameter called a Consequence. The test uses the otherwise method that creates a Consequence instance with the given failure message,

public void test() {
    boolean a = true;
    boolean b = false;
    assertThat(a, eq(true), otherwise("a should be true"));
    assertThat(b, eq(true), otherwise("b should be true"));
}

and fails more helpfully:

junit.framework.AssertionFailedError: b should be true

Literate Functional Tests

My colleague Robert Chatley, who worked on the jMock extensions described above, has recently taken the idea of literate testing further and applied it to functional (acceptance) tests. He has built a framework for testing a markup language for digital TV services, where as well as the assertions, the steps for driving the test are literate. Here is a sample:

public void testArrivalsWithUnambiguousOriginAndUnambiguousDestination() throws Exception {

    goToTheTrainArrivalsSearchScreen();

    enter(LONDON_PADDINGTON, into(textBox().labelled("Arriving at")));
    enter("Bristol Temple Meads", into(textBox().labelled("Calling at")));
    enter("1554", into(textBox().labelled("Arriving after")));
    select(option("Today"), from(selectBox().labelled("When")));

    clickOn(button("Select"));

    assertThat(theCurrentPage, has(title("Train arrivals after 15:54 today")));
}

Notice that you can't tell that it is testing a digital TV service. Those details are hidden behind the small collection of methods that allow you to perform actions: goTo() (called by goToTheTrainArrivalsSearchScreen()), enter(), select(), clickOn(); or extract data from the page: textBox().labelled("Arriving at"), button("Select").

Taking a tip from Nat Pryce, we can apply a neat trick and change Eclipse's settings so that punctuation is white, to make the code even more readable (but less editable)! (Steve Yegge had the same idea for Scheme.) This is what you get:

public void testArrivalsWithUnambiguousOriginAndUnambiguousDestination() throws Exception {

    goToTheTrainArrivalsSearchScreen();

    enter(LONDON_PADDINGTON, into(textBox().labelled("Arriving at")));
    enter("Bristol Temple Meads", into(textBox().labelled("Calling at")));
    enter("1554", into(textBox().labelled("Arriving after")));
    select(option("Today"), from(selectBox().labelled("When")));

    clickOn(button("Select"));

    assertThat(theCurrentPage, has(title("Train arrivals after 15:54 today")));
}

Truly literate.