Marc Hadley's Blog

Web Services and XML Archives

JAX-RS Public Review Draft and JavaOne

The JAX-RS public review draft is now available for, er, review - download it here.

If you are attending JavaOne we'd love to see you at any or all of the following sessions:

• TS-5425, Tuesday May 06 12:10 - 13:10, Esplanade 307-310
• BOF-5613, Tuesday May 06 20:30 - 21:20, Esplanade 307-310
• TS-5425 (repeat), Friday May 09 14:50 - 15:50, Esplanade 304/306

I'll also be on Glen's panel "REST Versus SOA: Can We All Just Get Along?" which follows our BOF in the same room at 21:30 (Tuesday is shaping up to be a busy day).

JAX-RS Implementations

The Restlet team just announced a new release. Amongst the new features is support for JAX-RS, see an example here. With the RI (Jersey), Apache CXF, and JBoss RESTeasy, that makes a total of four implementations currently underway. The feedback we're receiving as a result of these parallel implementations is proving very useful and I'm grateful for the all the good input we've been getting.

The graph below shows the monthly totals for emails sent to the dev, users, issues and commits mailing lists for both JAX-RS and Jersey.

As you can see, traffic on the lists has picked up quite a bit over the last few months and a good proportion of the increase is due to the additional implementation work flushing out issues. We've also been working through our issues list with a view to publishing a JCP public review draft in the not too distant future.

Authentication in Jersey

I'm working on an internal project building some RESTful services using a combination of Jersey, JPA, Glassfish and Derby. Actions on some resources require authentication and I need access to the name of the authenticated user in the resource method. This entry describes the steps to set this up - kudos to my colleague Hubert for working out much of the below.

Obviously you're going to need all the ingredients listed above. I'm using NetBeans which came bundled with Glassfish and Derby so all I had to do was install the RESTful Web Services plug-in, YMMV.

Next you need a database to store username, password and group information. Here's the table definitions I used:

CREATE TABLE users (
    id INTEGER GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
    username VARCHAR(64) UNIQUE NOT NULL,
    password VARCHAR(64) NOT NULL,
);
CREATE INDEX username ON users(username);

CREATE TABLE groups (
    id INTEGER GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
    username VARCHAR(64) NOT NULL REFERENCES users(username) ON DELETE CASCADE,
    groupname VARCHAR(64)
);

Depending on your needs you could lose the integer fields, the important thing is to have a column in the group table with the same name as the column in the users table that holds the username. Those two columns are used to join the users table to the groups table. The Glassfish security realm is going to execute a query like this to retrieve the groups (which map to security roles) that a particular user is a member of:

SELECT groupname FROM groups g, users u where g.username = u.username and u.username = ?

Add some users and groups for testing purposes, note that the password should be an md5 hash, not plaintext - there are several web sites that offer online MD5 generators that you can use. I used groups named USERS and ADMINISTRATORS, you'll see where these come in later.

Next you need to define a JDBC data source, connection pool and security realm in Glassfish. The first two are pretty straightforward and will likely be taken care of already if you have deployed a web application using the database. Setting up the security realm is also straightforward once you have a suitable database structure as described above. You do this via the admin console as shown below (there are a couple of additional fields on the form, you can leave those blank):

Its now possible to require authentication for access to a JAX-RS resource via web.xml. Say I have a resource that only authenticated users can use, e.g.:

@Path("dropbox")
public class DropBox {
    @Context
    SecurityContext security;
    
    @POST
    public Response drop(InputStream data) {
        String username = security.getUserPrincipal().getName();
        ...
    }
}

Note the use of dependency injection to get an instance of SecurityContext and how that interface provides access to the username. In order for this to work we have to tell the container that authentication is required for access to the /dropbox URI. Add the following lines to your web.xml to restrict use of this resource to members of the USERS group:

<security-constraint>
    <display-name>DropBox</display-name>
    <web-resource-collection>
        <web-resource-name>DropBox</web-resource-name>
        <description></description>
        <url-pattern>/dropbox</url-pattern>
        <http-method>GET</http-method>
        <http-method>POST</http-method>
        <http-method>HEAD</http-method>
        <http-method>PUT</http-method>
        <http-method>OPTIONS</http-method>
        <http-method>TRACE</http-method>
        <http-method>DELETE</http-method>
    </web-resource-collection>
    <auth-constraint>
        <description>Have to be a USER</description>
        <role-name>USERS</role-name>
    </auth-constraint>
</security-constraint>
<login-config>
    <auth-method>BASIC</auth-method>
    <realm-name>userauthn</realm-name>
</login-config>
<security-role>
    <description/>
    <role-name>USERS</role-name>
</security-role>

That about it, after you redeploy the application, the next time you try to access /dropbox you'll get a 401 Unauthorized response unless you include a suitable Authorization header in the request.

Integrating Jersey and Abdera

I'm working on an internal project that involves adding Atom Publishing Protocol support to a data store. Naturally, I'm using Jersey for the HTTP side of things and decided to give Apache Abdera a try for simplifying working with feeds and entries.

With JAX-RS I can write a feed resource pretty easily:

import java.net.URI;
import java.util.Date;
import javax.ws.rs.ConsumeMime;
import javax.ws.rs.GET;
import javax.ws.rs.POST;
import javax.ws.rs.Path;
import javax.ws.rs.ProduceMime;
import javax.ws.rs.UriParam;
import javax.ws.rs.core.HttpContext;
import javax.ws.rs.core.Response;
import javax.ws.rs.core.UriInfo;
import org.apache.abdera.model.Entry;
import org.apache.abdera.model.Feed;

@ProduceMime("application/atom+xml")
@ConsumeMime("application/atom+xml")
@Path("myfeed")
public class FeedResource {
   @HttpContext
   private UriInfo uriInfo;

   @GET
   public Feed getFeed() {
       Feed f = AbderaSupport.getAbdera().getFactory().newFeed();
       f.setTitle("...");
       f.setId(...);
       f.addAuthor(...);
       f.setUpdated(...);
       URI feedLink = uriInfo.getRequestUri();
       f.addLink(feedLink.toString(),"self");
       for (...) {
           Entry e = f.addEntry();
           URI entryLink = ...
	       f.addLink(entryLink.toString(),"alternate");
           ...
       }
       return f;
   }

   @POST
   public Response addEntry(Entry e) {
       Entry newEntry = AbderaSupport.getAbdera().newEntry();
       URI entryLink = ...;
       ...
       return Response.created(entryLink).entity(newEntry).build();
   }
}

The getFeed method creates an Abdera Feed instance and returns it in response to a GET request on the myfeed URI. The addEntry method is called with an Abdera Entry instance that corresponds to the body of a POST request. It does whatever is necessary to persist the new entry and then creates and returns a 201 Created response with a Location header and a new Entry corresponding to the persisted entry as the body of the response.

To allow use of Abdera's Entry and Feed types in resource method arguments and return types I had to implement the JAX-RS interfaces MessageBodyReader and MessageBodyWriter for those types. This turned out to be quite simple with Abdera:

import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import javax.ws.rs.ProduceMime;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.MultivaluedMap;
import javax.ws.rs.ext.MessageBodyReader;
import javax.ws.rs.ext.MessageBodyWriter;
import javax.ws.rs.ext.Provider;
import org.apache.abdera.Abdera;
import org.apache.abdera.model.Document;
import org.apache.abdera.model.Element;
import org.apache.abdera.model.Entry;
import org.apache.abdera.model.Feed;

@Provider
@ProduceMime("application/atom+xml")
@ConsumeMime("application/atom+xml")
public class AbderaSupport implements MessageBodyWriter<Object>, 
       MessageBodyReader<Object> {

   private final static Abdera abdera = new Abdera();

   public static Abdera getAbdera() {
       return abdera;
   }

   public long getSize(Object arg0) {
       return -1;
   }

   public boolean isWriteable(Class<?> type) {
       return (Feed.class.isAssignableFrom(type) || Entry.class.isAssignableFrom(type));
   }

   public void writeTo(Object feedOrEntry, MediaType mediaType, 
           MultivaluedMap<String, Object> headers, 
           OutputStream outputStream) throws IOException {
       if (feedOrEntry instanceof Feed) {
           Feed feed = (Feed)feedOrEntry;
           Document<Feed> doc = feed.getDocument();
           doc.writeTo(outputStream);
       } else {
           Entry entry = (Entry)feedOrEntry;
           Document<Entry> doc = entry.getDocument();
           doc.writeTo(outputStream);
       }
   }

   public boolean isReadable(Class<?> type) {
       return (Feed.class.isAssignableFrom(type) || Entry.class.isAssignableFrom(type));
   }

   public Object readFrom(Class<Object> feedOrEntry, MediaType mediaType, 
           MultivaluedMap<String, String> headers, 
           InputStream inputStream) throws IOException {
       Document<Element> doc = getAbdera().getParser().parse(inputStream);
       Element el = doc.getRoot();
       if (feedOrEntry.isAssignableFrom(el.getClass())) {
           return el;
       } else {
           throw new IOException("Unexpected payload, expected "+feedOrEntry.getName()+
               ", received "+el.getClass().getName());
       }
   }
}

Notice I actually implemented MessageBodyReader<Object> rather than MessageBodyReader<Entry> and MessageBodyReader<Feed>. This allowed me to support both types in the same class. Same for MessageBodyWriter. The isReadable and isWriteable methods allow a class to reject types it doesn't support.

Abdera is shaping up to be a nice API for working with Atom documents. Hopefully the above demonstrates how easy it is to integrate support for new formats into Jersey.

WADLing in Jersey

For a long time you've been able to retrieve a WADL document that describes the set of resources that make up a Jersey application. However, the returned WADL was generated at compile time and would only include resources that were statically known. In recent weeks we've switched many parts of Jersey including the WADL subsystem to use a shared abstract resource model that is built at runtime. This has allowed us to add some additional functionality that I'll describe here.

The main new feature is that you can now retrieve a WADL for an individual resource. The resource can be a static resource which is described in the application WADL or it can be a resource for which you've received a link in some returned representation. For an example, consider the Simple Console example in the Jersey distribution. This example has two resource:

http://127.0.0.1:9998/resources/form
http://127.0.0.1:9998/resources/form/colours

If you retrieve the "global" WADL (available at http://127.0.0.1:9998/resources/application.wadl) you get the following:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<application xmlns="http://research.sun.com/wadl/2006/10">
	<resources base="http://localhost:9998/resources/">
		<resource path="/form">
			<method name="GET">
				<response>
					<representation mediaType="text/html" />
				</response>
			</method>
			<method name="POST">
				<request>
					<representation mediaType="application/x-www-form-urlencoded" />
				</request>
				<response>
					<representation mediaType="text/html" />
				</response>
			</method>
			<resource path="colours" />
		</resource>
	</resources>
</application>

From this you can see that you can get the "form" resource and post a HTML form to it and both will return some HTML. Also notice that there's a resource element with path colours but no other information about that resource. This is due to the "colours" resource being dynamically created by the subresource locator in Form.java:

@Path("/form")
@ProduceMime("text/html")
public class Form {
    
    @Path("colours")
    public Colours getColours() {
        return coloursResource;
    }
    
    ...
}

With the latest changes to Jersey you can now get information on such dynamic resources by GETing the resource with an Accept: application/vnd.sun.wadl+xml header that specifies that you want WADL rather than another representation. The Jersey runtime will inspect the object returned by the subresource locator and dynamically generate a WADL description. Thus:

GET /resources/form/colours
Accept: application/vnd.sun.wadl+xml

will return:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<application xmlns="http://research.sun.com/wadl/2006/10">
	<resources base="http://localhost:9998/resources/">
		<resource path="form/colours">
			<method name="GET">
				<request>
					<param xmlns:xs="http://www.w3.org/2001/XMLSchema" type="xs:string" style="query" name="match" />
				</request>
				<response>
					<representation mediaType="text/plain" />
				</response>
			</method>
			<method name="GET">
				<request>
					<param xmlns:xs="http://www.w3.org/2001/XMLSchema" type="xs:string" style="query" name="match" />
				</request>
				<response>
					<representation mediaType="application/json" />
				</response>
			</method>
		</resource>
	</resources>
</application>

which corresponds to the resource class Colours.java.

The above works except in the case that the resource supports the GET method but does not declare the supported media types (equivalent to @ProduceMime("*/*")). In this case the resource GET method will be invoked for any value of the Accept including WADL so we've added an alternate way of getting the WADL. Rather than GETing the resource you can use a HTTP OPTIONS request. The body of the returned response will be a WADL describing the resource.

All of the above is currently implemented in the trunk and will be included in the forthcoming 0.5 snapshot release.

Integrating Jersey and Guice

Further to my recent post about integrating Jersey and Spring, Christian Rivasseau shows how to integrate Jersey and Guice. As Paul notes in comments to that post, we plan to further simplify integration of template engines like Freemarker but its great to see the existing resource provider SPI working for another IoC framework.

Early Draft Review of JSR 311

The first early draft of JSR 311 is now available for review, download the spec and Javadoc here. As the name suggests this is an early draft and the specification is far from complete but it does provide an early look at the current API. You can also find a list of current issues here.

Please send feedback on the specification, Javadocs or current issues to users [at] jsr311.dev.java.net.

Integrating Jersey and Spring

To further test the new resource provider SPI I described earlier I thought I'd try building a new resource provider that defers to Spring for resource creation. Note that I've never used Spring before so if there are better ways to accomplish what I did please let me know. I also ran into a problem that limited the integration more than I'd prefer (details at the end of this post). The Jersey dev list would be a great place to post any suggestions on how to improve the integration.

Implementation

The first task was to write a ResourceProvider implementation that uses Spring for resource creation. This is pretty straightforward and the getInstance method turned out to require only a few lines of code:

public Object getInstance(ResourceProviderContext context) {
  try {
    initSpringContext(context);
    initBeanName();
    Object resource = springContext.getBean(beanName, resourceClass);
    context.injectDependencies(resource);
    return resource;
  } catch (Exception ex) {
    throw new ContainerException("Unable to create resource", ex);
  }
}

The ResourceProvider interface only supplies a ResourceProviderContext to the getInstance method so I had to defer some initialization code to that method that would have fit better in the init method. I'll check if it makes sense to add a ResourceProviderContext to the init method but for this experiment I stuck with the SPI as is. The initSpringContext method uses the ResourceProviderContext to get the ServletContext which is then used to get the Spring ApplicationContext which has been initialized in the suggested way.

protected static synchronized void initSpringContext(
      ResourceProviderContext context) {
  if (springContext==null) {
    DummyResource r = new DummyResource();
    context.injectDependencies(r);
    springContext = WebApplicationContextUtils.getRequiredWebApplicationContext(
        r.servletConfig.getServletContext());
  }
}

DummyResource is a simple annotated static inner class that is used as an injection target:

public static class DummyResource {
  @Resource
  public ServletConfig servletConfig;
}

Jersey is based on annotated classes whereas Spring is based on named beans. The Spring ResourceProvider needs to map a class name to a Spring bean name which is accomplished with the initBeanName method:

protected synchronized void initBeanName() {
  if (beanName==null) {
    String names[] = springContext.getBeanNamesForType(resourceClass);
    if (names.length==0)
      throw new RuntimeException("No configured bean for "+resourceClass.getName());
    else if (names.length>1)
      throw new RuntimeException("Multiple configured beans for "+resourceClass.getName());
    beanName=names[0];
  }
}

I couldn't see any straightforward way to determine which bean to use for a given resource class if there is more than one defined at the top level so the provider requires a single top-level bean for each resource class.

The final task was to define a new annotation that instructs the Jersey runtime to use the Spring resource provider:

@Target({ElementType.TYPE})
@Retention(RetentionPolicy.RUNTIME)
@Documented
@ResourceFactory(SpringProvider.class)
public @interface SpringFactory {}

Example

With the above implemented you can now write a resource class whose instantiation is controlled by Spring:

@UriTemplate("{id}")
@SpringFactory
public class SpringResource {
    
  private String name;
    
  public SpringResource() {
    name="unset";
  }

  public String getName() {
    return name;
  }
    
  public void setName(String name) {
    this.name = name;
  }
    
  @HttpMethod("GET")
  @ProduceMime("text/plain")
  public String getDescription() {
    return "Name: "+getName();
  }
}

With the following applicationContext.xml in the WEB-INF directory the resource will report its name as "Mr. Bean" thus demonstrating Spring-provided resource injection.

<beans xmlns="http://www.springframework.org/schema/beans"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="
      http://www.springframework.org/schema/beans
      http://www.springframework.org/schema/beans/spring-beans-2.0.xsd">
  <bean id="bean1" scope="prototype" class="com.sun.ws.rest.spring.resources.SpringResource">
    <property name="name" value="Mr. Bean"/>
  </bean>
</beans>

The complete code is available here. No libraries are included in the ZIP file so you'll have to patch up the references to the required Jersey and Spring libraries.

Integration Shortcoming

I couldn't see any way to add support for Jersey-defined resources as constructor parameters. In an earlier post I described the new support for non-empty constructors. When using the Spring resource provider, the constructor can only include resources defined within the Spring configuration, not values supplied by Jersey. Is there a Spring API I could use to programatically add support for Jersey-defined constructor parameters ?

NetBeans Jersey Tooling

A couple of video's demonstrating NetBeans 6 support for creating Jersey-based Web services:

• Part 1: Building and Deploying
• Part 2: Testing and Invoking

This is an up-to-date version of the demo we showed in our JavaOne session this year.

New Jersey Features

I recently merged the experimental "resourcefactory" branch back into the main Jersey trunk. The new code includes the following features and changes:

• A change to the default resource object lifecyle from singleton to per-request
• Support for non-empty resource class constructors
• A new SPI for plugging-in new resource providers

Details on these changes below

Lifecycle Changes

Prior to the merge the Jersey runtime would create a single instance of a resource class and use that for all requests. Resource classes had to be re-entrant and thread safe. Following the merge the default scope for a resource object is per-request, i.e. the Jersey runtime creates a new instance of a resource class for each request. This change brings Jersey back in line with the current JSR 311 specification which requires a default per-request life-cycle. The old behavior can be recovered by annotating a resource class with the Singleton annotation from the com.sun.ws.rest.spi.resource package. If you want to be explicit about a resource classes life-cycle rather than rely on the default you can also use the PerRequest annotation from the same package. More on how these annotations work in the description of the new SPI below.

Non-empty Constructors

Prior to the merge a resource class was required to have a zero argument constructor that the Jersey runtime would use when creating the singleton instance. Following the merge this requirement has been relaxed such that constructors can now have arguments. This in conjunction with the change to a default per-request lifecycle allows a more natural use of constructors to perform instance initialization, e.g.:

@UriTemplate("entries/{id}")
public class Entry {
  
  EntryEntity entity;

  public Entry(@UriParam("id") String id) {
    entity = findEntryEntity(id);
    ...
  }

  @HttpMethod
  @ProduceMime({"application/json", "application/xml"})
  public EntryEntity getEntity() {
    return entity;
  }
}

The arguments allowed in a resource class constructor depends on the resource provider used to create an instance of the resource class (see the description of the new SPI below). For default per-request resource classes you can use any combination of parameters annotated with UriParam, UriParam, QueryParam, MatrixParam, HeaderParam or HttpContext. Singleton resource classes must still have a zero argument constructor.

Resource Provider SPI

The new resource provider SPI allows plugging-in of custom resource class factories and is used internally to support the per-request and singleton life-cycles. If you're not planning to develop a new resource provider you can safely stop reading here unless of course you are curious as to how the above features are implemented.

A resource provider is responsible for creating instances of resource classes for the Jersey runtime to use. A resource provider implements ResourceProvider and must have a zero-argument constructor. The Jersey runtime will create a new instance of a resource provider for each resource class that uses that provider so there is no need for a provider to maintain a list or map of resource classes and instances thereof.

The Jersey runtime identifies the resource provider to use for a particular resource class by examining the annotations on that resource class. It examines each class-level annotation in turn looking for a ResourceFactory meta-annotation and then uses the value of that annotation or the default per-request provider if none is found as the resource provider. This approach allows you to define new annotations that point to a particular resource provider, e.g. Jersey defines the Singleton annotation as follows:

@Target({ElementType.TYPE})
@Retention(RetentionPolicy.RUNTIME)
@Documented
@ResourceFactory(SingletonProvider.class)
public @interface Singleton {}

Note the ResourceFactory meta-annotation which points to SingletonProvider, the Jersey-supplied resource provider for singleton resource classes.

The getInstance method of ResourceProvider supplies a ResourceProviderContext that a resource provider can use to obtain values for the resource class constructor's arguments and perform dependency injection on a newly created resource class instance, PerRequestProvider shows an example of its use. A resource provider is expected to supply values for constructor arguments unknown to Jersey (anything not annotated with UriParam, UriParam, QueryParam, MatrixParam, HeaderParam or HttpContext) and inject additional dependencies as required.

Its hoped that this SPI will support integration of Jersey with a variety of frameworks. Let us know if anything is unclear or needs improvement.

Comments on JSR 311

Patrick Mueller has some nice things to say about JSR 311 but also has some concerns that I'd like to address.

The first and easiest to address is the license under which the RI (Jersey) is distributed. Rather than GPL as Patrick states, Jersey is actually available under the CDDL license. Apologies if that wasn't entirely clear from the Jersey home page, I've just fixed that.

Patrick also raised a metaphorical eyebrow over some of the non-goals listed in the latest editors draft of the specification (PDF):

Support for Java versions prior to J2SE 5.0: The API will make extensive use of annotations and will require J2SE 5.0 or later. Read: Sorry, J2ME. Sorry, folks stuck on Java 1.4.

Yes, certainly it would have been good to be able to offer something to Java 1.4 users but we felt that an annotation-based approach offered the best solution going forward and decided to focus on that. On the Java ME side, the fact that 311 is primarily a server side API (see the final point) means it is less suitable for the ME environment anyway.

Description, registration and discovery: The specification will neither define nor require any service description, registration or discovery capability. Read: We'll figure this out later; hopefully we won't have to change anything in this spec once we start thinking about this aspect of the problem.

Several folks whose opinions I respect think that description is antithetical to REST and given that there's no accepted standard for describing RESTful Web services (beyond the resource representations themselves) its hard to see what we could have done here. Certainly I have my own favorite description language and Jersey will automatically produce a WADL description of a set of JSR 311 resource classes but it would be wrong to require support for something like WADL in JSR 311.

Client APIs: The specification will not define client-side APIs. Other specifications are expected to provide such functionality. Read: We'll figure this out later; hopefully we won't have to change anything in this spec once we start thinking about this aspect of the problem.

The goal of JSR 311 is to make it simple to write RESTful Web services in Java. Of course we're keeping an eye out for classes and interfaces that might be useful on the client side (those concerned with URI manipulation seem like reasonable candidates) but the primary focus is on exposing Java objects as RESTful services so I don't think there's too much danger of unexpected overlap with any new client-side HTTP API that might emerge in the future.

Comments on JSR 311

Patrick Mueller has some nice things to say about JSR 311 but also has some concerns that I'd like to address.

The first and easiest to address is the license under which the RI (Jersey) is distributed. Rather than GPL as Patrick states, Jersey is actually available under the CDDL license. Apologies if that wasn't entirely clear from the Jersey home page, I've just fixed that.

Patrick also raised a metaphorical eyebrow over some of the non-goals listed in the latest editors draft of the specification (PDF):

Support for Java versions prior to J2SE 5.0: The API will make extensive use of annotations and will require J2SE 5.0 or later. Read: Sorry, J2ME. Sorry, folks stuck on Java 1.4.

Yes, certainly it would have been good to be able to offer something to Java 1.4 users but we felt that an annotation-based approach offered the best solution going forward and decided to focus on that. On the Java ME side, the fact that 311 is primarily a server side API (see the final point) means it is less suitable for the ME environment anyway.

Description, registration and discovery: The specification will neither define nor require any service description, registration or discovery capability. Read: We'll figure this out later; hopefully we won't have to change anything in this spec once we start thinking about this aspect of the problem.

Several folks whose opinions I respect think that description is antithetical to REST and given that there's no accepted standard for describing RESTful Web services (beyond the resource representations themselves) its hard to see what we could have done here. Certainly I have my own favorite description language and Jersey will automatically produce a WADL description of a set of JSR 311 resource classes but it would be wrong to require support for something like WADL in JSR 311.

Client APIs: The specification will not define client-side APIs. Other specifications are expected to provide such functionality. Read: We'll figure this out later; hopefully we won't have to change anything in this spec once we start thinking about this aspect of the problem.

The goal of JSR 311 is to make it simple to write RESTful Web services in Java. Of course we're keeping an eye out for classes and interfaces that might be useful on the client side (those concerned with URI manipulation seem like reasonable candidates) but the primary focus is on exposing Java objects as RESTful services so I don't think there's too much danger of unexpected overlap with any new client-side HTTP API that might emerge in the future.

Jersey

We just released the source code for the reference implementation of JSR 311 as project Jersey. Paul's post has more details.

It seems we've already had our first bug report, keep them coming.

What Leonard Said

I've been offline on holiday in the UK for the last couple of weeks and it seems I've been missing out on an interesting debate about whether a description language for RESTful services is useful or not. The debate so far is nicely summarized by Arnon Rotem-Gal-Oz in Debate: Does REST Need a Description Language?.

Obviously as the author of the WADL specification I think there is some utility in such a language and normally I'd be tempted to write a longer post addressing some of the points raised but rather than do that I'll instead direct you to Leonard Richardson's It's Just A Hypermedia Format which covers most of the things I'd have said had they not already been expressed so well.

One thing I would like to add since many people are concerned with the link to XML schema: the element attribute of the WADL representation element is optional; WADL can use XML schemas but it isn't built on them. In fact WADL doesn't presume XML will be used for entity bodies and it can easily describe an application built using other media types.

Google REST Describe

Thomas Steiner just released the first working demo of Google REST Describe. This tool takes a novel approach to creating WADL descriptions of Web applications by analyzing sample requests and responses and applying heuristics to extract descriptive information. You can then manually add or modify metadata to improve the automatically generated description. Very cool.

Check it out here.

Sun Web Developer Pack R1 with WADL and RESTful WS goodies

As Dave already blogged: "The Sun Web Developer Pack (SWDP) finally uncloaked today". In addition to the items Dave mentioned I'd also like to point out that the pack contains prebuilt binaries of some WADL tools and an early access API and runtime for development of RESTful Web services.

I blogged earlier about JSR 311, I expect the API produced by JSR 311 will differ significantly to the RESTful WS API in SWDP R1 but the experience we gained building it and the runtime to support it has been very useful for testing some of our early assumptions and clarifying our thinking about the JSR. We hope you'll take it for a spin and let us know what you think.

Google WADL Tools

Thomas Steiner has announced plans for a couple of tools to simplify RESTful development based around WADL. Tentatively titled "Google REST Compile" and "Google REST Describe", the first will read a WADL description and output client code for a variety of programming languages whilst the second will aid in producing a WADL file from an existing textual description. I'm delighted to see others picking up on WADL and hope the project goes well.

JSR 311: Java API for RESTful Web Services

I was planning to wait until after the ballot to blog this but it seems lots of folks have already noticed and blogged about it so I figure I might as well put my 2p in now.

Reaction so far has been mixed, ranging from "cool... I hope they don't screw it up" through "please ask Sun to reconsider this proposal" so I thought I'd try to address a couple of the negative perceptions noted so far.

The API will be overly abstract/generic

HTTP is the target of this API, we went back and forth a bit on the name and in the end decided to use RESTful in there to highlight that it will focus on RESTful use of HTTP. It doesn't mean we plan to develop an abstract REST API with a binding to HTTP.

The API will presume developers are incompetent

This one seems to come from this sentence in the JSR submission: "Correct implementation requires a high level of HTTP knowledge on the developer's part." OK, perhaps this wasn't put very well, but the plan is to offer default implementations of things like content negotiation and precondition support to save developers having to implement those things themselves (unless they really want to).

A couple of people also complained that the JSR wasn't very specific and didn't include much in the way of detail. JSRs are like that, they describe the problem not the solution. FWIW, here's the sort of thing I currently have in mind though, of course, the final APIs will probably be very different following expert group discussions.

@UriTemplate("widgets/{widgetid}")
@ConsumeMime("application/widgets+xml")
@ProduceMime("application/widgets+xml")
public class Widget {

  @HttpMethod(GET)
  public Representation getWidget(@UriParam("widgetid") String id) {
    String replyStr = getWidgetAsXml(id);
    return new StringRepresentation(replyStr,
      "application/widgets+xml");
  }
  
  @HttpMethod(PUT)
  public void updateWidget(@UriParam("widgetid") String id,
    Representation<Source> update) {
    updateWidgetFromXml(id, update);
  }
  
  @LastModified
  public Date getChangeDate(@UriParam("widgetid") String id) {
    return getLastChanged(id);
  }
}

The above is a resource with a URI of {prefix}/widgets/{widgetid} where {prefix} is the base URI that depends on where the resource is deployed and {widgetId} is a URI template variable that varies for each widget. The resource can produce representations using the application/widgets+xml media type and can also accept updates using the same format. The API runtime would do the work of identifying the correct class for an inbound request based on the URI, the HTTP method, the media type of any request entity and the desired format of the response. The API runtime would also arrange for responses to have a Last-Modified header and provides precondition support based on the output of the getChangeDate method.

URI Templating

One of the features added to WADL in the latest revision of the specification was the ability to embed parameters in resource path definitions. E.g., the following describes a set of resources identified by a parameterized URI:

<resources base="http://example.com/">
  <resource path="applications/widgets/{widget_id}">
    ...
  </resource>
<resources>

The above extract describes a set of resources identified by the URIs conforming to the following template:

http://example.com/applications/widgets/{widget_id}

Where {widget_id} is replaced by the value of the widget_id parameter when accessing a particular resource.

Defining URIs in this way is useful in a number of scenarios and for the last few months I've been working with Joe Gregorio, Mark Nottingham, David Orchard, James Snell and DeWitt Clinton on a more general specification for parameterized URIs which we are calling URI Templates. The first draft of the specification has now been published as an Internet Draft. To provide feedback on the specification, join the W3C URI mailing list.

URI Templates are deliberately simple; WADL offers a mechanism by which additional metadata can be supplied for a template variable, e.g.:

<resources base="http://example.com/">
  <resource path="applications/widgets/{widget_id}">
    <param name="widget_id" style="plain" type="xsd:int">
      <doc xml:lang="en">
        <xhtml:p>Specifies the identifier of a particular widget.</xhtml:p>
      </doc>
    </param>
    ...
  </resource>
<resources>

When the value of the name attribute of a child param element matches the name of an embedded URI template variable it supplies additional information about that variable. In the above example the XML Schema type of widget_id is xsd;int and some human readable documentation describing the variable is also provided.

The param element also allows specification of a list of possible values (using the option element), default or fixed values, whether a value is required or not, and whether the value is single valued or repeating.

Announcing wadl.dev.java.net

I'm pleased to announce the formation of a new project on java.net for the development of WADL and associated tools: http://wadl.dev.java.net/. The goals of the project are spelled out here.

I've primed the pump with an initial release of a tool that generates Java stubs for Web applications described using WADL. Also included is a sample application that demonstrates use of the tool with the Yahoo News Search Service. The code implements the ideas I blogged here but is still beta quality. I'm hoping public availability will help flush out the bugs and highlight areas that need more work.

If you are interested in WADL and would like to help then please consider joining the project - there are many ways to contribute.

WADL Revision

In a previous entry I described some enhancements that were under consideration for an updated version of the WADL specification. The resulting updated WADL specification is now available along with the associated W3C XML Schema and RelaxNG Schema.

In revising the specification I made a couple of breaking changes to simplify the language and improve clarity so this version of the specification uses a new XML namespace for the WADL elements. Here's what's new:

Resource References

As previously described, resource elements can now act as references to resources described elsewhere to avoid duplication and promote re-use.

Identification of Links

As previously described, a link element has been added to support identification of links within representations and to indicate the resource such links identify.

Documentation

A doc element has been added as an optional first child element of all of the other WADL defined elements. The doc element has an xml:lang attribute to identify the language of the documentation, a title attribute that provides a short plain text description of the element being documented, and contains mixed content for the detailed documentation. E.g.:

<application xmlns="..." xmns:xhtml="...">

  <doc xml:lang="en" title="Online widget catalogue">
    <xhtml:p>An online widget catalogue for all your widgeting needs.</xhtml:p>
  </doc>
  
  <resources base="http://example.com/">
    <doc xml:lang="en" title="Commercial widgets">
      <xhtml:p>Widgets for the commercial sector.</xhtml:p>
    </doc>
    ...
  </resources>
  
</application>

Multiple sibling doc elements can be used when documentation is available in multiple languages.

Resource Rework

A resource element can now encompass multiple path segments within a URI and path segments can either be static or contain embedded path parameters. E.g.:

<resources base="http://example.com/">
  <resource path="applications/widgets/{widgetId}">
    ...
  </resource>
</resources>

The above extract describes a set of resources identified by the URIs conforming to the following template:

http://example.com/applications/widgets/{widgetId}

Where {widgetId} is replaced by the value of the widgetId parameter when accessing a particular resource.

Parameter Rework

The path_variable, query_variable and representation_variable elements have been replaced by a single param element.

The param element has the union of the attributes from the variable elements it replaces and adds a style attribute that indicates the use of the parameter (plain, matrix or form) and a default attribute to indicate the default value of the parameter if not explicitly specified.

A new option element supports enumerated parameter values without the need to resort to XML Schema.

Future Work

With these changes I'm now pretty happy with the base language but there remain a couple of things that still need more work:

• Support for declaring security requirements. Declaring the need for HTTP basic or digest authentication would be straightforward but often other schemes are used and this area needs some further research.
• WADL can already be used to describe Web applications that use a variety of data formats but I'd like to explore whether specific formats, e.g. JSON, would benefit from additional support in WADL.

I believe that both of the above can be addressed with non-breaking extensions to the language.

WADL Enhancements

In this entry I describe some of the language enhancements that will be in the next version of the WADL specification. These enhancements are the result of feedback I've received from other folks using the language and experience gained implementing a Java processor for WADL.

Resource References

Currently you can define a method or representation in one place and then include that definition by reference elsewhere. This is useful if the same representation is used in multiple method definitions or if the same method applies to more than one resource. Extending this capability to the resource element allows a common resource hierarchy to be re-used in multiple places. E.g.:

<application>

  <resources base="http://example.com/">
    <resource href="stock">
      <resource href="#widgets"/>
    </resource>
    <resource href="order">
      <resource href="#widgets"/>
    </resource>
  </resources>

  <resource id="widgets" path="widgets">
    ...
  </resource>

</application>

Identification of Links

Whilst it would be possible at runtime to compare a URI embedded in a representation with the URIs of all of the resources in a WADL description to identify which resource it identifies, it would be preferable to be able to specify the resource targeted by a link up front where that information is known.

This is achieved in WADL by adding a new element to the language: link. The link element is a child of the existing representation_variable element and is used to indicate that the value of a representation component is a link to a described resource. The link element is syntactically similar to the HTML link element but differs semantically in that its href attribute identifies a resource element rather than a linked document. The parent representation_variable element identifies that part of the representation that contains the link.

E.g., if an application consists of the resources:

http://example.com/widgets
http://example.com/widgets/{widgetid} (where {widgetid} is a variable component to the URI)

and a GET on the first resource gets me a WidgetList:

<eg:WidgetList xmlns:eg="...">
  <eg:Widget uri="http://example.com/widgets/xyzzy1"/>
  <eg:Widget uri="http://example.com/widgets/abccb1"/>
  ...
</eg:WidgetList>

such that eg:Widget/@uri links to one of the http://example.com/widgets/{widgetid} resources, then this can be described as follows:

<application xmlns:...>

  <resources base="http://example.com/">
    <resource uri="widgets">
      <method href="#widgetList"/>
      <resource id="widgetResource" uri="{widgetid}">
        ...
      </resource>
    </resource>
  </resources>

  <method name="GET" id="widgetList">
    <response>
      <representation mediaType="application/xml" element="eg:WidgetList">
        <representation_variable path="/Widgets/Widget/@uri" repeating="true">
          <link href="#widgetResource" rel="child" rev="index"/>
        </representation_variable>
      </representation>
    </response>
  </method>

</application>

The values of the rel and rev attributes are an open set including those used by convention in HTML. Similar to HTML, a profile attribute may be added to the application element to define a namespace for additional link types.

Documentation

The content model of WADL-defined elements is open for extension and I originally thought that documentation could be added by including html or xhtml elements within a WADL description. Thinking on this some more I've come to the conclusion that it would be better to define a standard documentation element that is an optional first child element of all of the other WADL defined elements and whose content is the documentation for the parent element. E.g.:

<application>

  <documentation><p>An online widget catalogue.</p></documentation>
  
  <resources base="http://example.com/">
    <documentation><p>Commercial widgets.</p></documentation>
    ...
  </resources>
  
</application>

Multi-segment Resource Paths

Its a pain to require multiple nested resource elements when the intermediate resource elements don't represent a useable resource. Relaxing the constraint on the content of the resource/@uri attribute allows you to write:

<resources base="http://example.com/">
  <resource uri="applications/catalogues/widgets">
    ...
  </resource>
</resources>

instead of

<resources base="http://example.com/">
  <resource uri="applications">
    <resource uri="catalogues">
      <resource uri="widgets">
        ...
      </resource>
    </resource>
  </resource>
</resources>

Much more compact and readable !

Matrix URI Support

The path_variable element will be extended with a matrix attribute of type xsd:boolean and default value of "false". When the value of this attribute is "true" the path variable is treated as a matrix variable and is inserted into the uri as a name=value pair with a leading ';' character. E.g.:

<resource uri="widgets">
  <path_variable name="maxPrice" type="xsd:decimal" matrix="true"/>
  <path_variable name="minPrice" type="xsd:decimal" matrix="true"/>
  ...
</resource>

If the value of maxPrice is 10.00 and the value of minPrice is 5.00 then the relative URI of the above resource is:

widgets;maxPrice=10.00;minPrice=5.00

A null value for a matrix path variable results in it being omitted from the uri.

Still on the Drawing Board

Other enhancements still on the drawing board include:

• Shorthand notation for parameterized URIs, e.g. widgets/{widgetid} where {widgetid} in the URI literal takes the place of a path_variable element.
• Support for declaring security requirements.
• Better support for non-XML payloads including JSON.

Mapping WADL to Java

In recent entries I've described the REST (or Web Style) oriented features of JAX-WS and touched on a new language, WADL, that aims to provide a description of Web style services. In this entry I'll bring together these two threads to show how a language like WADL can be used to generate strongly-typed client-side stubs layered on JAX-WS and JAXB.

One of the key differences between Web-style and the style commonly associated with WS-* is the inversion of the balance between Web resources (something identified by a URI) and methods that can be applied to those resources. In the former style you often find a single URI (or a port in WSDL terminology) that exposes many custom methods described by a WSDL port type. In the Web-style you find many URIs, each of which supports a subset of a few standard HTTP methods.

The WS-* style maps quite naturally to a single Java class/interface per WSDL port type that exposes Java methods corresponding to each WSDL method. This is the approach taken by JAX-WS which describes in detail how to map between a WSDL port type and a Java class/interface (see chapters 2 and 3). When considering Web-style however, the mapping to Java is less obvious but it seems reasonable to try to preserve the hierarchical nature of HTTP URIs while maintaining the mapping between a Web resource and a Java class/interface. There are a couple of candidate hierarchical structures in the Java language: packages and nested inner classes. Given that the relationship between resources isn't always obvious it seems prudent to avoid member classes which require an instance of the parent class for instantiation. That leaves packages or nested static member classes, I chose the latter after a fair bit of back-and-forth but I think either approach would work.

Lets look at an example. The following WADL file describes the Yahoo! News Search service:

<?xml version="1.0"?>
<application xmlns:xsd="http://www.w3.org/2001/XMLSchema"
  xmlns:yn="urn:yahoo:yn"
  xmlns:ya="urn:yahoo:api"
  xmlns="http://research.sun.com/wadl">
 
  <grammars>
    <include href="NewsSearchResponse.xsd"/>
    <include href="NewsSearchError.xsd"/>
  </grammars>
  
  <resources base="http://api.search.yahoo.com/NewsSearchService/V1/">
    <resource uri="newsSearch">
      <method href="#search"/>
    </resource>
  </resources>
    
  <method name="GET" id="search">
    <request>
      <query_variable name="appid" type="xsd:string" required="true"/>
      <query_variable name="query" type="xsd:string" required="true"/>
      <query_variable name="type" type="xsd:string"/>
      <query_variable name="results" type="xsd:int"/>
      <query_variable name="start" type="xsd:int"/>
      <query_variable name="sort" type="xsd:string"/>
      <query_variable name="language" type="xsd:string"/>
    </request>
    <response>
      <representation mediaType="application/xml" element="yn:ResultSet"/>
      <fault id="SearchError" status="400" mediaType="application/xml"
        element="ya:Error"/>
    </response>
  </method>

</application>

The service actually supports a few more optional query parameters but I've omitted those to keep the length of the example down. The WADL file describes a single resource (http://api.search.yahoo.com/NewsSearchService/V1/newsSearch) that supports the HTTP GET method with a set of URI query parameters and returns an XML document whose root element is either yn:ResultSet or ya:Error. The WADL file includes two XML Schema files provided by Yahoo! that define the content models for these two elements. Lets see how this looks when mapped to a client-side Java stub:

package com.yahoo.search;

public class Endpoint {

  ...

  public static class NewsSearch {

    public NewsSearch()
      throws JAXBException
    {
      ...
    }

    public ResultSet getAsResultSet(String appid, String query)
      throws SearchError
    {
      ...
    }

    public ResultSet getAsResultSet(String appid, String query, String type, 
      Integer results, Integer start, String sort, String language)
      throws SearchError
    {
      ...
    }
  }
}

I've omitted the implementation code as I want to concentrate on the structure of the generated code rather than implementation details. The first thing to note is the top level class Endpoint has no methods, it is there purely as a container for nested static inner classes mapped from resources in the WADL file. In this example there is a single nested class (NewsSearch) corresponding to the single resource in the WADL file, if this resource had sub-resources then these would be represented using nested static inner classes of NewsSearch. The constructor of the NewsSearch class takes no arguments in this example; if the resource had any variable path components (see <path_variable> in WADL), the constructor would include parameters that provide the value of each variable component so that the complete URI of the corresponding resource can be generated when the class is instantiated.

Once an instance of NewsSearch has been obtained, its methods may be used to invoke the news search service. In this example there are two variants of the GET method described in the WADL file: for convenience the first includes only the query parameters marked as required in the WADL file, the second includes both required and optional query parameters. Both methods return an instance of the same ResultSet class, this is a JAXB-generated class corresponding to the yn:ResultSet XML element. At first sight the Java method naming might seem a little odd since the name of the returned class is duplicated within the method name, this is done to handle the case where a resource supports multiple different representations of the same resource (not shown here) since two Java methods with the same name in the same class cannot differ only by return type.

Notice also that both methods throw SearchError, this is a generated exception class that follows the pattern used by JAX-WS for WSDL declared faults:

package com.yahoo.search;

public class SearchError
    extends Exception
{
    protected Error m_faultInfo;

    public SearchError(String message, Error faultInfo) {
        super(message);
        m_faultInfo = faultInfo;
    }

    public Error getFaultInfo() {
        return m_faultInfo;
    }
}

where Error is a JAXB-generated class corresponding to the ya:Error XML element.

Using this approach we can write client code for the Yahoo! News Search service as follows:

Endpoint.NewsSearch s = new Endpoint.NewsSearch();
ResultSet results = s.getAsResultSet("jaxws_restful_sample", "java");
for (ResultType result: results.getResult()) {
    System.out.println(result.getTitle()+" ("+result.getClickUrl()+")");
}

Hopefully you'll agree that this is a distinct improvement when compared to using JAX-WS and JAXB without code generation though, as is often the case, you sacrifice ultimate flexibility and increase the coupling between client and server as the price for such convenience.

I'm currently working on an implementation of a WADL to Java command line tool and Ant plug-in using the techniques described above. Its currently still a prototype but it produces working code for a variety of interesting WADL feature combinations and I hope to release something people can play with soon. In the meantime, if you are interested in this kind of tooling and/or have a WADL file you'd like me to add to my test set then please email me at firstname.lastname@sun.com (substituting the relevant values).

JAX-WS and Binary Data

In previous entries I've shown how to use the JAX-WS APIs to publish and use RESTful Web services. Up till now I've focussed mainly on XML/HTTP since this is what most services use but JAX-WS can also handle other kinds of data. This entry shows how to use the JAX-WS APIs to work with binary image data and uses the popular Flickr service to illustrate the capabilities.

The Flickr APIs come in three varieties: REST, XML-RPC and SOAP. We'll concentrate on the REST API for this entry and develop some code that will search a particular users photo stream using keywords. Before you start developing your own application you'll need to obtain an API key from Flickr; API keys are free for non-commercial use, I'll include the one I obtained in the code below but please get a new key for any new applications you might develop.

The initial boilerplate code is the same as shown in previous Dispatch examples but I'll repeat it here for completeness:

URI baseURI = new URI("http://www.flickr.com/");
URI staticBaseURI = new URI("http://static.flickr.com/");
QName serviceName = new QName("FlickrService", baseURI.toString());
Service flickrService = Service.create(serviceName);
QName flickrPortName = new QName("flickr_port",baseURI.toString());
URI address = new URI("http", null, baseURI.getHost(), baseURI.getPort(),
    baseURI.getPath()+"services/rest/", null, null);
flickrService.addPort(flickrPortName, HTTPBinding.HTTP_BINDING, address.toString());
Dispatch<Source> flickrDispatch = flickrService.createDispatch(flickrPortName, 
    Source.class, Service.Mode.PAYLOAD);

The Flickr photo search method takes a user ID rather than a user name so we first have to call the find by username method to convert the human-friendly username into the corresponding userId:

// set up the method call and parameters
Map<String, Object> requestContext = flickrDispatch.getRequestContext();
requestContext.put(MessageContext.HTTP_REQUEST_METHOD, "GET");
String queryString = "method=flickr.people.findByUsername" +
    "&api_key=f2b38aa733c997301cd7153f05d15410" + 
    "&username=" + java.net.URLEncoder.encode(username, "utf-8");
requestContext.put(MessageContext.QUERY_STRING, queryString);
            
// execute query
Source result = flickrDispatch.invoke(null);
DOMResult domResult = new DOMResult();
Transformer trans = TransformerFactory.newInstance().newTransformer();
trans.transform(result, domResult);
            
// parse results
XPathFactory xpf = XPathFactory.newInstance();
XPath xp = xpf.newXPath();
String userId = xp.evaluate("/rsp/user/@nsid", domResult.getNode());

With userID and a set of keywords (aka tags) we can now execute the search:

queryString = "method=flickr.photos.search" +
    "&api_key=f2b38aa733c997301cd7153f05d15410" +
    "&user_id=" + java.net.URLEncoder.encode(user, "utf-8") + 
    "&per_page=10" + 
    "&tags=" + java.net.URLEncoder.encode(tags, "utf-8");
requestContext.put(MessageContext.QUERY_STRING, queryString);
            
// execute query
result = flickrDispatch.invoke(null);
domResult = new DOMResult();
trans.transform(result, domResult);
            
// parse results
XPath xp = xpf.newXPath();
NodeList resultList = (NodeList)xp.evaluate("/rsp/photos/photo", 
    domResult.getNode(), XPathConstants.NODESET);
ArrayList<FlickrItem> list = new ArrayList<FlickrItem>();
for (int i=0;i<resultList.getLength();i++) {
    String id = xp.evaluate("@id", resultList.item(i));
    String title = xp.evaluate("@title", resultList.item(i));
    String secret = xp.evaluate("@secret", resultList.item(i));
    String server = xp.evaluate("@server", resultList.item(i));
    FlickrItem item = new FlickrItem(staticBaseURI.toString(), id, server, secret, title);
    list.add(item);
}

In the above I'm using a home-grown utility class FlickrItem to hold the data for a particular photo. I won't reproduce it in its entirety here since its a simple JavaBean with only one real trick: it can produce a URL to a particular photo using the algorithm described here. The relevant method is:

public URL getMediumURL() {
    return new URL(baseURI+server+"/"+id+"_"+secret+".jpg");
}

All of the above was really just a necessary precursor to get to what I really wanted to show: working with binary data using Dispatch. Given a particular FlickrItem member of list resulting from the photo search we can retrieve the photo as an Image as follows:

public Image getImage(FlickrItem item) {
    URL url = item.getMediumURL();
    Dispatch<DataSource> flickrStaticDispatch = flickrService.createDispatch(
        flickrPortName, DataSource.class, Service.Mode.MESSAGE);
    Map<String, Object> requestContext = 
       flickrStaticDispatch.getRequestContext();
    requestContext.put(MessageContext.HTTP_REQUEST_METHOD, new String("GET"));
    requestContext.put(Dispatch.ENDPOINT_ADDRESS_PROPERTY, url.toString());

    DataSource result = flickrStaticDispatch.invoke(null);
    BufferedImage img = ImageIO.read(result.getInputStream());
    return img;
}

Note the use of DataSource and Service.Mode.MESSAGE in the creation of the Dispatch instance. This creates a Dispatch that works at the message (vs payload) level using a DataSource to represent each message - this type of Dispatch can be used to work with any kind of binary data desired. It can also be used to perform HTTP PUT and POST with arbitrary types of request payload.

Its a simple matter of programming to take the above code and turn it into an application like that shown below. If you'd like to see this working alongside other code samples from previous entries then join me at JavaOne on Tuesday 16th May at 3.15pm for my session on JAX-WS and RESTful Web Services.

WADL now a Sun Labs Technical Report

The Web Application Description Language (WADL) specification has now been published as a Sun Labs Technical Report (TR-153).

RESTful Web Service Endpoints in JAX-WS

In previous entries I've covered use of the client side JAX-WS Dispatch<T> interface to consume RESTful Web services. In this entry I'd like to demonstrate use of the Provider<T> interface to develop a RESTful endpoint and then publish that endpoint using the Endpoint API. To try out the code in this entry you'l need to grab a recent nightly build of JAX-WS since the functionality was not implemented in JWSDP 2.0.

A JAX-WS Provider-based endpoint has to implement the javax.xml.ws.Provider interface:

public interface Provider<T> {
  public T invoke(T request);
}

You also have to pick the service mode in which the endpoint will operate and annotate the endpoint class to allow the JAX-WS runtime to identify the correct class to publish. Here's the start of our endpoint class (I'm going to interleave code and narrative but if you extract and stitch together the code segments below you'll end up with a complete class):

@WebServiceProvider 
@ServiceMode(value=Service.Mode.PAYLOAD)
public class SystemPropertiesProvider  implements Provider<Source> {

    @Resource
    protected WebServiceContext context;

Arun has an informative blog entry that discusses the difference between PAYLOAD and MESSAGE mode and describes which types of Provider work in which mode. The context member variable is initialized by the JAX-WS runtime using resource injection and provides access to the message context which contains various properties that an endpoint may find useful - more on this below.

The endpoint we'll develop here will allow a client to query the Java system properties of the endpoint implementation class, here is the implementation of the Provider.invoke method.

  public Source invoke(Source source) {
    Map<String, List<String>> query = getQueryParameters();
    String replyData = getSystemInfoAsXML(query.get("property"));
    StreamSource reply = new StreamSource(new StringReader(replyData));
    return reply;
  }

The getQueryParameters method obtains the HTTP query string using the appropriate message context property (recall the context member varaible introduced above) and then parses that string into a Map.

  protected Map<String, List<String>> getQueryParameters() {
    MessageContext msgCtx = context.getMessageContext();
    String queryString = (String)msgCtx.get(MessageContext.QUERY_STRING);
    HashMap<String, List<String>> params = 
      new HashMap<String, List<String>>();
    for (String s: queryString.split("&")) {
      String[] keyVal = s.split("=");        
      try {
        String key = URLDecoder.decode(keyVal[0], "UTF-8");
        String val = URLDecoder.decode(keyVal[1], "UTF-8");
        if (params.get(key) == null) {
          ArrayList<String> list = new ArrayList<String>();
          list.add(val);
          params.put(key,list);
        }
        else {
          ArrayList<String> list =
            (ArrayList<String>)params.get(key);
          list.add(val);
        }
      } catch (UnsupportedEncodingException ex) {
        ex.printStackTrace();
      }
    }
    return params;
  }

Its entirely possible that the above functionality already exists in some other class but, if it does, I didn't find it in the brief search I made. The getSystemInfoAsXML method takes a list of Java system property names and creates an XML document containing the name and value of each parameter.

  protected String getSystemInfoAsXML(List properties) {
    StringBuffer buf = new StringBuffer();
    buf.append("<?xml version='1.0' encoding='UTF-8'?>\n");
    buf.append("<?xml-stylesheet href='http://127.0.0.1/info.xsl' type='text/xsl'?>\n");
    buf.append("<properties>");

    for(String property: properties) {
      buf.append("<property name='");
      buf.append(property);
      buf.append("'>");
      buf.append(System.getProperty(property));
      buf.append("</property>");
    }

    buf.append("</properties>");
    return buf.toString();
  }

The xml-stylesheet processing instruction in the generated XML document will allow a modern browser to display the generated document in a human readable form; the stylesheet is included at the end of this entry for completeness. You'll have to put the stylesheet on a Web server somewhere and modify this line to point to the location you chose.

If we wanted to deploy the endpoint in a Java EE container then the above is all the code needed. However, for this example, we'll use the new JAX-WS Endpoint API to publish the endpoint without a container (this is really intended for lightweight endpoints and testing, don't throw out your application server). To do this we add a main method to the class.

  public static void main(String[] args) {
    Endpoint e = Endpoint.create(HTTPBinding.HTTP_BINDING,
      new SystemPropertiesProvider());
    e.publish("http://127.0.0.1:8084/system/info");
    System.out.println("Endpoint running");
    try {
      Thread.sleep(30000);
    } catch (InterruptedException ex) {
    }
    e.stop();
    System.out.println("Endpoint stopped");
  }
}

In the above we publish the endpoint for 30 seconds and then stop it, you could also wait for a keypress in the main thread or use some other scheme to determine when to stop the endpoint.

Now, stitch together the code segments from above, compile and run the class. Fire up a browser, point it to http://127.0.0.1:8084/system/info?property=os.version&property=os.name and see what OS the Java runtime thinks the endpoint is hosted on.

Here's the XSL stylesheet that converts the XML document into a HTML table:

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
  <xsl:template match="/">
    <html><body><table border="1">
      <tr>
        <th>Property</th>
        <th>Value</th>
      </tr>
      <xsl:for-each select="/properties/property">
        <tr>
          <td>
            <xsl:value-of select="@name"/>
          </td>
          <td>
            <xsl:value-of select="."/>
          </td>
        </tr>
      </xsl:for-each>
    </table></body></html>
  </xsl:template>
</xsl:stylesheet>

RESTful Web Services with JAX-WS and JAXB

In a previous blog entry I described how to use the dynamic client functionality of JAX-WS in combination with JAXP XPath capabilities to query a service providing data using XML/HTTP. In this entry I'd like to focus on how to use JAXB instead of XPath to get strongly typed access to the same data.

To use JAXB we have to obtain an XML schema for the documents that will be exchanged. The documentation for the Yahoo News Search service includes a link to an XML schema for the results produced by the service. The JAXB tool time components transform an XML schema file into a set of Java classes that can be used to interact with an instance of a document that conforms to the schema. The JAXB runtime components take care of converting between the XML serialization of the data and its in-memory Java class equivalent.

To integrate JAXB we have to run the xjc tool against the XML schema and then compile the resulting Java classes into the project. Using a standard NetBeans project, this is easily accomplished by overriding the -pre-compile Ant task by adding the following lines to the build.xml file:

<property name="jaxb.home" value="/path/to/jwsdp-2.0/jaxb" />
<taskdef name="xjc" classname="com.sun.tools.xjc.XJCTask">
  <classpath>
    <fileset dir="${jaxb.home}" includes="lib/*.jar" />
  </classpath>
</taskdef>
  
<target name="-pre-compile">
  <echo message="Compiling the schema..." />
  <xjc schema="NewsSearchResponse.xsd" package="com.yahoo.search" target="gen-src">
    <produces dir="gen-src/com/yahoo/search" includes="**/*.java" />
  </xjc>
</target>

Note that the above will produce a new directory separate to the application source code. You'll need to edit the NetBeans project preferences to include this directory as a source code directory to ensure the generated code is compiled into the project.

As before the first step in the application is to add a part for the Yahoo News Search service:

URI nsURI = new URI("urn:yahoo:yn");
QName serviceName = new QName("yahoo",nsURI.toString());
QName portName = new QName("yahoo_port",nsURI.toString());
Service s = Service.create(serviceName);
URI address = new URI("http", null, "api.search.yahoo.com", 80,
  "/NewsSearchService/V1/newsSearch",
  "appid=jaxws_restful_sample&type=all&results=10&sort=date&query=java",
  null);
s.addPort(portName, HTTPBinding.HTTP_BINDING, address.toString());

As before, note that the URI constructed above contains a set of input parameters to the search service. The appid parameter identifies the application to Yahoo, if you are writing a new application you should register a new ID for it here. In this instance we are creating a URI that will search for news about Java and return at most 10 results sorted by date.

Once the port is created we can use the Service instance to create a Dispatch instance:

JAXBContext jbc = JAXBContext.newInstance( "com.yahoo.search" );
Dispatch<Object> d = s.createDispatch(portName, jbc, Service.Mode.PAYLOAD);
Map<String, Object> requestContext = d.getRequestContext();
requestContext.put(MessageContext.HTTP_REQUEST_METHOD, new String("GET"));

Note the following differences in the code compared the previous version that used the XPath APIs:

• We create a JAXBContext object using the same Java package that we instructed the xjc tool to use for code generation above.
• The JAXBContext object is passed in when creating the Dispatch object and will be used by the Dispatch object for marshalling and unmarshalling between XML and Java.

Now that we have a Dispatch object we can execute a query and iterate through the results:

ResultSet rs = (ResultSet)d.invoke(null);
for (ResultType r: rs.getResult()) {
  System.out.printf("%s: (%s)\n",r.getTitle(),r.getClickUrl());
}

The ResultSet and ResultType are JAXB generated types that correspond to types in the XML schema we fed to the xjc tool above.

Unfortunately, due to a bug in the JAX-WS implementation included in JWSDP 2.0, the above code will not work and requires the following work-around. I'm told this is fixed in the latest builds but haven't confirmed that personally.

Unmarshaller u = jbc.createUnmarshaller();
ResultSet rs = (ResultSet)u.unmarshal((StreamSource)d.invoke(null));
for (ResultType r: rs.getResult()) {
  System.out.printf("%s: (%s)\n",r.getTitle(),r.getClickUrl());
}

RESTful Web Services with JAX-WS

The JAX-WS implementation that is included in the, recently released, JWSDP 2.0 preview now supports the XML/HTTP binding. This new functionality allows JAX-WS applications to implement and use RESTful Web services. Here is a worked example that demonstrates how to use JAX-WS to query the Yahoo News Search service.

The first step is to create a JAX-WS port for the Yahoo News Search service

URI nsURI = new URI("urn:yahoo:yn");
QName serviceName = new QName("yahoo",nsURI.toString());
QName portName = new QName("yahoo_port",nsURI.toString());
Service s = Service.create(serviceName);
URI address = new URI("http", null, "api.search.yahoo.com", 80,
    "/NewsSearchService/V1/newsSearch",
    "appid=jaxws_restful_sample&type=all&results=10&sort=date&language=en&query=java",
    null);
s.addPort(portName, HTTPBinding.HTTP_BINDING, address.toString());

Note that the URI constructed above contains a set of input parameters to the search service. The appid parameter identifies the application to Yahoo, if you are writing a new application you should register a new ID for it here. In this instance we are creating a URI that will search for news about Java and return at most 10 English language results sorted by date. Note that its not necessary to create a new port each time you change the arguments, more on that at the end.

Once the port is created we can use the Service instance to create a Dispatch instance. Dispatch is a new JAX-WS interface designed with dynamic, document-oriented services in mind. Note that it is a generic interface, in this instance we create a Dispatch<Source> instance since we are working with simple XML documents. You can also create Dispatch<SOAPMessage> instances when exchanging SOAP-based messages and Dispatch<DataHandler> for working with arbitrary types of data.

Dispatch<Source> d = s.createDispatch(portName, Source.class, Service.Mode.PAYLOAD);
Map<String, Object> requestContext = d.getRequestContext();
requestContext.put(MessageContext.HTTP_REQUEST_METHOD, new String("GET"));

Now that we have a Dispatch instance we can send the search request and retrieve the results. In this case we retrieve the results and transform them into a DOM tree.

Source result = d.invoke(null);
DOMResult domResult = new DOMResult();
Transformer trans = TransformerFactory.newInstance().newTransformer();
trans.transform(result, domResult);

Now we have the results as a DOM tree we can apply XPath queries to extract the data we are interested in. Here we first use an XPath query to obtain the number of search results returned and then use that knowledge to interate through the results using additional queries.

XPathFactory xpf = XPathFactory.newInstance();
XPath xp = xpf.newXPath();
xp.setNamespaceContext(new NSResolver("yn", nsURI.toString()));
NodeList resultList = (NodeList)xp.evaluate("/yn:ResultSet/yn:Result", domResult.getNode(),
    XPathConstants.NODESET);
int len = resultList.getLength();
for (int i=1;i<=len;i++) {
    String title = xp.evaluate("/yn:ResultSet/yn:Result["+i+"]/yn:Title", domResult.getNode());
    String click = xp.evaluate("/yn:ResultSet/yn:Result["+i+"]/yn:ClickUrl", domResult.getNode());
    System.out.printf("[%d] %s (%s)\n",i,title,click);
}

The NSResolver class in the above is a custom class based on an example I found here. It simply maps the yn namespace prefix used in the XPath queries to the Yahoo namespace URI (urn:yahoo:yn) used in the result XML document.

I mentioned above that there's no need to create a new JAX-WS port when the input data changes, the following lines submit a new query using the existing objects we've already created.

requestContext.put(MessageContext.QUERY_STRING,
    "appid=jaxws_restful_sample&type=all&results=10&sort=date&language=en&query=solaris");
result = d.invoke(null);

That's it for now, in future blogs I'll demonstrate how to use JAXB instead of the XPath APIs to get strongly typed access to the results and how WADL can be used to generate JAX-WS based stubs for RESTful services.

WADL Revision

I got some useful feedback following the publication of the original version of the Web Application Description Language (WADL) and have revised the specification accordingly. Thanks particularly to Mark Nottingham and Philippe Le Hegaret for their useful comments, many of which are addressed in the latest version of the WADL specification (PDF) and WADL schema

Changes since the previous version:

• Resources may now be nested. This allows the resources to be specified in a tree that closely parallels typical URI structures
• I've added global operation groups as a lexical shortcut to specifying the same set of operations for multiple resources. This isn't quite a resource typing capability as requested by Mark but provides much the same functionality (IMO)
• URIs are now used to refer to WADL components instead of XML QNames
• Clarified escaping requirements for URIs with embedded parameters
• Assorted other clarifications and additional explanatory text

Still to do:

• Specify the extensibility model including scope and inheritance
• Investigate registering a media type for WADL documents

The W3C has just created a mailing list for discussion of Web description, if you have comments or thoughts about WADL or similarly targetted description languages then why not join the list and share your ideas.

Introducing WADL

Seems like lots of folks are either doing it or talking about it (publishing proposals for alternatives to WSDL that is) so here's mine: Web Application Description Language (PDF) or WADL for short.

A number of Web based enterprises (Google, Yahoo, Amazon, Flickr to name but a few) are developing XML/HTTP based applications/services that provide access to their internal data. Anecdotal evidence suggests there's a lot of simple XML/HTTP integration going on within other companies too. Add to this the recent interest in AJaX and you're left with a need for a simple way to describe such applications and services.

WADL is designed to provide a simple alternative to WSDL for use with XML/HTTP Web applications. To date such applications have been mainly described using a combination of textual description and XML schema, WADL aims to provide a machine process-able description of such applications in a simpler format than is possible using WSDL.

Here's an example WADL description of the Yahoo news search application to illustrate the language:

01 <?xml version="1.0" standalone="yes"?>
02 <application targetNamespace="urn:yahoo:yn"
03  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
04  xmlns:xsd="http://www.w3.org/2001/XMLSchema"
05  xmlns:yn="urn:yahoo:yn"
06  xmlns:tns="urn:yahoo:yn"
07  xmlns:ya="urn:yahoo:api"
08  xmlns="http://research.sun.com/wadl">
09    
10  <types>
11    <include 
12      href="http://.../NewsSearchService/V1/NewsSearchResponse.xsd"/>
13    <include
14      href="http://.../Api/V1/error.xsd"/>
15  </types>
16    
17  <resources>
18    <resource uri="http://.../NewsSearchService/V1/newsSearch">
19      <operationRef ref="tns:NewsSearch"/>
20    </resource>
21  </resources>
22    
23  <operation name="NewsSearch" method="get">
24    <request>
25      <parameter name="appid" type="xsd:string" required="true"/>
26      <parameter name="query" type="xsd:string" required="true"/>
27      <parameter name="type" type="xsd:string"/>
28      <parameter name="results" type="xsd:int"/>
29      <parameter name="start" type="xsd:int"/>
30      <parameter name="sort" type="xsd:string"/>
31      <parameter name="language" type="xsd:string"/>
32    </request>
33    <response>
34      <representation mediaType="text/xml" element="yn:ResultSet">
35        <parameter name="totalResults"
36          type="xsd:nonNegativeInteger"
37          path="/ResultSet/@totalResultsAvailable"/>
38        <parameter name="resultsReturned"
39          type="xsd:nonNegativeInteger"
40          path="/ResultSet/@totalResultsReturned"/>
41        <parameter name="resultPosition"
42          type="xsd:nonNegativeInteger"
43          path="/ResultSet/@firstResultPosition"/>
44        <parameter name="results" path="/ResultSet/Result"/>
45      </representation>
46      <fault name="SearchError" status="400"
47        mediaType="text/xml" element="ya:Error">
48        <parameter name="msg" path="/Error/Message"
49          type="xsd:string"/>
50      </fault>
51    </response>
52  </operation>
53</application>

Lines 2-8 begin an application description and define the XML namespaces used elsewhere in the service description. Lines 10-15 define the XML schemas used by the service, in this case two schemas are included by reference. Lines 17-21 describe the Yahoo News Search Web resource and the operations it supports. Lines 23-52 describe the NewsSearch operation: lines 24-32 describe the inputs to the operation; lines 33-51 describe the possible outputs of the operation.

The WADL specification (PDF) and WADL schema describe the features of the language in detail but a few points are worth highlighting:

• Generative URIs are handled by including support for parameterization of URI components.
• The base set of HTTP methods (GET, POST, PUT, DELETE, HEAD) are specifically supported by the WADL schema but the method enumeration is open to use with other methods such as those specified by WebDAV.
• Representation parameters are hints to processors that point out interesting parts of a resource representation. As such they may be used or ignored as desired. I think they'll be useful for RPC-like interactions but less so for more document oriented work.
• Sibling representation elements represent alternative representations of the same resource. This means you can describe a resource that is offered in alternate formats, e.g. XML or HTML.
• The design center is XML/HTTP but this doesn't preclude use with alternate representation formats: the representation/@mediaType attribute provides the necessary hook.

I'm currently working on a WADL to Java compiler and will report back as things progress.

Attachments for SOAP

This week has seen a bumper crop of specification releases in the area of SOAP attachments. On Tuesday WS-I released the final version of Attachments Profile 1.0 along with the associated Basic Profile 1.1 and Simple SOAP Binding Profile 1.0. On Thursday W3C released Candidate Recommendations of XML-binary Optimized Packaging (XOP), SOAP Message Transmission Optimization Mechanism (MTOM) and the Resource Representation SOAP Header Block.

So, you may be asking yourself, what are the differences between the WS-I and W3C approaches and do I need to worry about them ?

• The WS-I approach assumes that there is a SOAP envelope and one or more associated attachments. E.g. one could envision an insurance claim that consists of a SOAP envelope containing an XML based insurance claim in the SOAP body plus a photo of the crash scene as a JPEG attachment. The WS-I attachment profile defines how to use the WSDL MIME binding to describe the presence of attachments, the details of how to use MIME to create a package of a SOAP envelope and its associated attachments, and a standard mechanism to reference attachments from within the SOAP envelope.
• In contrast, the central conceit of the W3C approach is that there is no such thing as an attachment to a SOAP message but instead all of the data of interest is contained within the infoset of the SOAP Envelope itself. Using the example above this might entail a SOAP envelope containing an XML based insurance claim in the SOAP body with an embedded photo of the crash scene. Unfortunately XML cannot directly contain binary data so the photo would need to be encoded using base64 prior to embedding. MTOM/XOP describe how to serialize such a SOAP envelope without incurring the penalty of base64 encoding by extracting those parts of the message that would require base64 encoding (i.e. the photo) and instead serializing them as separate parts in a MIME package.

The serialized form of the two approaches is remarkably similar (there's only so many ways to boil an egg) but processing semantics differ. E.g. if I use XML signature to sign the insurance claim in the WS-I approach then I would actually be signing the reference to the photo rather than the photo bytes (WS-I is working on a solution to this in the Basic Security Working Group), if I do the same in the W3C approach then I would sign the base64 serialization of the photo (at the cost of having to do the base64 encoding that XOP/MTOM would otherwise avoid after all).

To answer the second part of my question (do I need to worry about the differences): the JAX-RPC 2.0 (JSR 224) and JAXB 2.0 (JSR 222) Expert Groups are working hard to make sure that use of either mechanism is as transparent to Java developers as possible. These two specifications will support use of either approach and will hide as much of the complicated plumbing and conceptual doublethink as is practical. I expect the next drafts of both specifications to address attachments support.

Java Web Services Developer Pack 1.4 Released

The latest version of the Java Web Services Developer Pack (version 1.4) is now available and offers support for both the WS-I Basic Profile 1.1 and WS-I Attachments Profile 1.0. Support for the WS-I attachments profile now allows interoperable description and exchange of SOAP messages that include attachments based on the WSDL 1.1 MIME binding and the SOAP Messages With Attachments specification. In addition, the Java WSDP 1.4 also includes a full implementation of the OASIS Web Services Security (WSS) standards that provide message-level security for SOAP. This allows messages to be sent and stored securely independent of transport or storage security measures.

The Java WSDP 1.4 provides developer choice and flexibility by supporting deployement on the following containers: the Sun Java System Application Server Platform Edition 8, the Sun Java System Web Server 6.1, and Tomcat 5.0 for Java WSDP 1.4. The Java WSDP 1.4 and supported Web containers are all available for free download.

JAX-RPC 2.0 Early Draft

It looks like I've already been scooped by Eduardo but just in case you missed that, the first early draft of JAX-RPC 2.0 is now available for review. It addresses the following goals and requirements:

• Integration with JAXB
• Addition of client side asynchrony
• Improved support for document and message centric usage
• Improvements to the handler framework
• Default alignment with the WS-I Basic Profile
• Improved protocol neutrality

Of these, integration wih JAXB is the most significant departure from previous versions of JAX-RPC. Earlier versions defined their own mapping between XML and Java data structures but in JAX-RPC 2.0 all data binding functionality is delegated to JAXB 2.0 while JAX-RPC still handles mapping between Java and WSDL constructs. This separation of responsibilities allows a developer to resuse code between JAXB and JAX-RPC based applications or to easily extend an existing JAXB based application to use JAX-RPC facilities. It also provides JAX-RPC with full support for XML Schema where previously only a subset was required to be supported.

Other significant additions inlcude a new document-centric API for dynamic Web service invocation, support for dynamic and type-safe client side asynchrony and several improvements to the handler framework including protocol agnostic handlers.

This early draft is still a work in progress, subsequent versions of the specification will address the following additional goals and requirements:

• Support for WS-I Basic Profile 1.1 and Attachments Profile 1.0
• Support for SOAP 1.2
• Support for WSDL 2.0
• Versioning and evolution of web services
• Web services security
• Integration with JSR 181 (Web Services Metadata)
• Service endpoint model
• Runtime services

The expert group solicits your input on the current draft, details of where to send comments are included in the specification.

W3C charters new XML Binary Characterization Working Group

Following the successful W3C Workshop on Binary Interchange of XML Information Item Sets hosted by Sun Microsystems last year, the W3C has now chartered a new XML Binary Characterization Working Group to further investigate this fascinating area.

From the WG home page: "The XML Binary Characterization Working Group is tasked with gathering information about uses cases where the overhead of generat