Before we get into the source code, I want to point it out again that the entire trick bases on HTTP/1.1 Chunked Encoding at http://www.ietf.org/rfc/rfc2616.txt

A successful use also relies on that all servers between your applications fully comply with the Chunked Encoding and do not cache either request or response. These servers may be proxy, firewall, load balancer, http server, servlet container, etc. However today's most server products support HTTP/1.1 very well. And usually they do not cache for POST. A POST may contain a large file in MB even GB.

Lets get started by downloading the source code at http://weblogs.java.net/blog/rexyoung/archive/storage/full-duplex-http-src.zip Unfortunately there are zero documentation with the source code. My apology. I will try to explain the design ideas and how to use it in details here. Feel free to raise a question in the comments, or send me an email (I see old friends in the comments. Thanks for your support.:)

Development environment

I developed and tested these codes using Eclipse 3.3.2 and Tomcat 6 on a desktop. Connections are local and do not go through any proxy or http server. The Tomcat takes HTTP requests directly.

NetBean would be nicer as it has a feature that captures all HTTP traffic for later analysis.

In order to user Apache Commons HttpClient, you need three apache commons projects: commons-httpclient-3.1.jar, commons-logging-1.1.1.jar, and commons-codec-1.3.jar.

There are two packages port package and demo package. As the name suggested, port package contains the main body of the project and you will find sample user applications in the demo package.

Chunked encoding classes

The first part in the port package contains two classes that handle Chunked Encoding for input and output Http11ChunkedInputStream and Http11ChunkedOutputStream.

They are not required when running in a servlet container which handles Chunked Encoding for all servlets transparently.

These two stream wrappers are designed with a soft close approach which just release the underlying stream without closing it. This feature is mandatory for a reusable connection. Tomcat reuses a connection for multiple pairs of request and response. There is a demo for reusable connection.

Port hierarchy

public interface Port {
    InputStream   getInputStream();
    OutputStream  getOutputStream();
    void          close();
    boolean       closed();
}

Port is a very simple interface. It defines two getters as a storage function, and a close (a soft close as well) as a clean up function. It leaves construction and customization functions to implementation that varies significantly.

DefaultPort is a default implementation to the Port interface and a starting point to other solid implementations. It recognizes our own chunked streams and will invoke their soft close() method. This is very important. The chunked encoding requires a zero-sized chunk as the indicator to the receiver on the other side that the chunked encoding is finished and the underlying stream may be reused for other protocols and encodings.

PortViaTomcatServlet, PortViaHttp11Server, PortViaHttpClient31, and PortViaHttp11Client are four solid Port implementations. There are something in common among them. First, they all establish a connection in a constructor and do not have other customizations. Second, they do not have their own execution power, namely threads. Execution power comes from their user/caller. This design makes user application being flexible with different thread models. Thus developer has freedom on this subject.

Handshaking protocol

We have to talk about handshaking protocol before looking into those solid ports. Once a pair of ports are connected, they just know this is a HTTP POST connection with chunked encoding. This could be a regular POST. If we do not detect whether it is the right party you are talking to at the beginning, it will slip into data reading part and make programming more difficult.

So I decided to do a handshaking right after a connection is established. This way, if any error, it would be treated as any other connection error. We only need one set of code to handle all connection errors.

The handshaking protocol is very simple. Please take a look at PortUtil class. It is a C style zero terminated string. The string has a text part followed by a number part. Text can be used as the identification of your application and the number may be seen a version.

PortViaTomcatServlet & DemoTomcatServlet

This port is going to run in the doPost() method of a servlet. The connection is already established when the execution enters the doPost() method. The httpServletRequest.getInputStream() supplied by Tomcat is already chunked encoding ready. But the httpServletResponse.getOutputStream() is not.

According to Servlet specification (2.4 I just checked,) developer must tell Tomcat to make the output stream chunked encoding ready by calling httpServletResponse.addHeader("Transfer-Encoding", "chunked"). The tricky part is that you can not use your own chunked stream. Because Tomcat is going to assume that you are sending a known amount of data if the Transfer-Encoding header is not presented. It then check the Content-Length header instead. And if not found, it caches your output!!

As the connection is up, the constructor only performs a handshaking.

Note: For Oracle Container 4 J2EE, it does not handle Chunked Encoding for either input and output. In this case, you still need add the Transfer-Encoding header as HTTP/1.1 required. Also you need wrap container's both input and output stream with our own chunked stream wrappers. However, this is just my guess. I did not do a real test on this.

PortViaHttp11Server & DemoHttp11Server

This server side port assumes a HTTP connection is already established, or it is a reused connection (Tomcat reuses connection too.) So it starts from receiving both raw input and output streams.

Since this port is not behind any container like a Tomcat, it needs to handle HTTP protocol from the raw streams. That is why its constructor parses the incoming HTTP request to see whether it is a POST with chunked encoding. If yes, it reply with a HTTP/1.1 200 response with a chunked message body. After this point, it is the same like the Tomcat port. It continues our own handshaking protocol.

Obviously this port uses our own chunked streams.

The demo creates a thread for a new incoming socket. It also accepts multiple full-duplex-http sessions over a single socket connection.

PortViaHttpClient31 & DemoHttpClient31

This client side port works with apache commons HttpClient 3.1. It should also work with other 3.x version. The original PostMethod class is designed to be used with one request one response model. It would not start to check response without completely sent out the request.

The private PortPostMethod class is created to replace the sequential one. It sends out HTTP headers and our handshaking string using the writeRequestBody method, then immediately check HTTP response headers and handshaking string using the readResponseBody method. Both methods do not touch either the request message body or the response message body. This way, the streams to the message body are passed to the port in our hands.

By the way, the ChunkedEntityPlaceHolder.getContentLength() method returns -1. This would make HttpClient 3.1 believe Chunked Encoding is required.

HttpClient 3.1 allows HTTP connection reuse. I have not figured out how to take advantage of it. I think it is not important (our full-duplex connection is intend to last for long time) and the cost is too high. The cost refers to development time, as it needs check many of source code.

PortViaHttp11Client & DemoHttp11Client

This client side port is similar to PortViaHttp11Server. It is the exactly same idea behind them. The only difference is one receiving HTTP request and another issuing HTTP request.

The demo code shows how to reuse the socket connection for multiple full-duplex-http sessions.

P.S. Proper closing port is important

Usually a full-duplex communication involves multi-threading. It is very important to close all streams and port before move on. For example, you have to wait until all closing is done properly and then leave the doPost() method. According to Servlet specification, a container should consider all input and output data are finished once the execution is out of the scope of the doPost() (or any doXXX) method. If your other thread is still working on either input or output while the servlet thread leaves the doPost(), Tomcat will mess up your data (its data as well, ) and produce garbage over communication. No need to say, reuse is not possible in this case.

P.P.S. Idle causes timeout

Idle will cause timeout. We all know that. It happens faster with a servlet container, say 30 seconds. So you need a mechanism to send heartbeat-kind information to the other side when the connection is idle to keep it alive. This may cause large changes to the existing code you just downloaded. All classes are not thread safe. They are designed to be used by one thread a time. When adding another timer-like thread to send heartbeat, it will impact the code and require the code to be thread safe to a certain degree. Anyway, this is not a difficult task.

I developed the code mainly for the purpose of demonstrating and proving my idea how to use HTTP for full-duplex (or half-duplex if the performance is not your concern but convenience is.) communication. It is too tiny to be a project, also it is difficult to be small and to fit all at the same time.

So I think the best approach to use it is to include its source code to your project and customize it anyway you want. The license I picked for it is MIT. It basically means what you see is what you got, and do whatever you want. :)