I'm getting a lot of questions about NIO in general, and since most of them apply not only to HTTP handling, I've decided to blog about my experience with NIO in Grizzly. The observations I've measured might not apply to all NIO based servers implementation, but I suspect it will cover the majority of them. Anyway there is not much documentations about NIO in general (except basic tutorial), so it might not hurt to blog about it, whatever I'm right or wrong. But first, I recommend reading an NIO tutorial (if you don't know what NIO is) before reading this blog :-).

When building a scalable NIO server, you always have to handle three important NIO operation set bit:

• OP_ACCEPT: Operation-set bit for socket-accept operations
• OP_READ: Operation-set bit for read operations
• OP_WRITE: Operation-set bit for write operations

Handling OP_ACCEPT and OP_READ has been well documented in several NIO tutorial. Strangely, the OP_WRITE is sometimes not described at all. Not handling the OP_WRITE correctly can make your server performance pretty bad, and on win32, can produce disastrous performance problem, like freezing the OS by eating all the CPU. How come? Well, let start with an example. Usually, you will write to a SocketChannel by doing:

     while ( bb.hasRemaining() ) {
        int len = socketChannel.write(bb);
        if (len < 0){
           throw new EOFException();
        } 
     }

This code will works most of the time....until the Selector on which the SocketChannel has been registered is exhausted, e.g the Selector isn't able to let the socketChannel flush the content of the ByteBuffer. , which means:

     while ( bb.hasRemaining() ) {
        // *** socketChannel will always return 0
        int len = socketChannel.write(bb);       
        if (len < 0){
           throw new EOFException();
        } 
     }

Hence the CPU will be consumed by looping over and over, producing disastrous performance problem (try it in win32 :-)). OK, but what can we do? There is several ways of handling this. In GlassFish, Grizzly uses a pool of temporary Selector to register the SocketChannel on it:

        try {
            while ( bb.hasRemaining() ) {
                int len = socketChannel.write(bb);
                attempts++;
                if (len < 0){
                    throw new EOFException();
                }

                if (len == 0) {
                    if ( writeSelector == null ){
                        writeSelector = SelectorFactory.getSelector();
                        if ( writeSelector == null){
                            // Continue using the main one.
                            continue;
                        }
                    }

                    key = socketChannel.register(writeSelector, key.OP_WRITE);

                    if (writeSelector.select(30 * 1000) == 0) {
                        if (attempts > 2)
                            throw new IOException("Client disconnected");
                    } else {
                        attempts--;
                    }
                } else {
                    attempts = 0;
                }
            }
        } finally {
            if (key != null) {
                key.cancel();
                key = null;
            }

            if ( writeSelector != null ) {
                // Flush the key.
                writeSelector.selectNow();
                SelectorFactory.returnSelector(writeSelector);
            }
        }

If the main Selector is exhausted, a temporary Selector will be used to handle the OP_WRITE. In Grizzly, since the OP_READ also use the pool of Selector, the pool might return null. In that case, the main Selector will be re-used instead, like the Mina framework is doing. The Jetty Web server seems to create a temporary Selector lazily (I don't know Jetty enough to know the life cycle of SocketChannelOutputStream, but I suspect this object is recycled amongst requests so there is not an infinite Selector creation). EmberIO doesn't seems to handle OP_WRITE at all, which is surprising knowing the popularity of this framework.

Another alternative is to use a ThreadLocal to store a temporary Selector. Unfortunately the benchmarks I did demonstrated that this approach is slower that using temporary pool of Selector.

That's it. Next time I will discuss the evil method SelectionKey.attach() gold candidate for memory leak. technorati: grizzly nio glassfish