Rémi Forax's Blog

Rémi Forax is Maitre de Conférence at University of Marne-la-Vallée since 2003 where he obtained his PhD on multi-method implemntation in Java. He has been using Java for many years and enjoys himself hacking the JDK.

Closure and groovy builder

One think i really like in Groovy, it's its concept of Builder.
It allows to simply create tree of objects like XML trees using a concise syntax.

An HTML tree in Groovy is defined like that

  html {
      head {
        title "hello groovy builder"
      }
  }

JavaFX uses a quite similar syntax but the syntax is built-in. The beauty of the groovy beast is that the builder syntax relies on closure.

Closure

The trick is the following:

1. An XML node is a method.
2. The content of a XML Node is a closure taken as argument of the method

Ok, so let's try to do the same in Java using the BGGA closure proposal. CICE is too verbose for that case and it should work seamlessly with FCM+ JCA

Proxiiiiiiiiii

But before, how to define all the allowed markups of an XML dialect ?
Ok, we need one method by markup, but because the code is identical for two different markups, instead of writing boilerplate code, I will use the class java.lang.reflect.Proxy.
To use the Proxy, we have to define an interface. The one below allow us to generate XHTML.

public interface XHTMLBuilder {
  public void html(Object textOrClosure);
  public void head(Object textOrClosure);
  public void title(String text);
  public void body(Object textOrClosure);
  public void h1(Object textOrClosure);
  public void br();
}

A Proxy acts as a multiplexer a call to any of its methods is redirected to a single generic method named invoke.

public class XMLBuilderFactory {
  final Appendable appendable;
  
  public XMLBuilderFactory(Appendable appendable) {
    this.appendable=appendable;
  }
  
  public void text(CharSequence seq) {
    try {
      appendable.append(seq);
    } catch (IOException e) {
      throw new RuntimeException(e);
    }
  }
  
  public <T> T createBuilder(Class<T> generatorInterface) {
    return generatorInterface.cast(Proxy.newProxyInstance(
      generatorInterface.getClassLoader(),
      new Class<?>[]{generatorInterface},
      new InvocationHandler() {
        public Object invoke(Object proxy, Method method, Object[] args)
            throws Throwable {
          
          ...
          return null;
        }
      }));
  }
}

Here appendable acts as a writable stream of characters, something on which I can append characters.
The method createBuilder takes an interface as argument and returns a class that implements that interface. All methods of thi interface are routed to the method invoke of the InvocationHandler.

Java Builder

Now, suppose that the method invoke is written, we will see after how to write it.
To write an XHTML tree on the standard ouput, using the BGGA closure syntax, the code is:

public class Main {
  public static void main(String[] args) {
    XMLBuilderFactory factory=new XMLBuilderFactory(System.out);
    XHTMLBuilder b=factory.createBuilder(XHTMLBuilder.class);
    
    b.html({=>
      b.head({=>
        b.title("hello Java 7 builder");
      });
      
      b.body({=>
        b.h1("Hello");
        factory.text("Greetings from");
        b.br();
        factory.text("Java 7 builder");
      });
    });
  }
}

It's even more readable if we use the control invocation syntax.

    b.html() {
      b.head() {
        b.title("hello Java 7 builder");
      }
      
      b.body() {
        b.h1("Hello");
        factory.text("Greetings from");
        b.br();
        factory.text("Java 7 builder");
      }
    }

This construct has several advantages. I can't produce ill formed XML, markup are opened and closed by the same code. Because the builder uses an interface I can't create markup with the wrong name. Even better, refactoring the markup name will works !

How to write the method invoke ?

For each argument of the method, i have to determine if it's a closure or any other objects. If it's a closure, i have to invoke it. If it's another object, i can append it into the Appendable.
The question is how to dynamically test if an object is or not a closure.
I don't want any closure i just want a closure that takes no parameter and returns void, a {=>void}.

  if (arg instanceof {=>void}) {
    (({=>void})arg).invoke();
  }

There is two poblem with that code. First, there are two kinds of closure (function type), unrestricted and restricted one. A restricted closure is a closure that can't break the control flow using break, continue or return. I am not sure i want my closure be able to break the control flow but the control invocation syntax creates unrestricted closure, so i need unrestricted closure. So instead of using '=>' in the function type, i have to use '==>'
By the way, i think the spec should be revised to swap the two notations ('=>' and '==>'), '=>' should be used for unrestricted closure. If i want a restricted closure which is not the default case, i have to take care about that and use the '==>' syntax. Append a second '=' is equivalent to say don't worry compiler i know what i'am doing.
So currently, the code should be:

  if (arg instanceof {==>void}) {
    (({==>void})arg).invoke();
  }

But there is another problem, currently, {==>void} is a parameterized type and so it's illegal to use it in an instanceof. Here, i have ask Neal, he says he planed to make "types such as this one non-generic".
There is a workaround, currently the closure prototype works with 1.4/1.5 VM and thus generates interfaces corresponding to function types. The interface corresponding to the raw type of {==>void} is javax.lang.function.unrestricted.V.
Here is the code that works:

  String name=method.getName();
  if (args==null || args.length==0) {
    appendable.append('<'+name+"/>");
    return null;
  }
          
  appendable.append('<'+name+'>');
  try {
    for(Object arg:args) {
      if (arg instanceof javax.lang.function.unrestricted.V) {
        ((javax.lang.function.unrestricted.V)arg).invoke();
      } else {
        appendable.append(arg.toString());
      }
    }
  } finally {
    appendable.append("');
  }

A zip containing the wole codes is here: closure-builder.zip
You need the prototype of the BGGA closure too.

Cheers,
Rémi

Da Vinci runtime properties

After a week without any internet access point surfing the snow of the Alp, monday, my fingers was eager to touch the keyboard again. Why not finishing my prototype of runtime properties that use the Da Vinci VM (i really love that name).

One ugly thing of the draft v3 of the property spec, is a property object is implemented by a supplementary class generated by the compiler. This means the compiler must create one class by property and so bloat the application with lot of stupid code.
Ok, let's try to do better. No class at compile time means reflection or runtime time generation. In fact, it's not a real choice because reflection since 1.4.1 generates byte-code when a method is often called.

Da Vinci VM

The Da Vinci VM is the prototype implementation of JSR 292, a modified hotspot VM patched with new entry points that help to implement dynamic languages on top of the Java platform.
The first feature available, anonymous class (VM anonymous class not compiler one) allows you to create classes with some interesting features:

1. The class is not referenced by a classloader, so is GCed when there is no more instances of that class
2. Class is like an inner-class of an host class and therefore can access to members of the host class.
3. In order to create the class you can patch an existing class.

How property object works

If you are not familiar with properties, you can read an old blog entry.
A property object stands for a property and permit access to the value of the property of any instance of a Class that declare that property.
By example, the following code:

public class Bean {
  private int x;
  public int getX() {
    return x;
  }
  public void setX(int x) {
    this.x = x;
  }
  
  private String text;
  public String getText() {
    return text;
  }
  public void setText(String text) {
    this.text = text;
  }
    
  public static void main(String[] args) {
      Bean bean = new Bean();
      Property propertyX =
          Property.create(Bean.class, "x");
      bean.setX(0xCAFEBABE);
      System.out.printf(propertyX + " %x\n", propertyX.getValue(bean));
      
      Property propertyText = Property.create(Bean.class, "text");
      propertyText.setValue(bean, "hello property");
      System.out.println(bean.getText());
  }
}

prints:

  int Bean.x cafebabe
  hello property

How to test it ?

If your OS is Windows, Solaris or MacOs you have to compile the VM by yourself, sorry. On linux, you can download the binary of the jdk7 b24 if you haven't already it, download the following zip davinci.zip and unzip it in your JAVA_HOME/jre/lib/i386/
It adds a new server JVM lib named 'davinci' that can be launched using

  java -davinci ...
 

Now download the property runtime support property-runtime.jar, because it adds new classes and overrides some existing classes to the JDK you must prepend the jar to the bootclasspath.
To launch the example above:

  java -davinci -Xbootclasspath/p:property.jar Bean

The source of the property runtime support are available on the kijaro web site here: https://kijaro.dev.java.net/source/browse/kijaro/branches/properties/.

What's the next step, finish to write a new version of the property spec and provide a modified compiler according to that spec. I think i've solved most on the corner cases, so it's just a matter of time.
To be continued...
Rémi

Sorry but i've toasted your pet

Introduction

The idea is simple. I want to create a calendar service, like the server part of Google Calendar. I want to create a server that is able to parse a specific protocol allowing to query calendars and send a response using by example the ical format.
This service can be queried by different languages so i've decided to use a text based protocol, like HTTP.
By example the request to get a calendar named mycalendar as user forax between November, the 6th and November, the 7th will be something like that.

GET mycalendar forax CAL/1.0
password: AjxHKRFkRwxx3j9lM2HMow==
from: Sun Nov  6 08:49:37 1994
to: Mon Nov  7 12:34:31 1994

More formally, requests can be described by the following grammar using the Tatoo EBNF form.
Tatoo EBNF form use sections:

• section tokens declares terminals and their regex.
• section blanks declares the regex whom the lexer will not send a terminal to the parser.
• section productions declares sequences of terminal and non terminal which will be reduced to a non terminal.

tokens:
  service='GET'
  uri= '([^ ])+'
  user= '([a-z][A-Z])+'
  protocol= 'CAL/1.0'
  colon= ':'
  header_key= '([^ :\r\n])+'
  header_value= '[^ \r\n]([^\r\n])+'
  eoln= '(\r)?\n'
 
blanks:
 space= "( |\t)+"

productions:
 start = request+
       ;
 request = firstline 'eoln' header* 'eoln'
         ;
 firstline = 'service' 'uri' 'user' 'protocol'
           ;
 header = 'header_key' 'colon' 'header_value' 'eoln'
        ;

Now Tatoo, a parser generator we have developed is able to create a non-blocking push lexer/parser for that grammar.
By non-blocking, i mean a parser that work on NIO buffers, and since christmas, Tatoo has a companion named banzai, a server using NIO that can embed a Tatoo parser generator.
Great, but how to specify the semantics, i.e how to specify that the password need to be checked, how the response is computed etc.

The semantics

The semantics is specified by creating a class that inherits from ProtocolHandler. A ProtocolHandler is an object called each time a terminal is recognized (shifted) or a production is reduced. Furthermore, it provides an object that own some methods like asyncWrite() to send back data to the client or endRequest() to end the request.

 public class CalendarProtocolHandler extends ProtocolHandler { 
  private String uri;
  private String username;
  private String header_key;
  private String header_value;
  private HashMap headerMap=...
  ...

  public void shift(RuleEnum rule,TerminalEnum terminal) {
    switch(terminal) {
      case uri:
        uri=decode();
        return;
      case user:
        user=decode();
        return;
      case header_key:
        header_key=decode();
        return;
      case header_value:
        header_value=decode();
        return;
    }
  }

  public void reduce(ProductionEnum production) {
    if (production==ProductionEnum.header) {
      headerMap.put(header_key,header_value);
      return;
    }
    if (production==ProductionEnum.request) {
      handle();
      return;
    }
  }
  
  private void handle() {
    try {
      String password=headerMap.get("password");
      boolean passwdOK=verifyPassword(username,password);
      if (!passwdOK) {
        putUnauthorizedResponse(outBuffer);
        analyzer.asyncWrite(outBuffer);
        return;
      }
      ...
      Calendar c=...
      putICalendar(outBuffer,c);
      analyzer.asyncWrite(outBuffer);
    
    } catch(IOException e) {
      analyzer.endRequest(e);
    }
  }
}

Short explanation of the code above: shift(), if a specific terminal is found, decode the buffer to find its value. reduce() if we reduce a header key/value pair, store it into a map, if we readuce a request, call handle(). handle() verify the password and write a response

Banzaï !

I know what you are thinking:
"Ok, it's interresting. But embeding a parser in a webserver will damage performance".
No, Tatoo parser is carefully designed to not hurt performance.
I know that you don't believe me. So let me try to convince you with a stupid benchmark(tm).
I have written a subset of the HTTP/1.1 grammar (a subset because the wole HTTP/1.1 grammar is huge and i'm lenient), generated the corresponding parser with Tatoo, written a ProtocolHandler corresponding to HTTP and beanchmark banzai (the server) with that protocol handler and comparing it with Grizzly and Jetty.

Benchmarks

So i've borrowed two DELLs (config) with ethernet Gigabit cards and a gigabit switch in my labs, plug them and play.
Servers are set up with a Gentoo (2.6.19-gentoo-r5) without major modifications. I have just raised the number of descriptors to 65535.
Because i want to do a stress test, i've used apache bench (ab) as client so this is not a real scenario, just a stupid benchmark(tm).
If you want to reproduce the test, checkout the code here svn checkout https://svnigm.univ-mlv.fr/svn/tatoo/trunk, compile Tatoo using ant.
Banzai is located in a sample directory named httpserver. use ant all compile to compile banzai, and to launch it using

 java -server -cp classes:../../lib/tatoo-runtime.jar fr.umlv.tatoo.samples.httpserver.banzai.Main

First test: how many requests can be handled by banzai

Serving files of 4k, 8k, 16k and 32k with different numbers of concurrent connections (8, 16, 32 etc.). The value is a mean over 25 runs of 50 000 requests.
First, when i have seen the result of the benchmark I was astonished. Wow, linux 2.6 and a Core Duo offers great perf, more than 25 000 requests by second for a 4k file.
Furthermore, it seems that banzai have a problem if there are lot of concurrent connections. Perhaps because banzai doesn't have a strategy like closing keepalive connections if there are lot of connections.
Now, a more interesting graphs.

Second test: comparing with the others

Serving a 4k with different concurrent connections
Yes, banzai performs slightly better than grizzly ...
at least if there are not lot of concurrent connections.
About Jetty, I think I screw up the conf because the slope of the curve is weird and I am not able to explain why.

What's next

I think it's possible to integrate non blocking parsers technology directly into grizzly.
Another idea is to wrap banzai to use RESTlet API.
To end, i have found small improvements to the OpenJDK and i will send patches that will benefit to anyone.

Cheers

CICE prototype available and FOSDEM

It's an old news but i've just discovered that Mark Mahieu provide an implementation of CICE closure proposal which is an aternative to BGGA prototype.

By the way, i will be at FOSDEM'08, if you want to meet me, i will try to attend to all Free Java Meetings.

I have decided to finish this entry a la Chris Campbell.
In my ears: Morcheeba, "Dive Deep"
In my eyes: Some cryptic codes like always.

Yet another closure proposal

Everybody comes with its own closure proposal, why not me :)

Unlike BGGA, CICE, or FCM (not entirely sure about FCM), i don't like the fact a closure is an instance of a class that implement a method. I prefer the John Rose's vision, closure are at runtime method handler on autonomous block of code.

There is another thing I don't like in the current BGGA proposal: its syntax of function type, {int,int=>int} is not enough Javaish for me. My proposal comes with a new way to declare a function type

I've separated my proposal in three parts:

1. function type declaration
2. how to use function type
3. closure syntax

Function types declaration

A function type is declared using the 'almost' keyword function.

// function types declaration
class Ops {

  // function are basically like typedef
  // top-level functions are illegal
  function <T> int comparator(T value1,T value2);

  // here function is not a real keyword, the compiler
  // recognize it as a keyword only in this construction
  function <E> void block(E element);

  // in the function below, E can be instantiated to any
  // object types or any primitive types,
  // it works because function are only declaration with no code. 

  // another function
  function <R> R task();
}

How to use function type

Function types are types so they can be used anywhere a type can be used except as bound of a type variable or type argument of a type variable.

// usage
import static Ops.*;

class Utils {
  // note: comparator<?> is illegal
  // (wildcard are prohibited for generics functions because
  // they are not needed).
  static <T> void sort(List<T> list,comparator<T> comparator) {
    ...
  }

  static <T> void forEach(Collection<T> c,block<T> block) {
    for(T t:c) {
      // invoke is a magic method that execute the block of code,
      // a special rule in the compiler is able to infer its
      // parameter type.
      block.invoke(t);
    }
  }
}

Closure syntax

Unlike function type syntax of BGGA, i like its closure syntax so basically my proposed closure syntax is identical. I have borrow method references from FCM even if we need to add a special lookup rule (because methods and fields are not in the same lookup space) for that in the compiler.
Like BGGA, my proposal include non local transfer and access to local variable (read and write) but only for closures and not for any anonymous classes, a closure is not an anonymous class. Like any proposal, my proposal allow interface conversion of function to interropt with legacy codes but only if closure use final local variable.
At last, return is not allowed in a closure to avoid stupid puzzlers but a replacement syntax exists.

// at calling site
class Main {
  static int f(Object o1,Object o2) {
    ...
  }

  void g(int value) {
    ...
  }

  public static void main(String[] args) {
    Ops.comparator<Object> c=Main.f;

    // using a primitive type here is legal,
    // it creates a block of code that call g()
    Ops.block<int> b=new Main().g; 

    List<String> l=Arrays.asList(args);
    Utils.sort(l,c); // ok contravariance

    // closure syntax, this syntax allow to specify the block of code
    Utils.sort(l,{Charsequence c1, CharSequence c2 =>
      c1.toString().compareTo(c1.toString());
    });

    // function to interface conversion
    java.util.Comparator<Object> javaUtilComparator=c;

    // it's roughly equivalent to
    final Ops.comparator<Object> tempc=c;
    java.util.Comparator<Object> javaUtilComparator=new java.util.Comparator<Object>() {
       public int compare(Object o1,Object o2) {
         return tempc.invoke(o1,o2);
       }
    };
    
    // closure to interface conversion
    java.util.Comparator<String> javaUtilComparator2={String s1, String s2 =>
      -s1.compareTo(s2);
    }); 
  }

Examples of non-local transfer and local variable mutation.

  // this is not allowed in contrast to BGGA.
  class NotAllowed implements Ops.block {
    ...
  }

  static int counter(Collection<Integer> c) {
    int count=0;
    // bloc of code (closure) can mutate local variable,
    // a function that returns an int is a subtype of a
    // function that returns void.
    Utils.forEach(c, {Integer i=>
      count+=i;
    });
    return count;
  }

  static boolean startsWith(Collection<String> c, String prefix) {
    // closure with non local tranfer,
    // moreover prefix doesn't need to be final
    // because this is not an interface conversion
    Utils.forEach(c, {String s=>
      if (s.startsWith(prefix) {
        return true; // doesn't compile ambiguous
        contains.return true; // ok
      } 
    });
  }
}

I wait your comments,
Rémi

April 2008

Search this blog:

Categories

Archives

April 2008
March 2008
February 2008
January 2008
November 2007
October 2007
September 2007
August 2007
June 2007
May 2007
March 2007
February 2007
January 2007
December 2006
November 2006
October 2006
September 2006
August 2006
July 2006

Recent Entries

Closure and groovy builder

Da Vinci runtime properties

Sorry but i've toasted your pet

This work is licensed under a Creative Commons License.