Kohsuke Kawaguchi's Blog

Introducing Dalma Project

Many applications of today need to have a part of the program that waits for other entities. For example, a mailing list manager waits for an e-mail to trigger actions. A web application waits for incoming HTTP requests.

Sometimes those applications are conversational; they not only just receive messages, but also send out messages. Those programs often maintain some state and change it as the conversation progresses. Often there are multiple conversations running concurrently. Those are what we call "workflow" applications.

For example, where I work, I have a little process every Monday. On Sunday night, cron triggers an automated weekly build of our software. Upon a successful build, it sends out an e-mail. On Monday morning, a manager and an engineer replies to those e-mails, authorizing the posting of the build to a publicly visible web server. Another program waits for those authorization e-mails, push the binaries to the server, and then report the result in an e-mail, concluding a conversation. This little ritual happens concurrently for each project in my group.

Today, those applications can be written, but one can't write it very productively. For one thing, every time an incoming message comes, the program has to retrieve the state information, act on it, and then store it back. This needs a lot of boilerplate code. You also need to write it as an event-driven program, where your application consists of a set of message handlers. This makes it difficult to follow the flow of the logic.

Dalma makes it very easy to write those workflow applications by letting you write it as an ordinary procedural program without any boilerplate. For example, consider the following program that subscribes an user to a mailing list by using a simple e-mail base confirmation:

EmailEndPoint eep = ...;

void subscribe(MailingList ml, InternetAddress userEmailAddress) {
  // ask the user to confirm subscription
  MimeMessage msg = new MimeMessage();
  msg.setText("are you sure you want to subscribe?");
  msg.setRecipient(TO,userEmailAddress);
  
  // send it and then wait for the reply
  msg = eep.waitForResponse(msg,1,WEEK);
  
  // if the user acknowledges ...
  if(msg!=null)
      // then subscribe
      ml.add(userEmailAddress);
  else
    // if no response is received in a week,
    // abort.
    ;

While this is very easy to write, traditionally, writing such a program like this would be considered a bad idea, because:

1. this requires one thread per one conversation, making it less scalable
2. if the JVM shuts down, all information about the on-going conversations will be lost.

Dalma solves those issues by allowing you to run them on a "virtual thread". A virtual thread is like an ordinary thread, except that it can be suspended, persisted, and resumed later, possibly in a different JVM. For those of you who have some experience in Scheme, this is similar to call/cc.

Inside the waitForResponse method, dalma suspends the virtual thread, and then persists its execution state to a disk. The actual Java thread that was used to run this virtual thread can be then reused to run other virtual threads. Later, when an e-mail is received, the persisted execution state is restored, and the virtual thread resumes its execution inside the waitForResponse method. The JVM can even shut down and restart between this.

This mechanism eliminates the needs of the explicit state persistence; those states are now captured in terms of local variables and call stacks. Dalma engine handles persistence transparently behind the scene. This mechanism also allows few threads to support much larger number of virtual threads, which is essential for supporting many concurrent conversations.

The call/cc part of the functionality is available as Apache Jakarta commons Javaflow project. The workflow engine part is available from the dalma project here at java.net. Both are still work in progress, but let me know what you think.

Comments

• very cool!

  Posted by: jhook on October 12, 2005 at 09:13 PM

• This is very nice... But instead of calling this virtual threading, you can call it "continuations". That is really what this is about and people will more easily associate this project with the concept (as the continuation concept has been getting some attention these past few months).
  

  Posted by: epirsch on October 13, 2005 at 06:27 AM

• Very cool indeed. I have an Apache James mailet that's in need of something like this.

  Posted by: markswanson on October 13, 2005 at 07:51 AM

• Yes, if you are a kind of person who knows "continuation", it's really similar in the concept. In the javaflow project, we use the term continuation without hesitation, but I thought it's still a foreign word for Java developers. (It's also somewhat weak use of continuation, as here we aren't executing the same continuation more than once, nor am I throwing away continuations.)

  
  But I guess I'll use it more then. Thanks for the feedback!
  

  Posted by: kohsuke on October 13, 2005 at 08:25 AM

• Is the above sample code javaflow enabled or do you have to program it in a different way? Does this work with all functions or do the caller of the event handler do something special (like restarting and catching and thread pooling).
  

  Posted by: ecki on October 13, 2005 at 09:39 PM

• sounds like a lot of overlap with JavaSpaces... have a look at it !

  Posted by: triadic on October 13, 2005 at 09:46 PM

• it needs to be instrumented by javaflow to run (which can be done either at the compile-time or at runtime.) The workflow program needs to be implemented as a Runnable object, and it will be invoked by the workflow engine. In that sense the code example isn't complete. The tutorial on the project webpage is more complete with this regard.

  Posted by: kohsuke on October 13, 2005 at 10:03 PM

• re: JavaSpace --- I've never heard of it before. I'm sure to find out!

  Posted by: kohsuke on October 13, 2005 at 10:04 PM

• this is very slick! i'll give it a spin and try out some new Endpoints.

  Posted by: gonzo on October 13, 2005 at 10:29 PM

• re: virtual threads vs continuations ... - I actually appreciate trying to make some distinction from full-fledged "continuations". As you say, this is a very weak use case, and full-fledged continuations have a number of implications that this does not have.

  
  Maybe you could say something about "(a very simple case of continuations)", and make everybody happy?
  

  Posted by: pelegri on October 21, 2005 at 10:36 AM

• I'm curious about the "implications" you mentioned, Eduardo.

  On top of what you suggested, I think I'll put a separate page in the website that explains how it's using javaflow behind the scene to make this happen. I can explain how it relates to continuation and etc there. The word "continuation" has some buzzword factor in it , so it's hard to resist the temtation not to use it at all.

  Posted by: kohsuke on October 21, 2005 at 11:41 AM

• Hi Kohsuke

  I tried a simple Javaflow sample based on ideas you have put together in the Dalma project. But I am failing miserably. I keep getting IllegalStateException: stack corruption stating that the MyRunnable class is not configured for javaflow. What am I doing wrong? Please advice.

  Here is what I am trying to do:

  BcelClassTransformer.debug = true;
  // creates a new class loader that loads test classes with javaflow enhancements
  URLClassLoader baseLoader = (URLClassLoader)TestRunner.class.getClassLoader();
  URL[] urls = baseLoader.getURLs();

  ClassLoader cl = new ContinuationClassLoader(
  baseLoader.getURLs(),
  new MaskingClassLoader(baseLoader));
  cl.loadClass(this.getClass().getName());

  Here is the MaskingClassLoader class:
  public class MaskingClassLoader extends ClassLoader
  {
  public MaskingClassLoader(ClassLoader parent)
  {
  super(parent);
  }

  protected synchronized Class loadClass(String name, boolean resolve) throws ClassNotFoundException
  {
  if (name.indexOf("test.") != -1)
  throw new ClassNotFoundException(name + " is masked");
  return super.loadClass(name, resolve);
  }

  }

  Thanks
  Avinash
  

  Posted by: avinashdha on October 19, 2007 at 09:19 AM

This work is licensed under a Creative Commons License.