Skip to main content

JVM Language developpers: Your house is burning

Posted by forax on May 24, 2010 at 5:20 PM PDT

Dear JRuby, Jython, Rhino or Groovy developers, you should drop your own runtime codebase now and use the JSR 292 API instead. It's just an advice from a friend(*) that hacks its own JVM langage during its spare time, but you should really consider to build a new version of your favourite language on top of JSR 292 API.
* I am also one of the expert of the JSR 292 expert group.

Why JSR 292 API is important ?

Because it provides two killer features.

Method handles, that are safe function pointers with currying and bunch of calling argument transformations. For a runtime this means that a function can be a method handle to the interpreter and later if the function is called often the method handle can be replaced by another one that will reference a dynamically generated bytecode blob. Currying is also important because it permits to insert live objects directly into a bytecode blob.

The second feature is invokedynamic which let the language developer to specify the linking between an operation like +, * or a method call and the method handle that will be called. Moreover, the developer can install guards on runtime type and relink any call site if necessary. This allow to provide an approach that will scale with the complexity of the code. If the code is only call with integers, no need to test if the runtime has to call the + between to double. If later the code is used with double, call sites can be relinked with the corresponding operations. Moreover, a reified call site is a Java object so you can also use the invokedynamic infrastructure in the interpreter sharing the code that deal with dynamic calls between the interpreter and the runtime compiler.

The only big problem of JSR 292 API is that to work effectively, you have to use it everywhere. You can't use it partially (Charles this sentence is for you) without sacrificing performance somewhere. This means that to be 'JSR 292 compatible', your runtime will have to be totally rewritten. I really think that it's time to seriously consider the version 2.0 (3.0 for Jython) of your runtime.

So if a runtime don't have to deal with method arity it can focus on other things. What about using runtime informations ?

Behave like a Trace Compiler

We all know how to optimize a JVM language. We have to provide as much types as possible to the VM and let the VM generates a nice optimized assembler from that. So our job is easy, we just have to create runtimes that record runtime type information during the execution and generates bytecode based on these information when the code becomes hot. In short, our runtime must behave like a kind of trace compiler.

Motivation

Let's use a simple numeric benchmark. I have stolen it from the JRuby bench directory, at that time, I was trying to see if my own pet language wasn't too far from JRuby and Jython that are known to be competitive with their counterparts written in C.
With the syntax of my language, I get something like that:

total = 0.0
x = 1.0
while(x <= 2000.0) {
  result = (5.4*math_pow(x, 5.0) - 3.211*math_pow(x, 4.0) + 100.3*math_pow(x, 2.0) - 100 +
    20*math_sin(x) - math_log(x)) * 20*math_exp(-x/100.3)
  total = total + result / 0.0001
  x = x + 0.0001
}
echo total

Here are the same file for JRuby 1.5.0, Jython 2.5.1, Rhino 1.7R2, Groovy 1.7.2 and Java. And no, this is not an error if the Javascript file and the Groovy one are identical :)
Update: Groovy uses BigDecimal by default (see comment from Guillaume Laforge), to use double instead you have to postfix all real numbers with 'd'. The updated file is here: Groovy 1.7.2 (d) and the corresponding bar in the chart is named Groovy(d). Guillaume, I am not sure that using BigD by default is a good idea.

Here are the execution time (using the unix command time):

execution times

I let you drawn your own conclusions.

If you want to reproduce the test my pet language is here. It currently only works with jdk7b94 binary.

Cheers,
Rémi

AttachmentSize
house-burning-bench.png15.25 KB
Related Topics >>

Comments

D'oh!

Unix "time" command for benchmarks? Without loops? No averages? Ramp-up time included? That's not serious. Next blog.

Re: D'oh

I want to test how some scripting languages run a specific script.
why should I add a loop (i.e modify my script) or don't care about the startup time ?

About the values. even if they were relatively stable, I have used a shell script that run the bench
ten times, remove the highest and lowest value and do an average on the remaining values.

Next comment.

This is called

This is called a Flawed Comparison. The scope, scale and domain of the PL being tested are not exactly the same.
Programming language implementations are compared against each other as though the designers intended them to be used for the exact same purpose - that just isn't so
It's an interesting exercise, but the conclusions you may take from it are obvious: BigDecimals are slow, but accurate, and Java Floating Point is fast, but dangerous (PDF).

Flawed Comparison ?

Come on, I compare JVM scripting languages not Lua vs Erlang.

Rémi

java.dyn

Perhaps you could comment on the following:

1) how stable is the API? - is it intended for inclusion into JDK >6?

2) presumably it is dependent on invokedynamic, so that would I believe make it JVM dependent, do you really expect DSL developers to constrain their current, or even next version RTs to that JVM?

3) perhaps you could directy demonstrate how such API might improve/simplify the development?

Thanks

Larry Cable

Re: java.dyn

Hi Larry,
the API is still moving, some method names or signatures may change but the key concepts: method handles/invokedynamic are frozen.

Yes it relies on method handle and invokedynamic. It's not JVM dependent, it depends if the VM of your choice implements JSR 292 or not.
I know that IBM and JRockit team currently work to release JSR 292 compatible VMs.

I have also implemented a backport of JSR 292 for 1.5/1.6 VM:
http://wiki.jvmlangsummit.com/JSR292Backport

How does JSR 292 simplify the development of language runtime is the tagline of my talk at next JVM Summit :)

Rémi

Too much math calls

I expect that most of the time is actually spent in trigonometry calls. You should see order of magnitude bigger difference between java and other languages here. Try implementing few of your own methods, doing some basic number crunching (vector dot product? matrix multiplication?) and do the calls on them - it should allow you to measure overhead of calls more explicitly. @shemnon So you claim that any effort to make dynamic languages faster is a waste? Just because you can implement performance critical section in pure java means that you don't care if you get twice speedup everywhere else FOR FREE, just by upgrading language runtime to something using invokedynamic/methodhandles? And if you want such speed, how do you want to measure it, if you will shoot down each benchmark as wrong, because it measures speed?

Re: Too much math calls

@abies, that's interresting.
Currently my runtime doesn't allow profile information to cross function call boundary,
it's not hard to do but I want to do it right.
Once implemented I will post a new entry with this kind of bench.

Rémi
 

Use doubles all the way for the Groovy sample

I think your Groovy sample is not totally correct. It's using a mix of BigDecimal and Double arithmetics which will slow down its result. So you're comparing apple and oranges here, as the other languages use doubles instead of BigDecimals. If you'd use doubles everywhere, you'd notice a huge difference in performance for the Groovy version. Could you please update the article with a proper Groovy sample and showing its result in the graph? Otherwise, I think it's very misleading here :-(

Re: Use doubles all the way for the Groovy sample

I've edited the groovy file to add 'd' everywhere and updated the post accordingly.
As mirko said, why groovy use BigDecimal by default ?

Rémi

A local variable vs a variable in the binding

By the way, there's also another thing that should be avoided inside Groovy scripts (unless you really need that ability to pass variables to a script) is to use variables from the binding, and instead, use local variable. When you do total = ... or x = ... to declare and assign a variable, it's actually adding a variable in the script's binding (way to pass data / values to a script a bit like global variables if you will), and it's not using a (faster) local variable like in the other languages demonstrated. So instead, using def total = ... or def x == ... or better double total = 3.14 (you don't need the d suffix then as Groovy will figure out it's a double and not a float). Just making those changes to avoid using the (slow) binding access and use normal (faster) local variables instead, makes the program even faster in Groovy (on my measures, by at least 30%!).

RE: A local variable vs a variable in the binding

 I don't agree.
A little more 'intelligent' compiler will be able to use local variables everywhere and creates bindings only when needed.
In my opinion, it's up to the runtime to do this optimization and not to the programmer.

Rémi

JSR-223 support

How does php.reboot handle JSR-223 bindings?

Re: JSR-223 support

It doesn't implement JSR-223 API :(
Feel free to contribute :)

But internally, the interpreter uses a compatible API.
see my answer to the post of Guillaume.

Rémi

Not about an intelligent compiler

It's not really about having an intelligent compiler or not. As a compiler won't necessarily know if a variable was or wasn't injecting in the binding of the script. This is more of an integration problem than anything else. The way a variable is defined in Groovy is with def or with a type, if you're not doing that, then Groovy tries to access that variable from the binding of the script instead.

Not about an intelligent compiler

Do you agree that you only set the value of a binding when the script starts and try to get the value at the end or perhaps each time you escape from Groovy by example when calling a Java method. A compiler can easily knows that thus use a local variable and copy its value to the binding object only when necessary.

BTW, this is exactly what the runtime of my language do when it just from a compiled code to the interpreter :)

Furthermore, here the variable is not bound to any binding. A runtime compiler can easily find this info if it starts by interpreting the code before compiling it.
As I said, I see that as a property of the runtime.

Rémi

In Groovy, you can change or

In Groovy, you can change or access the value from the binding even while the script is running. So a controlling thread may be able to influence (get / set) that variable binding (think of a progress bar that would update a UI element, etc). This is not necessarily the best approach to take, but this is something that people may be doing. The Groovy compiler know if the variable referenced is unbound or not, knowing it's a local variable or not, but it cannot say whether you added that variable to the binding or not. This is definitely a "runtime" thingy, of course. Perhaps we could indeed imagine some ways the compiler could introduce some local variables, or some atomic references, or something like that, but the benefit would only be for micro-benchmarks like this one, when someone accidentally use a variable binding instead of declaring a local variable.

Local variable or not ?

So binding variables can be modified by different threads.
but in the code groovy.lang.Binding doesn't offer garantee that a thread will see modification done by another thread ?

I agree with you. Knowing if a variable is bound or not is a runtime things.
And because you compile the whole script before runt it, you aren't able to optimize that case.
In my opinion, this is a limitation of the Groovy runtime.

If your runtime starts in interpreter mode before compiling things you don't have such problem.

Rémi

Knowing the language well is important

So a final remark, for micro-benchmarks, it's highly important to know how a language works to really have the best results in such tests. By using double arithmetics (like the other languages), and using local variables instead of binding variables, I get a 13x factor difference in my own tests. (the latter improvements wasn't taken into account in this last groovy(d) example)

Groovy's use of BigDecimal by default

It might be interesting to try this (or equivalent) in each of the languages:
assert 0.3 == 0.1 + 0.2
I think most will fail - some might fluke it with rounding happening to work but then another example will catch them out.

BigD by default

Using BigDecimal by default in Groovy is a very old decision we made early on, it'd be hard to change that now, and break all the (financial) applications leveraging that aspect. When you think about it for a second, why in Java 1.1 + 0.1 is not equal to 1.2? (1.20000...003). Isn't that surprising? What about things like 3 / 2 which equals 1? Ain't that surprising as well for business users? It can be surprising for such benchmarks to see that Groovy performs so badly... because it used BigDecimal, of course, if you don't know that those decimal literals create BigDecimals in Groovy. It seemed natural to have the good results to such addition puzzles as above, rather than playing nice in the benchmark games. And for business apps needing BigD support, I think 1.1 + 0.1 is nicer than new BigDecimal("1.1").add(new BigDecimal("0.1")). What's easier to read?

Where in the Groogy script is a BigDecimal created?

Hello, I do not see BigDecimal mentioned anywhere in the Groovy sample, is Groovy creating BD deliberately? Regards Mriko

Yes

Yes, Groovy creates BigDecimal numbers deliberately. This actually one of the reasons why lots of financial institutions chose Groovy as their language of choice, as this is required for acurate calculations. So 3.5 is a BigDecimal, whereas 3.5f is a float or 3.5d is a double. Also, when creating variables, if you say double constant = 3.14, you'll get the a double, although def constant = 3.14 will give you a BigDecimal.

driving nails with a screwdriver.

If you are using a dynamic language to do tight numeric loops, you deserve a slow application.
Each of these languages allows you to call a Java implemented version. So for that 0.1% of your application that crunches numbers, learn how to call a Java object directly (it's easier than you think).
However, when it comes to responding to mouse clicks or mapping a URL to a controller, the time you spend writing it will likely be longer than tha actual CPU time spent executing it.
Don't use a nail to measure a screwdriver's performance.

Re: driving nails with a screwdriver.

 > you deserve a slow application

Why ? As a user I want to use my dynamic language everywhere.
As a language developer, I don't see any technical evidence showing that a dynamic language must be slow when dealing with primitive types.

the time you spend writing
my runtime really acts like a JIT, I don't spend time writing something if it's not a hot path.

I don't care about screwdrivers, I want a swiss knife.

Rémi

swiss knife

> I don't care about screwdrivers, I want a swiss knife.
You see, that's just the problem. If you are going to build a house with a swiss knife it's going to look like a lean-to. If you want a lasting structure you are going to need more tools in your toolbox than what you can carry in your pocket on a jog around the neighborhood.

The gospel of one language for every purpose is vestige of the day when Java was the only viable language on the JVM. Put down the pocket knife and use some real tools.

Re: swiss knife

What all JVM (JVM: Just Virtual Magic) languages developers have is a magic wand.
Like any magic wands, if you cast the spell correctly you can even defeat Voldemor.

Ok, no more metaphoric comments for today..

Rémi