Benchmarks and (less) surprises (?)
As a follow-up of my
post, I've cleaned up my benchmark code (the log files have
been reduced in size from 40MB to 800kB, some bugs have been fixed and
the UUID strategy has been chosen for generating primary keys in the
database, thus reducing contention chances with multiple threads).
At this point the impact of logging should be neglectable, thus the
comparison makes more sense (if you haven't read my previous post, the
test reads metadata from about 150+ photos and insert them into a Derby
database, with both single- and multi-thread approaches; the test is
run with different software combinations but always on the same
hardware, my MacBook Pro first generation, 2GHz, 2GB RAM; numbers are
seconds per photo, the lower the better).
Workers Linux Ubuntu 8.04 Windows XP Mac OS X 10.5
Java 1.6.0_06 Java 1.5.0_15 Java 1.6.0_05 Java 1.5.0_15 SoyLatte 1.0.2 Java 1.5.0_13
1 0.50/0.35 0.55/0.47 0.48/0.37 0.52/0.47 0.80/0.66 0.80/0.74
2 0.40/0.26 0.42/0.42 0.32/0.25 0.34/0.33 0.49/0.43 0.47/0.50
3 0.39/0.26 0.41/0.42 0.32/0.25 0.43/0.39 0.48/0.43 0.52/0.53
4 0.39/0.25 0.41/0.45 0.31/0.26 0.45/0.41 0.49/0.43 0.53/0.55
In this round of tests I introduced a double benchmarking for every
platform, by using both the "Client" JIT (java -client) and the
"Server" one (java -server). Here are some points:
- I discovered my ignorance in a basic aspect of the
configuration of the VM. While I presumed that by default the "Client"
JIT was always selected, I learned that this isn't necessarily true
since Java 5. href="http://java.sun.com/docs/hotspot/gc5.0/ergo5.html">This
document clearly states that when running in machines defined
as "server-class" (at least 2 processors/cores and 2GB of RAM, and my
MacBook Pro falls within this class) the default JIT is the Server
- In spite of that, the Server JIT is selected by
default only on Linux. Window XP and Mac OS X still select the "Client"
one. While we are somewhat used to Apple's policy of doing things
differently, Windows is a bit of a surprise because it's another
- This explains a big deal of the differences seen in my
previous round of benchmarks: Linux resulted the winner since it was
the only one running with the Server JIT, which is clearly the best
option since the benchmark doesn not use any guy, but does repetitive
So at least part of the surprise in the previous set of numbers has
gone. But several new questions arise:
- Comparing Java 5 VMs omogeneously (Client JIT vs Client JIT
and Server JIT vs server JIT) Mac OS X still has the worse performance.
- Turning the Server JIT on Mac OS X and other operating systems does almost no good:
small improvements with a single processor, and no improvements at all
with multiple processors (indeed, it also looks like there's a slight
degradation of performance in some cases).
- After the verbose logging went off, SoyLatte Client JIT is
no better than Apple's JVM. Server JIT for SoyLatte is marginally
better, but much slower than Linux or Windows.
At first sight, one could argue that the Server JIT does not bring advantages on any operating system, but the bad performance of SoyLatte is raising
doubts about Mac OS X. Is there anything specific in Mac OS X that makes things go
worse (again, maybe the file system performance)? Or, given that
SoyLatte is still young, a lot of optimizations have been disabled?
The instructions to reproduce this are in my href="http://weblogs.java.net/blog/fabriziogiudici/">previous
post, this time you must use -r 5368 for checking out the
In any case, at least
with Windows and Linux I'm reaching my performance goal, which is 0.20
seconds per photo. Indeed, considering some known issues of my code, I
should be able to make better than it.
*** Edited to add:
To make things easier to read, I'm adding a table with normalized
values. The percentages indicates how slow a test runs (e.g. 100%
means that it runs at half speed). Best performers are marked in green,
worse performes in red; also a specific comparison among Java 5 only
VMs is given. In each column pair, the "Client" JIT figure is at the
left side, the "Server" JIT figure at the right.