Benchmarks and surprises

I'm posting some results from my latest benchmarks on the Metadata facility of blueMarine - take them with half a pinch of salt, since I've checked them but not double-checked, sorry but I have just a little time in these days. They gave some little surprise to me.

In short, the test takes about 160 photos and imports all the metadata into a Derby database (details for reproducing it below). In total, 10000+ records are imported. The tests run with a classic Master / Worker pool with 1, 2, 3 and 4 workers on a MacBook Pro 2GHz, 2GB RAM in all the combinations of (Java 5, Java 6) x (Mac OS X 10.5, Linux Ubuntu 8.04, Windows XP) - note that I've used Soylatte on Java 6 since I don't have a 64 bit computer and can't run the Java 6 previews from Apple (thanks, Apple). The benchmark figure is the number of seconds needed to import a photo, so the lower the number, the better.

Linux		Windows XP		Mac OS X 10.5
Java 6	Java 5	Java 6	Java 5	Java 6 (*)	Java 5
0.48	0.73	0.60	0.70	0.83	1.00
0.37	0.58	0.39	0.46	0.56	0.63
0.38 - 1.13	0.58	0.39	0.52	0.60	0.68 - 1.04
0.36 - 0.76	0.60	0.48 - 0.78	0.56	0.61	1.06

(*) SoyLatte 1.0.2

Before commenting, keep in mind the following points:

1. The tests still run with logs at maximum power (each run produces 40MB of files), so the computation is more disk intensive than it should
2. There are some optimizations I have to do yet, for instance photos are read multiple times
3. Tests with multiple workers have a bug so about 1% of data don't get imported (there must be a failed transaction somewhere and it doesn't get properly logged), but this low number of errors can't change numbers dramatically.

So I expect to have even lower numbers after some fixes, but the comparison is fair across all the environments. Here my points:

1. As expected, there's no advantage in having more workers than available processors (2). In this case, tests have a huge variance, since some times they trigger a lot of contentions in the database (with a lock timing out for one second and causing the transaction to be re-run).
2. Java 6 is really faster than Java 5 and for free.
3. Mac OS X is clearly the loser here, even in comparison with Windows XP, especially with a single processor. Even Java 6 is not exceptional, but I presume the problem is related to the low performance of the file system, that I've already measured some time ago. 

Now I've got a simple question. Unfortunately I don't have a quad-core, so I can't test: is it expected that Derby scales well with four workers? Or should I put a limit to 2? When contentions on the database don't occur, it seems that even with only 2 processors a larger number of workers don't introduce significant degradation, which is promising; on the other hand, when contentions occur things are really worse. And the more workers, the more chances of having a contention.

To reproduce:

1. Check out sources with svn co https://bluemarine.dev.java.net/svn/bluemarine/trunk/src/blueMarine-core -r 5237
2. Check out test files with svn co https://imaging.dev.java.net/svn/imaging/trunk/www/TestSets -r 49
3. Open the project blueMarine-core with NetBeans 6.0.1 or NetBeans 6.1
4. Quit NetBeans just after opening the project
5. Run from the command line ant generate-platform nbms
6. Edit Metadata/MetadataOperations/nbproject/private/private.properties and add the line test-unit-sys-prop.testset.folder=path-to-TestSets
7. From the command line, go to Metadata/MetadataOperations
8. run ant test |& grep seconds