Next Generation Performance Benchmark
On last saturday I have run a few experimental benchmarks on the typical new generation technology stack (or part of it). What I exactly did was running iAnywhere 10.0.1 database and Sun Application Server 9 (aka "Glassfish" aka "Java EE 5 SDK") in a VMware Server 1.0.3 virtual machine on my private laptop (AMD Turion 64 X2, 2 GB RAM). The benchmark was done using a small test application that I wrote in less than one hour (after learning about Java EE 5) using EJB 3.0 and WebServices. There is one POJO (EJB 3 Entity) that is using AUTOINC in the DB as automatic PK generation, ontop of that a Session Beans is implementing a small WebService to allow my benchmark client to create and read rows using SOAP. Despite the fact that the code is much easier to read and write and no more DDs are needed thanks to annotations (which makes programming approximately two to three times faster -- plus removes a lot of potential bugs to fix later) it also seems to be much, much faster than our current technology stack. I was quite impressed after I noticed how fast even my laptop is, compared to the current solution:
Creation of one row: 27ms (including SOAP call, transaction start / stop, INSERT, SELECT @@identity).
Actually one could say, that is slow, since in a real world scenario we like to have about five times better performance, but hey, first of all this is a slow laptop and it is running a 4 GB VM on a 2 GB machine, second we really are doing SOAP, and third nobody would do a single transaction and server roundtrip for a single row (typically we will write several rows in one TX). So in fact I think that is an impressive demonstration that SOAP is not so slow that I always thought and that it actually is usable for real world applications. In fact, the main reason for the performance penalties was not java.exe or the dbsrv10.exe (a.k.a Appserver and DBServer) but the disk -- 50% of the time the two CPU kernels just waited for the hard disk, according to the constantly lighted HDD LED. I expect that a RAID system will serve at least 10 times faster, so I expect a server to report a value of 3 to 5ms per row (what we should try out some day with a real server hardware, just to get excited). Also it might be possible that using another ID generation strategy might be faster: AUTOINC needs to do one SELECT per row to fetch the generated ID. An ID table will be scanned only after e. g. 50 inserts (adjustable in Glassfish) so the roundtrip will be about 30 to 50% faster.
Reading of 1000 rows: between 300ms and 900ms (including SOAP call, transaction start / stop, SELECT *, transfer ALL columns to the client).
This was the test that actually impressed me a lot. With EJB 2.1 on JOnAS 4.8.4, loading 1000 rows down to the client was much slower. Between 300 and 900ms for 1000 rows means that the performance, even using a verbose and complex to parse XML based protocol like SOAP, is fast enough to load all the rows of a table in a single call / single transaction (for the average amount of data to be shown on a GUI).
Also we need to keep in mind that a real server will (a) not run on a laptop, (b) use 64 Bit database engine, (c) use a real operating system instead of XP Home Edition, (d) have lots of RAM for Caching, (e) will make use of a multi-disk RAID 5 or RAID 10 system that serves times faster than my noise- and battery-optimized 3 inch laptop drive.