BEA beats IBM in SPECjAppServer2002
After the debacle around the infamous ServerSide report, you could be forgiven for treating all benchmarks and performance comparisons with a pinch of a salt. All the vendors cited in that exercise were very quick to distant themselves from the whole escapade and tout they would only be interested in official benchmarks.
One such body that does attempt to define a standard is the Standard Performance Evaluation Corporation, that lives at http://www.spec.org/. They have a body of tests specifically targeted at Java App Servers, where I believe the tests themselves are all agreed by the major vendors as a true benchmark for testing.
One of the redeeming facets of the setup here, is that each vendor is invited to tune their own systems, so we the reader, can have faith that the best possible optimization/configuration is in place.
A recent set of tests submitted by HP prove interesting reading. This is for BEA WebLogic running with JRocket, on Redhat with Oracle as the database. The hardware, naturally is HP with a dual CPU Intel configuration. With this configuration they are pushing 694.88 total operations per second (TOPS). Okay, a fairly meaningless figure in its own right until you compare it to some other results. IBM, for example, could only manage 448.12 TOPS.
IBM is using WebSphere ontop of their own JVM running on Windows2000, with DB2 as their database. What is interesting to note is that IBM choses to conduct these tests on Windows. You would have thought (especially with their marketing push) that Linux would have been a contender, or even their own AIX OS. One wonders what they know that we don't! When you look at the total cost for each system, the configuration used by HP comes in at $182K where as the IBM setup is $290K! Not only is IBM slower in this instance, but also more expensive.
But before you get too excited, it is worth noting that to date, the only app servers to be submitted for public scrutiny is BEA and IBM. So where are all the other J2EE licensees?