A scalable SPECjAppServer 2004 submission
Last week, Sun published a new SPECjAppServer 2004 benchmark score: 8439.36 JOPS@Standard . [I'd have written about it sooner, but it was published late Wednesday, and I had to go home and bake a lot of pies.] This is a "big" number, and frankly, it's the one thing that's been missing in our repertoire of submissions. We'd previously shown leading performance on a single chip, but workloads in general (and SPECjAppServer 2004 in particular) don't scale linearly as you increase the load. This number shows that we can scale our appserver across multiple nodes and machines quite well.
I've been asked quite a lot about what scalability actually means for this workload, so let me talk about Java EE scalability for a little bit. The first question I'm invariably asked is, isn't this just a case of throwing lots more hardware at the problem? Clearly, at a certain level the answer is yes: you can't do more work without more hardware. And I don't want to minimize the importance of the amount of hardware that you throw at the problem. There are presently two published SPECjAppServer scores that are higher than ours: HP/Oracle have results of 9459.19 JOPS@Standard  and 10519.43 JOPS@Standard . Yet those results require 11 and 12 (respectively) appserver tier machines; our result uses only 6 appserver tier machines. More telling is that the database machine in our submission is a pretty beefy Sun Fire E6900 with 24 CPUs and 96GB of memory. Pretty beefy, that is, until you look at the HP/Oracle submissions that rely on 40 CPUs and 327GB of memory in two Superdome chasis. So yes, if you have millions (and I mean many millions -- ask your HP rep how much those two Superdomes will cost) of dollars to throw at the hardware, you can expect to get a quite high number on the benchmark.
The database, in fact, is one reason why most Java EE benchmarks (and workloads) will not scale linearly -- you can horizontally scale appserver tiers pretty well, but there is still only a single database that must handle an increasing load.
On the appserver side, horizontal scaling is not quite just a matter of throwing more hardware at the problem. SPECjAppServer 2004 is partitioned quite nicely: no failover between J2EE instances is required, connections to a particular instance are sticky, and the instances don't need to communicate with each other. All of that leads to quite nice linear scaling.
But one part of the benchmark doesn't scale linearly, because it is dependent on the size of the database. SPECjAppServer 2004 uses a bigger database for bigger configurations. For example, our previous submission on a single SunFire T2000 achieved a score of 883.66 JOPS@Standard . The benchmark sizing rules meant that the database used for that configuration was only 10% as large at the database we used in our current submission. [More reason why that database scaling is important.] And in particular, it meant that the database in the small submission held 6000 items in the O_item table while our current submission had 60000 items in that table.
For SPECjAppServer 2004, that's important because the benchmark allows the appserver to cache that particular data in ead-only, container-managed EJB 2.1 entities. [That's a feature that's explicitly outside of the J2EE 1.3/1.4 specification, so your portable J2EE apps won't use it -- your portable Java EE 5 apps that use JPA can use cached database data, though somewhat differently.] Caching 6K items is something a single instance can do, but caching all 60K items will cause GC issues for the appserver. Hence, in some areas, the appserver will have to do more work as the database size increases, even if the total load per appserver instance is the same.
So a "big" score on this benchmark is a factor of two things: there are things within the appserver architecture that influence how well you will scale, even in a well-partitioned app. But the amount of hardware (and cost of that hardware) remains the key driving factor in just how high that score can go. As I've stressed many times, benchmarks like this are a proof-point: our previous numbers establish that we have quite excellent performance, and this number establishes that we can scale quite well. As always, the only relevant test remains your application: download the appserver now and see how well it responds to your requirements.
Finally, as always, some disclosures: SPEC and the benchmark name SPECjAppServer 2004 are registered trademarks of the Standard Performance Evaluation Corporation. Competitive benchmark results stated above reflect results published on www.spec.org as of 11/26/07. For the latest SPECjAppServer 2004 benchmark results, visit http://www.spec.org/. Referenced scores:
 Six Sun SPARC Enterprise T5120 (6 chip, 48 cores) appservers and one Sun Fire E6900 (24 chips, 48 cores) database; 8,439.36 JOPS@Standard
 Eleven HP BL860c (22 chips, 44 cores) appservers and two HP Superdomes (40 chips, 80 cores) database; 9,459.19 JOPS@Standard
 Twelve HP BL860c (24 chips, 48 cores) appservers and two HP Superdomes (40 chips, 80 cores) database; 10,519.43 JOPS@Standard
 One Sun Fire T2000 (1 chip, 8 cores) appserver and one Sun Fire T2000 (1 chip, 6 cores) database; 883.66 SPECjAppServer2004 JOPS@Standard