Scott Oaks's Blog

Sun Posts SPECjAppServer2004 results using Glassfish

Sun posted a result of 712.87 SPECjAppserver 2004 JOPS@Standard on a configuration of 3 Sun Fire X4100 application servers and 1 Sun Fire X4100 database. Direct comparison to our previous result on 3 application servers (Sun Fire v20z machines) is a little tricky: the newer machines are dual-core and have a slightly faster clock speed, so you'd expect this newer configuration to have slightly better than 100% performance of the old configuration. Yet our result shows a 167% improvement over that previous submission -- a substantial improvement in the software layer no matter how you look at it.

Congratulations to everyone who worked on glassfish: the fact that we were able to get such an improvement while at the same time dealing with the aggressive schedule to support Java EE 5 and the new scenario of working in the open source community is a great achievement!

Benchmark Description
SPECjAppServer 2004 is a multi-tier benchmark for measuring the performance of Java 2 Enterprise Edition (J2EE) technology-based application servers. Moreover, SPECjAppServer2004 also heavily exercises all parts of the underlying infrastructure that make up the application environment, including hardware, JVM software, database software, JDBC drivers, and the system network. The primary metric of the SPECjAppServer2004 benchmark is jAppServer Operations Per Second ("SPECjAppServer2004 JOPS") in either @Standard or @Distributed mode.

Required Disclosure Statement:
SPECjAppServer2004 3x Sun Fire X4100 appservers (12 cores, 6 chips) and 1 Sun Fire X4100 database (4 cores, 2 chips) 712.87 SPECjAppServer2004 JOPS@Standard. SPECjAppServer 2004 3x Sun Fire V20z appservers (6 cores, 6 chips) and 1 Sun Fire V20z database (2 cores, 2 chips) 266.01 SPECjAppServer2004 JOPS@Standard. All results from www.spec.org as of 05/25/06. SPEC, SPECjAppServer reg tm of Standard Performance Evaluation Corporation

Comments

• Nice! How does this compare to other SPECjAppServer 2004 results from a price/performance perspective?

  Posted by: ai109478 on May 25, 2006 at 03:44 PM

• Well, that of course depends on how you want to look at it. If you only want to consider the acquisition cost of software, then the JOPS/$ figure would be infinite. Probably not what you're looking for :-)

  Check out this blog, which has some good figures on price/performance:
  http://blogs.sun.com/roller/page/insider#open_source_comes_of_age
  

  Posted by: sdo on May 26, 2006 at 12:36 PM

• I just reported on this news at: http://www.infoq.com/news/Glassfish-SPECjAppServer2004

  Feel free to join and comment.

  Posted by: fmarines on May 26, 2006 at 04:47 PM

• Excellent, I've also seen a lot of improvement of start time, and memory utilization .... all good.
  

  Posted by: jcmartin on May 29, 2006 at 09:33 PM

• Thanks for noticing!

  Although there are no official benchmarks to "prove" it, we've measured about a 30% increase in startup and deployment time, and a 15-30% decrease in memory footprint (more on RISC architectures; less on CISC architectures). YMMV, of course.

  Posted by: sdo on May 30, 2006 at 07:46 AM

• Interesting. I'd like to see these results compared to other open source servers like the latest Tomcat. Since Tomcat is probably the defacto OS server for JSP/Servlets, it would be more likely for some of us folks to make a switch to Glassfish if there were some scalability comparisons. I know the best way to compare is the benchmark *my* app running on each, but a more generic comparison would motivate me to try it.
  Congrats to the Glassfish team.

  Posted by: bsavard on May 31, 2006 at 12:57 PM

• I'd also like other open source servers to post SPECjAppServer results, but none of them have. I wonder why...

  But this benchmark cannot run on Tomcat; Tomcat is not a Java EE application server. Tomcat doesn't have support for EJBs, or MDBs, or other things that this benchmark requires.

  At JavaOne last year, we presented some internal benchmarks we had showing Tomcat scalability vs. scalability of grizzly, the NIO-based, pure Java web container that comes with our appserver. In terms of scalabilty, there's no comparison: tomcat is based on the traditional Java I/O model and hence requires one thread per client (or it closes connections to the client so that the TCP stack overhead becomes a big problem; your appserver ends up spending all its time re-accepting connections). That limits Tomcat to about 2000 clients on a typical 4-8 CPU appserver -- in fact, regardless of the number of CPUs, ~2K is all you get because that's all the threads you can create.

  Grizzly, on the other hand, scales easily to handle many, many thousand connections depending on the size of the machine, since the multiple connections are multiplexed over a (relatively) small threadpool.

  Check out http://weblogs.java.net/blog/jfarcand/archive/2006/03/can_a_grizzly_r.html for more data.

  Posted by: sdo on May 31, 2006 at 02:48 PM

This work is licensed under a Creative Commons License.