Kohsuke Kawaguchi's Blog

JAX-WS RI 2.1 benchmark details

In this post I'm going to talk about the details of the benchmark Bharath did (kudos to him and the rest of the performance team.) For more about the JAX-WS RI 2.1 release in general, please refer to Vivek's post.

Summary

The basic idea of the benchmark is to have a lot of clients send a lot of requests to the server concurrently. The server echos back the data to the client, and then we measure how many requests a server is processing. The test is repeated with 15 different test payload; echo(Void|Integer|String|Date|Struct) tests a small payload, where the data is just one int, string, etc. echoSyntetic...K is the binary payload, where 1K,4K,8K,12K represents the size of the binary. Finally, echoArray and echoOrder tests have a significantly larger payload.

The picture on the right shows the summary. "Number of requests processed per second" is normalized so that you can always see Axis2 as 100%. Depending on the data point, you see that the JAX-WS RI is 30% to 100% faster.

On smaller payloads, such as echoString, echoInteger, etc, you tend to see a larger difference. This is because (relatively speaking) the weight of the databinding is small, and so the test reveals the true difference in the web service layer proper. On larger payloads, the time tends to be spent more on the databinding side, so the difference tends to become small.

Setup

The client machine is SunFire x4600 with 15.5 GB of memory and 8 Opteron CPUs. It runs JDK 1.5.0_10-b03 on Solaris 10. We used this monster just to make sure that we have enough clients to keep the server busy all the time. We run total of 32 threads on this machine, each uses JAX-WS to send a SOAP request to the server as fast as possible. We verified that the server CPU was fully saturated.

The server machine is another SunFire x4600. It has the same amount of memory, same OS, same JDK, except that there are only 4 CPUs. We used Glassfish v2 milestone 4 as the container, with -server -Xms2g -Xmx2g as the JVM option. Glassfish is a JavaEE 5 container, which includes StAX. So this means we are using its StAX implementation, SJSXP.

We tried Axis2 1.1.1 with XMLBeans and JAX-WS RI 2.1 with JAXB. We tried to use Axis Data Binding first, but we noticed that under a high load it fails with what seemingly like a concurrency related data corruption. So we decided to move on to XMLBeans, which is listed next to ADB in their quick start guide. We'll see if we can figure out what's going on with Axis+ADB in the future. It could be a Glassfish problem, who knows.

Each test was run for 2 minutes. The first 1 minute is just for the warm-up time, and the measurement only considers the 2nd minute. So one complete test run takes 2 minutes x 15 tests x 2 toolkits = 1 hour. Our harness runs this 4 times, and throw away the first two runs as additional warm-up. The data shown below were the result of the last 2 runs (out of 4.)

We plan to make the benchmark code available on java.net, so stay tuned.

Results

The raw numbers are as shown on the right:

The 'TPS' stands for 'transactions per second' and this represents the number of requests that was processed per second. The 'stddev' is the standard deviation between different runs, so you can use that to see how many digits of TPS you can trust.

Analysis

We've been working on this for a long time now, and coincidentally another group posted another web service stack benchmark just a few days ago. While we only had very limited time to look at it, we noticed that their benchmark, despite being run on 4-way Xeon system, records roughly around 3,000 reqs/sec (for example on echoVoid test.) Our benchmark recorded more than 10,000 reqs/sec, even for Axis2, on a 4-way Opteron system. While one cannot really compare reqs/sec on different systems in a meaningful way, we nevertheless wonder if their Xeon system could have done much better than 3,000 reqs/sec.

It's also clear we've got more work to do here. We need to get to the bottom of the ADB issue for one thing. We also want to test the scalability of these stacks.

In the end, however, what really matter to you is your own appliation with your own data. So we want you to compare the toolkits by yourself, with your own use cases, and let us know what your findings are.

Comments

• Hi Kohsuke, It's well known in the Axis2 community that Xmlbeans is quite slow, that's exactly the reason why we started writing ADB in the first place. So, please do help us with some info on the problem that you faced so that you/we can run RI/JAXB2 vs Axis2/ADB. That's much more meaningful as ADB is the default databinding in Axis2. thanks, dims

  Posted by: dims on February 04, 2007 at 04:21 AM

• Why only use 2GB heap on a 15.5 GB machine? Am I missing the point?

  Posted by: bvansomeren on February 04, 2007 at 06:37 AM

• dims — Thanks for the info, though I would like to point out that on a larger data set, where databinding weighs in more, Axis2 gets closer to the JAX-WS RI. This seems to indicate that at least the sustained unmarshalling performance of XMLBeans is not that bad.

  Also, is it true that ADB can handle only a limited subset of XML Schema? If so, it is not usable for many wsdl->java scenarios that people face.

  Posted by: kohsuke on February 04, 2007 at 07:41 AM

• bvansomeren — Good question. I believe the reason is to reduce the variation caused by GC by making sure that it occurs reasonably often. Or maybe it could be to adequately penalize a toolkit that produces a lot of garbages. I'll ask the performance guys to chime in.

  Posted by: kohsuke on February 04, 2007 at 07:43 AM

• I imagine that the small amount of VM flags for the server machine indicate a Java 6 VM.

  Posted by: alexismp on February 04, 2007 at 01:12 PM

• Any reason you guys did not consider testing with larger payloads? Your test messages seem unfairly small/simple to me, as they are not really exercising the complex aspects of JAXB. It would be also interesting to see the numbers by excluding data-binding from the picture.

  Posted by: sallamar on February 04, 2007 at 04:22 PM

• alexismp — I'm pretty sure that the server is using the JDK 1.5. It's somewhat unlikely that Bharath made a mistake in such a simple setting. Java5 comes with VM ergonomics, so it's supposed to do "the right thing" without specifying too many options.

  Posted by: kohsuke on February 04, 2007 at 08:30 PM

• sallamar — I don't know the exact payload size right now, but I believe echoOrder500 is fairly large, although how large is large enough is subjective issue. I believe it's also modeled after UBL, so you could say it's complicated enough.

  Also, echoVoid pretty much excludes databinding, as that's the smallest possible data you can send. But in practice, measuring those scenarios is not that interesting, because in real applications you'll be always sending some data, so there always will be databinding.

  Posted by: kohsuke on February 04, 2007 at 08:35 PM

• bvansomeren - These tests where run on a 32 bit JVM. We didn't find the necessity to go beyond 2GB for this benchmark. And tuning were the same for both JAX-WS 2.1 and Axis2.

  Posted by: bharathm on February 04, 2007 at 09:20 PM

• alexismp - Thanks for asking this question. These benchmarks were run on JDK 1.5.0_10-b03 as posted in the blog details. There are some JVM performance improvements were made in Java Reflection area but you will see these in future releases of JDK like 1.5.0 update 11 and in 1.6.0 update 2. You might have already aware of the JDK 1.6.0 performance improvements. For JAX-WS 2.1 running on Glassfish 9.1, if we run on JDK 1.6.0 instead of JDK 1.5.0, we saw 7-10% performance boost. I am sure other benchmarks might see more. Stay tuned on this front.

  Posted by: bharathm on February 04, 2007 at 09:35 PM

• Hi Kohsuke, which charting library was used to generate the radar chart ?

  Posted by: mheinz on February 05, 2007 at 02:54 AM

• Sorry, I overlooked the "same JDK" part of the description. Just to be clear - I like the fact there's so little options! :)

  Posted by: alexismp on February 05, 2007 at 12:03 PM

• It really does not make sense to me as well why use a 32 bit JVM on a 15.5 GB RAM machine. Moreover, I am sure Axis2 can be significantly optimized further, not only by switching the databinding framework, but by configuring dispatchers and phases as well. It is very difficult to achieve a non-biased performance evaluation indeed.

  Posted by: atodorov on February 05, 2007 at 11:56 PM

• We can probably tweak the kernel setting, or network setting to maximize the performance, or change other million settings to get the best out of these toolkits.

  But I think a benchmark is more useful if it does what most users do. Both JAX-WS and Axis2 are used out of the box. There's no tweaking of any kind for both.

  So while it's good to know that Axis is claimed to be tune-able, I suspect many people won't consider lack of tuning as a bias toward one over the other.

  Posted by: kohsuke on February 06, 2007 at 12:25 PM

• bharathm Thank you, makes some sense. Any stats on GC's during the test runs?

  Posted by: bvansomeren on February 06, 2007 at 02:24 PM

• atodorov - what eventually matters is your application performance, which is the reason we recommand you to try it on your own benchmark on your own platform and on your own usecase. No restrictions! Well, even if we had run with 64 bit JVM, which is going to be a common factor for both JAX-WS 2.1 and Axis2, would produce same outcome unless Axis2 generates lots of objects for garbage collection. We beleive, functionality, like good schema support, usability & spec complaint etc, is as important as performance. JAX-WS 2.1 comes to you with no compromise on both! Needless to say, we always welcome your findings on your application.

  Posted by: bharathm on February 06, 2007 at 10:10 PM

• By the way, you are probably aware that Axis2 is supposed to already have the JAX-WS implementation as an Axis2 module ... So it would be probably more interesting to benchmark (not immediately) the JAX-WS "RI" implementation vs. the Axis2 JAX-WS implementation.

  Posted by: atodorov on February 06, 2007 at 11:42 PM

• GlassFish V2 is a production-ready Java EE 5 application server currently in beta. Apart from being Java EE 5 compliant, it includes support for EJB 3.0, JPA, JSF 1.2, JSP 2.1, Servlet 2.5, JAX-WS 2.1, and JAXB 2.1 technologies. Some of the new features in V2 are clustering (screencast, blog), high availability (screencast), and load balancing (GF page, blog) to support highly scalable, volume enterprise deployments for service-oriented architecture (SOA) and Web 2.0 applications. GlassFish web services stack is one of the best (comparison matrix) in the industry. Microsoft interoperability with Web Services Interoperability Technology (WSIT) comes with GlassFish V2. GlassFish administration console offers one of the best user experience in the industry. One can manage the whole domain (multiple clusters spanning different machines) from the same console. Support for next-generation Web with Ajax, scripting, and RESTful web services is also available. While NetBeans IDE offers the best integration for Java EE 5 and GlassFish, Eclipse (MyEclipse, free) and Intellij plugins are also available.
  
  GlassFish community (users@GF alias, forum) is very active. We also have IRC chat for GlassFish. Full release of GlassFish V2 is planned for the Fall 2007.
  
  Source : Dodgebrakedustshieldshttp://blogs.sun.com/nazrul/entry/glassfish_javaone

  Posted by: angelhound02 on September 04, 2007 at 01:59 AM

This work is licensed under a Creative Commons License.