Scott Oaks's Blog

Understanding Performance

I'm prompted to look into this topic because of a recent blog by Eric Stahl, who discusses the performance of SPECjAppServer 2004. The thing about application server benchmarks -- and especially the ones from SPEC -- is that they are trivial to scale horizontally. It's not enough just to look and see which application server has the highest score, because any vendor at any time can put together a larger configuration of machines and get the new world's record. [There's a slight complication here, in that the benchmark is a system benchmark, not an application server benchmark, and hence is subject to pressure from a back-end database. That's an important topic for another day, but it doesn't affect today's point.]

Interestingly, Eric understands this at some level, because he concludes by calculating how many transaction each application server can get on their respective CPUs. When you know that the total score doesn't mean anything, it's natural to attempt to normalize the numbers from disparate software and hardware combinations. And of course, this particular normalization allows him to show Sun's score in a bad light and conclude "this is a perfect example of the potential for free software, such as Sun's app server or JBoss, to drive costs up by needing significantly more hardware." Right instinct; wrong analysis.

It's the right instinct because performance isn't who has the highest benchmark number. Performance is who performs the work "best" -- where it's up to you to define best. Maybe someone actually would define best as "most transactions per CPU" (even if, as in this case, the CPUs aren't equivalent, which makes the calculation that more irrelevent). But perhaps you'd like to define best as "most cost-effective overall." I'd argue that definition has more merit.

Let's look at the acquisition costs of these results. By my calculations, BEA's application server costs around $120K to produce 1664 JOPS on machines that cost around $8K. Sun's application server is free but requires more ~$4K machines. So that translates to roughly $100 per transaction for BEA and about $53 for Sun. Of course, you may argue that I'm being overly simplistic; there are additional costs like support costs, and database costs, and so on. Some of those might be important to your decision making and some may not -- in particular, the backend database is going to support a certain number of transactions regardless of the front end, so leaving that out is a way to concentrate only on the relative merits of the appserver tier (plus, database hardware and software pricing makes price comparisons between disparate systems much less interesting).

SPEC makes it theoretically possible (though tedious) to figure this out for yourself; all submissions include a full Bill of Materials from which you can figure out the total cost of the submission (assuming vendors or their resellers have the relevent prices on their websites), or just the software, or just what's needed to run the appservers without the database, with our without supports costs, or whatever parts you want to include or isolate.

And that, of course, is the point: it's up to you to determine what's important when you make a software/hardware decision. Just don't be swayed by incomplete arguments that free software is going to cost you more in the end.

Comments

• The other issue is simply that Eric is assuming that the CPU's used in the servers are comparable and equal. It's also assuming that the JVMs are equal, since they run JRockit on Xeon vs Suns SOLARIS JVM on AMD, or IBMs JVM. What would the results be running Suns JVM on x86?
  What's quite nice about the Sun benchmarks is simply that there are several of them. Some use better hardware than the others. Most of them use the lower end of the server line for the front end. By doing this, one can get a better idea about how the machine can affect the different architecture. It's quite interesting to see the SJAS/MySQL benchmark, for example.
  Another part of the SPEC benchmarks is that it lets folks know whether or not the app server is "fast enough". If I have an application with medium load for, say, 10-20 users, a two tier SJAS/MySQL solution may well fit the bill. $8K in hardware. There are a lot of departmental applications out there in smaller and medium sized businesses that should be able to leverage application servers. Just because my app only has five users, doesn't mean I can't or shouldn't use an application server.
  Finally, there's one more aspect that is shown here but not mentioned at all.
  Simply that you can take the same application, run it on different application servers and get different performance metrics. The actual changes that can be done to the SPEC application are pretty minor as I understand it. Build the schema, tweak the deployment descriptors, and off you go. There isn't a lot of leeway to recode the application itself for performance, putting most of the heavy lifting on the app server implementation itself.
  Why is this important? It's important because it shows the value of portability in the application, and portability at the JEE level. If I find SJAS is too slow, and the application is written with an eye towards the specification, then I can "simply" redeploy on WLS, and get a performance boost. Yes, I know it's not quite that easy, but it's certainly not the level of rewriting the application either.
  I think this is important for the JEE spec because there is this push from the "lightweight" camp that's implementing their services all within a servlet container and tossing aside the JEE app server entirely. One of the arguments is that these applications are very portable. Between servlet containers, yes, that's true since your application is bundling most of the services that a JEE app server provides for you. But if your application is performing poorly, you don't have the option of going to a faster app server, of stepping up from SJAS to WLS. So, of what real value is that portability over the simple platform portability inherent in Java?
  To paraphrase, "Lies, Damn Lies, and Benchmarks". As you mentioned, one needs to consider all aspect of the TOC when evaluating. When someone wants to try scaling via hardware, you need to consider environmental and power issues, management issues, etc., and you need to weigh those costs against software licensing, as well as the mangement issue of those licenses.
  But by the same notion, you shouldn't overbuild and overspecify. Don't pay for performance you don't need.
  

  Posted by: whartung on September 30, 2005 at 06:00 PM

This work is licensed under a Creative Commons License.