 |
|
 |
 |
|
|
 Scott Oaks's BlogSimple Benchmarking with FabanPosted by sdo on April 19, 2007 at 10:52 AM | Comments (7)A few weeks ago, I wrote about the shortcomings of ab, the Apache Benchmark program. It's major shortcoming is that it is single-threaded, making it very difficult to measure accurately a multi-threaded application or web server. At the time, I promised to blog soon about an alternative open-source load generator called Faban. Real-life intervened, but here is the promised blog about using Faban as an alternative to ab. Faban is actually a quite sophisticated benchmarking framework. It is designed to automate running benchmarks from a set of clients, which is of course quite useful when trying to generate load on an application server. As such, it has a very nice GUI where you can define and submit jobs, co-ordinate driver hosts, and manage all the tasks necessary for large-scale benchmarking. Faban also allows you to write your own load-generating driver, so that the interaction with your application can be quite sophisticated: you can look through the returned HTML and figure out what links to follow next; you can define interactions via IIOP or other protocols; you can have a variety of interaction models based on think time or cycle times; and so on. For HTTP, its timings are as accurate as possible because it interposes at the lowest-available socket level in the JVM and takes its timing measurements there. However, Faban also ships with a standard HTTP driver that allows moderately complex operations (so you don't need to write a driver) and a set of classes that allow you to run clients from simple scripts. Because I'm a simple, command-line oriented kind-of-guy, that's the way I typically use Faban, and that's the easiest way to use it as a replacement for ab. So that's what I'll discuss here. But I urge you to check out Faban's full set of features; it's really a very powerful tool. And it's interesting to know that Faban will form the basis of the driver used by SPEC in their next version of their appserver benchmark. So, for a better ab-like test, what do we need? First, download the latest nightly build of the faban client jars. Then you will need to write two things: a run.xml file that defines how you want to run the benchmark, and a quick script to start the benchmark. The run.xml file defines how a benchmark should be run. The faban documentation lists a lot of options for that file, but the simplest case for our purposes is this: <?xml version="1.0" encoding="UTF-8">So this file will execute calls to http://host:port/index.html from 64 threads with 0 think time between requests. You can get an idea from this how faban allows you to build more complex tests of your appserver. Next, you'll need a script to run the benchmark: #!/bin/shWhen you run this, a directory containing the results of the run is created (based on the outputDir stanza); the directories are numbered consecutively (the last value of that number is in $HOME/my_test.seq -- the run sequence file). There's a lot of data there, but the key file is summary.xml, which will contain the metric and whether the test passed or failed according to the parameters (desired cycle times, 90th percentile times, and so on) defined in your run.xml. Those lines look like this: <metric unit="ops/sec">9061.7</metric>So my test got 9061 requests per second. There's a nice XSTL transformation available with faban if you want to see the entire set of data. Because its a relatively new project, the power of Faban is somewhat exposed in the complexity to run it -- at least for the time being. There's no reason why a simple Java program couldn't take command-line argument and produce the necessary scripts to run Faban in this simple mode. If real life doesn't intrude, I'll work on that in the next few weeks. But don't wait for me: join the Faban project and contribute yourself! | |