Skip to main content

Simple Benchmarking with Faban

Posted by sdo on April 19, 2007 at 10:52 AM PDT

A few weeks ago, I wrote about the href="http://weblogs.java.net/blog/sdo/archive/2007/03/ab_considered_h.html">shortcomings
of ab, the Apache Benchmark program. It's major shortcoming is that
it is single-threaded, making it very difficult to measure accurately a
multi-threaded application or web server. At the time, I promised to
blog soon about an alternative open-source load generator called href="http://faban.sunsource.net/">Faban. Real-life intervened,
but here is the promised blog about using Faban as an alternative to ab.



Faban is actually a quite sophisticated benchmarking framework. It is
designed to automate running benchmarks from a set of clients, which is
of course quite useful when trying to generate load on an application
server. As such, it has a very nice GUI where you can define and submit
jobs, co-ordinate driver hosts, and manage all the tasks necessary for
large-scale benchmarking. Faban also allows you to write your own
load-generating driver, so that the interaction with your application
can be quite sophisticated: you can look through the returned HTML and
figure out what links to follow next; you can define interactions via
IIOP or other protocols; you can have a variety of interaction models
based on think time or cycle times; and so on. For HTTP, its timings
are as accurate as possible because it interposes at the
lowest-available socket level in the JVM and takes its timing
measurements there.



However, Faban also ships with a standard HTTP driver that allows
moderately complex operations (so you don't need to write a driver) and
a set of classes that allow you to run clients from simple scripts.
Because I'm a simple, command-line oriented kind-of-guy, that's the way
I typically use Faban, and that's the easiest way to use it as a
replacement for ab. So that's what I'll discuss here. But I urge you to
check out Faban's full set of features; it's really a very powerful
tool. And it's interesting to know that Faban will form the basis of
the driver used by SPEC in their next version of their appserver
benchmark.



So, for a better ab-like test, what do we need? First, download the
latest nightly build of the faban client jars. Then you will need to
write two things: a run.xml file that defines how you want to run the
benchmark, and  a quick script to start the benchmark.



The run.xml file defines how a benchmark should be run. The faban
documentation lists a lot of options for that file, but the simplest
case for our purposes is this:

<?xml version="1.0" encoding="UTF-8">
  <runConfig>
    <runControl unit="time">
      <rampUp>300</rampUp>     --> defines a 300 second rampup time
      <steadyState>300</steadyState>  --> defines a 300 second measurement cycle
      <rampDown>120</rampDown> --> defines a 120 second rampdown time
    </runControl>
    <benchmarkDefinition>
      <name>my_test</name>   --> used for the run sequence file
      <metric>ops/sec</metric>
    </benchmarkDefinition>
    <outputDir>
      /path/to/an/existing/directory   --> results will go in this directory
    </outputDir>
    <driverConfig name="http_driver1">
      <threads>64</threads>    --> number of clients to run, each in its own thread
      <requestLagTime>         --> defines think time for each client
        <uniform>
          <cyclteType>thinktime<cycleType>
          <cycleMin>0<cycleMin>
          <cycleMax>0<cycleMax>  --> in this case, 0 think time between requests
        </uniform>
      </requestLagTime>
      <operation>
        <name>getTest</name>   --> you can define multiple operations; give each a unique name
        <url>http://host:port</url>
        <get><![CDATA[/index.html]]></get>   --> could be post, and an arbitrary URL
        <max90th>.1</max90th>        --> 90% of responses must be received in this time period
      </operation>
      <operationmix>               --> Defines how many of each multiple operation is executed
        <name>getTest</name>
        <r>1</r>                   --> 100% of operations are getTest
      </operationmix>
    </driverConfig>
  </runConfig>

So this file will execute calls to http://host:port/index.html from 64
threads with 0 think time between requests. You can get an idea from
this how faban allows you to build more complex tests of your
appserver. Next, you'll need a script to run the benchmark:

#!/bin/sh
CLASSPATH=/path_to_faban/fabanagents.jar:/path_to_faban/fabancommon.jar:/path_to_faban/fabandriver.jar:$JAVA_HOME/lib/tools.jar
export CLASSPATH
java -Djava.security.policy=policy_file com.sun.faban.common.RegistryImpl & pid=$!
sleep 3
java $JVM_OPTIONS -Dbenchmark.config=run.xml com.sun.faban.driver.core.MasterImpl
kill $pid

When you run this, a directory containing the results of the run is
created (based on the outputDir stanza); the directories are numbered
consecutively (the last value of that number is in $HOME/my_test.seq --
the run sequence file). There's a lot of data there, but the key file
is summary.xml, which will contain the metric and whether the test
passed or failed according to the parameters (desired cycle times, 90th
percentile times, and so on) defined in your run.xml. Those lines look
like this:

    <metric unit="ops/sec">9061.7</metric>
    <passed>true</passed>

So my test got 9061 requests per second. There's a nice XSTL
transformation available with faban if you want to see the entire set
of data.



Because its a relatively new project, the power of Faban is somewhat
exposed in the complexity to run it -- at least for the time being.
There's no reason why a simple Java program couldn't take command-line
argument and produce the necessary scripts to run Faban in this simple
mode. If real life doesn't intrude, I'll work on that in the next few
weeks. But don't wait for me: join the Faban project and contribute
yourself!