Skip to main content

Testing Scalability Using Japex

Posted by spericas on February 9, 2006 at 10:03 AM PST

As more cores and harware threads are added to the new generation of micro-processors, the ability to easily test software scalability becomes increasingly important. Depending on the type and complexity of your application, using Japex may be a quick and simple way to verify the scalability of your code.

The SunFire T2000 is based on the code-named Niagara processor. The box I was able to get my hands on had 6 cores for a total of 24 hardware threads (a.k.a. strands). I ran two different types of tests. The first test looked at scalability by fixing the number of software threads to 24 and turning on one core at at time, from 1 to 6. The second test fixed the number of active cores to 6 and varied the number of software threads, from 1 to 24.

For the first test I used the Solaris command psrset to essentially reserve cores so that they are excluded in the test. The Japex driver that I wrote supported the parameters preCommand and postCommand to execute an arbitrary command before and after a driver's run: the preCommand was used to create a new processor set and the postCommand was used to delete that processor set. Here is a shortened version of the Japex configuration file:

  <testSuite name="FastInfoset StAX Parser Scalability">

    <param name="japex.warmupTime" value="30"/>
    <param name="japex.runTime" value="30"/>
    <param name="japex.chartType" value="linechart"/>

    <driver name="1Core_FastInfosetStAXDriver">
      <param name="japex.driverClass" value="FastInfosetNiagaraDriver"/>
      <param name="description" value="FastInfoset StAX driver using 1 core"/>
      <param name="japex.numberOfThreads" value="24"/>
      <param name="preCommand"
        value="psrset -c 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23"/>
      <param name="postCommand" value="psrset -d 1"/>
    </driver>

    <driver name="2Core_FastInfosetStAXDriver">
      <param name="japex.driverClass" value="FastInfosetNiagaraDriver"/>
      <param name="description" value="FastInfoset StAX driver using 2 cores"/>
      <param name="japex.numberOfThreads" value="24"/>
      <param name="preCommand"
        value="psrset -c 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23"/>
      <param name="postCommand" value="psrset -d 1"/>
    </driver>

    <driver name="3Core_FastInfosetStAXDriver">
      <param name="japex.driverClass" value="FastInfosetNiagaraDriver"/>
      <param name="description" value="FastInfoset StAX driver using 3 cores"/>
      <param name="japex.numberOfThreads" value="24"/>
      <param name="preCommand"
        value="psrset -c 12 13 14 15 16 17 18 19 20 21 22 23"/>
      <param name="postCommand" value="psrset -d 1"/>
    </driver>

    ...
  <testSuite>

Note how the number of threads is fixed for each driver (which means that japex.numberOfThreads could have been defined globally) and how the processor set defined by each driver gets smaller to make more cores available to the driver under test. The resulting chart showing all the means in shown below.

For the second test, there was no need to run any commands as the number of cores was fixed. Thus, the Japex configuration file simply defined different values of japex.numberOfThreads for each driver as shown below.

  <testSuite name="FastInfoset StAX Parser Scalability">

    <param name="japex.warmupTime" value="30"/>
    <param name="japex.runTime" value="30"/>
    <param name="japex.chartType" value="linechart"/>

    <driver name="1T_FastInfosetStAXDriver">
      <param name="japex.driverClass" value="FastInfosetNiagaraDriver"/>
      <param name="description" value="FastInfoset StAX driver using 1 thread"/>
      <param name="japex.numberOfThreads" value="1"/>
    </driver>

    <driver name="2T_FastInfosetStAXDriver">
      <param name="japex.driverClass" value="FastInfosetNiagaraDriver"/>
      <param name="description" value="FastInfoset StAX driver using 2 threads"/>
      <param name="japex.numberOfThreads" value="2"/>
    </driver>

    <driver name="4T_FastInfosetStAXDriver">
      <param name="japex.driverClass" value="FastInfosetNiagaraDriver"/>
      <param name="description" value="FastInfoset StAX driver using 4 threads"/>
      <param name="japex.numberOfThreads" value="4"/>
    </driver>

    ...
  <testSuite>

And the resulting chart showing the arithmetic, geometric and harmonic means is shown next.

Note that the X axis is not linear as it has a data point for 2 threads between 1 and 4. Modulo that observation, the parser scales linearly up to when the number of software threads equals approximately twice the number of cores. From that point on, it appears that the fact that the strands share resouces within a core causes the curve to flatten as the peak throughput is reached. I should point out that these results were obtained without tuning the VM (in particular, with the default number of parallel GC threads) and that the benchmark under test reads all the data from a memory buffer, making it very CPU intensive.

To summarize, Japex can be used to run some simple scalability tests by setting the value of the japex.numberOfThreads parameter on a per driver basis. Although this sort of test is not a replacement for heavy-weight macro-benchmarks, it takes very little time to get some results and can be used to easily spot resource contention problems. Hopefully you'll find this useful too. Happy benchmarking!

Related Topics >>