Santiago Pericas-Geertsen's Blog

Performance Tracking using Japex

Japex 1.0.9 now includes a new performance tracking tool. If you are a Japex user, or are planning on becoming one soon, you should check this out. The performance tracking tool can be used to send e-mail notifications for regressions (or progressions) based on a pre-defined threshold. This new tool complements the trend reporting tool already available in Japex.

If you have been a Japex user for a while, you probably have benchmark with a directory that looks like this,

  reports/1-thread-http/2006_04_10_00_01
  reports/1-thread-http/2006_04_11_00_01
  reports/1-thread-http/2006_04_12_00_01
  reports/1-thread-http/2006_04_13_00_01
  reports/1-thread-http/2006_04_14_00_01

In particular, the directory above is generated by a cron job that runs a JAX-WS benchmark every night using the latest bits from Java.net. You can aggregate all the information collected over a period of time by using the Trend Report Tool in Japex. Using this tool you can get an HTML page with a line chart that shows the performance of each driver in your benchmark over a period of time. This is nice, but you still have to go and check the page on regular basis to make sure that there are no performance regressions.

Using the performance tracking tool, you can extend your cron job to compute a delta between the last two runs and automatically send an e-mail notification when a certain threshold is exceeded --which means it will report both regressions and progressions. You can do this using a shell script or an Ant task, I'll show the latter solution next:

  <target name="regression-tracker">
      <java dir="." fork="true" classname="com.sun.japex.RegressionTracker">
          <classpath refid="run.classpath"/>
          <jvmarg line="-Dmail.smtp.host=smtp-server.yourcompany.com"/>
          <jvmarg line="-Dmail.recipients=you@yourcompany.com"/>
          <jvmarg line="-Dmail.subject='Performance Tracking Notification'"/>
          <arg line="-threshold 5 ${basedir}/reports/1-thread-http 
                    ${basedir}/reports/1-thread-http/regression-tracker"/>
      </java>
  </target>

As shown above, the regression tracker uses a number of self-explanatory system properties to configure the JavaMail API. In addition, it takes a percentage variation threshold (the -threshold option) as well as the location of the input and output directories. In the example above, if the performance variation of any driver exceeds 5%, an e-mail notification will be sent to you.

So how does the notification look like? The content of this notification is likely to evolve as we learn more about how to best take advantage of this tool, but here is a sample of what you would be getting today:

Parameters

• Last report: file:/export/home/wsperf/MicroBenchmarks/Jaxws/WSpex/reports/1-thread-http/2006_04_13_00_01/report.xml
• Next report: file:/export/home/wsperf/MicroBenchmarks/Jaxws/WSpex/reports/1-thread-http/2006_04_14_00_01/report.xml
• Result unit: TPS
• Threshold: 1.0%

Hope you find this tool useful. And, as always, happy benchmarking!

Comments

This work is licensed under a Creative Commons License.