Skip to main content

Trend Reports in Japex

Posted by spericas on August 18, 2006 at 2:30 PM PDT

Back in April, I blogged about using the Japex regression tracking tool to monitor performance over time. This tool computes a diff between the last two reports and sends alerts via e-mail if a regression (or a progression) higher than a threshold is detected. In addition to looking at the last two reports, performance can be tracked by analyzing trends, i.e. by plotting performance results over the last N reports. The Japex trend report tool can be used for this purpose.

The original trend report tool was contributed by Joe Wang. I have recently re-design this tool to address some new requirements, and at the same time I reduced the number of options to make it easier to use.

If you have been a Japex user for a while, you probably have benchmark with a directory that looks like this (using JAXBpex as an example):

  reports/unmashaller/2006_08_04_01_33
  reports/unmashaller/2006_08_04_09_54
  reports/unmashaller/2006_08_04_13_02
  reports/unmashaller/2006_08_04_14_17
  reports/unmashaller/2006_08_04_18_36
  reports/unmashaller/2006_08_05_01_33
  reports/unmashaller/2006_08_06_01_33
  reports/unmashaller/2006_08_07_01_33
  reports/unmashaller/2006_08_07_11_14
  reports/unmashaller/2006_08_08_01_35
  reports/unmashaller/2006_08_09_01_34
  reports/unmashaller/2006_08_11_01_35
  reports/unmashaller/2006_08_12_01_48
  reports/unmashaller/2006_08_16_01_35

Using Ant, you could add this target to generate a trend report for this benchmark as follows:

   <target name="generate-trends">
        <java dir="." fork="true" classname="com.sun.japex.TrendReport">
            <classpath refid="class.path"/>
            <arg value="JAXBpex Unmarshalling"/>
            <arg line="reports/unmarshaller"/>
            <arg line="reports/unmarshaller/trends"/>
        </java>
   </target>

The first parameter is just a title for the output report. The next two refer to the directory where the Japex test suite reports are located and the output directory, respectively. Every time your run this target, the report in the output directory will be overwritten. Two optional arguments not shown in this target are date and offset which can be use to select a subset of the reports in the directory.

So how does a report look like then? Here is a screenshot:




trendreport.jpg

A trend report includes a chart for each mean (arithmetic, geometric and harmonic) across all drivers and all reports, as well as a chart for each test case (not shown above) also across all drivers and all reports. In addition, it includes a pull-down menu with an index for quickly accessing a particular chart.

I mentioned in passing the new requirements that prompted the re-design of this tool. One such requirement is easy integration into Hudson. Kohsuke and I have been discussing this on the Japex and Hudson mailing lists. We are already using Hudson to build projects and run conformance tests and performance tests automatically, so it is only natural to include trend charts as part of a Hudson project page. Stay tuned ...

Related Topics >>