Trend Reports in Japex
Back in April, I blogged about using the Japex regression tracking tool to monitor performance over time. This tool computes a diff between the last two reports and sends alerts via e-mail if a regression (or a progression) higher than a threshold is detected. In addition to looking at the last two reports, performance can be tracked by analyzing trends, i.e. by plotting performance results over the last N reports. The Japex trend report tool can be used for this purpose.
The original trend report tool was contributed by Joe Wang. I have recently re-design this tool to address some new requirements, and at the same time I reduced the number of options to make it easier to use.
If you have been a Japex user for a while, you probably have benchmark with a directory that looks like this (using JAXBpex as an example):
Using Ant, you could add this target to generate a trend report for this benchmark as follows:
<java dir="." fork="true" classname="com.sun.japex.TrendReport">
<arg value="JAXBpex Unmarshalling"/>
The first parameter is just a title for the output report. The next two refer to the directory where the Japex test suite reports are located and the output directory, respectively. Every time your run this target, the report in the output directory will be overwritten. Two optional arguments not shown in this target are date and offset which can be use to select a subset of the reports in the directory.
So how does a report look like then? Here is a screenshot:
A trend report includes a chart for each mean (arithmetic, geometric and harmonic) across all drivers and all reports, as well as a chart for each test case (not shown above) also across all drivers and all reports. In addition, it includes a pull-down menu with an index for quickly accessing a particular chart.
I mentioned in passing the new requirements that prompted the re-design of this tool. One such requirement is easy integration into Hudson. Kohsuke and I have been discussing this on the Japex and Hudson mailing lists. We are already using Hudson to build projects and run conformance tests and performance tests automatically, so it is only natural to include trend charts as part of a Hudson project page. Stay tuned ...