David Herron's Blog

Test suites and harnesses, continued

Last week I wrote a bit about the OpenJDK quality team we plan to launch, and about one of our primary concerns. We are pretty sure there are many people who would like to contribute to Open JDK quality, because there are many people who see the vision of Java and want to see it live. This concern we have is about test harness choice and whether the choice of test harness could interfere with our hope to collaborate with the public on test suite development.

Last week I discussed the different modes of tests we have for Java SE: the JCK, the developers unit and regression tests, the performance tests, and the SQE team's functional tests. And, to remind you, by "we" I'm referring to the SQE team and our functional tests.

The JCK is executed using Java Test (a.k.a. JTHarness), as is required by the licensing of the JCK. Java Test has a long history and is the official harness of the JCK. The JTHarness project is an open source release of java test.

The developers unit and regression tests are executed using jtreg, which is a layer over Java Test. A binary release of jtreg is being made through the OpenJDK project.

The SQE team uses javatest for JCK execution, uses jtreg for some tests, but we tend to prefer using "Tonga" for our test suites. Tonga is an internal name, so if it violates anybodies trademark please forgive me. It also has a long history in our team and we find it to have features which make it worth our while to have developed this harness and continue maintaining it. There's also a practical issue that we have over 30,000 tests written using Tonga and it would be costly to convert them to a different harness.

We have a number of questions, and few answers. But before I get into them I want to address a question inspired by my previous blog entry: I'm curious how Tonga differs from xUnit/TestNG

It might be best to go through the functionality of Tonga. The essential difference is that xUnit and TestNG both have the test extend a superclass which provides the test framework. Tonga by contrast runs tests from a main method, and Tonga can also test command lines because it provides a shell script interpreter.

Tests are listed in a test list: The testlist is a simple text file which lists the names of test cases along with a few parameters.

Supports multiple test types: One of the parameters in the testlist is the test type. We use tonga to execute jtreg tests, for example. We believe Tonga can support xUnit or TestNG test types. The native Tonga test format is a directory hierarchy, not package hierarchy, which contains test directories.

Testlist entries are tagged with keywords: And the user can select subsets of tests by specifying the tags.

Configuration through bourne-shell workalike thingy: This makes configuration of a test suite pretty straightforward, given that shell scripting is common knowledge, and also very flexible, due to shell scripting. It also makes for easy automation of command line tests, since you're dealing with shell scripts and hence shell commands. This gives you tons of flexibility in test implementation, giving data to tests, reusing the same test multiple times with different datasets, etc.

Tests are each executed in a fresh VM: Each test execution gets its own JVM instance. If a test pollutes the JVM, following tests know they aren't affected by the previous tests, because each has a fresh (clean) VM to execute in. Tonga can execute tests in "same VM" mode, but this is rarely used.

Distributed test execution: Some complex tests might set up several server processes and run test cases that depend on those servers. For example our networking and security tests regularly use this to set up the required servers, so our tests can make LDAP etc queries against a known environment. This makes it easy to set up and tear down complex test configurations.

XML results format: Tonga can produce its execution results in a well documented and understood XML format. We funnel the results into a results reporting database and web application. The XML file itself is pretty useful and we sometimes XSLT it into standalone web pages. The file contains extensive information about the test environment, JDK version, etc, plus the execution results of each test.

There's more but this should give a flavor of what Tonga offers, and those who know xUnit/TestNG better than I do will see some differences.

My questions:

Does it seem strange for us to consider publishing another test harness? Will you guys be confused over which test harness to use?

Or is that worry overblown? I think it might be... for example I understand most or all of the subteams of the Mozilla project use different harnesses. So it seems like the nature of software engineers to want specific features in test harnesses and sometimes to build new ones when the existing ones don't work the way they want. I read an article recently that software source reuse is often not done in practice, while most people do clearly agree it's a worth thing to try and do.

Do you think the XML results format is attractive? We hope to ask the public to do test execution and send in reports, and would like the reports to be in an XML format. We think the format we've developed is pretty good and think it might be attractive to other test harnesses as well.

Comments

• Your harness should whatever you need it to be: "The question is," said Humpty Dumpty, "which is to be master -- that's all." to quite Lewis Carroll.

  But using a new framework will shrink the pool of time and experience of people able to use it, though hopefully not much.

  Rgds

  Damon

  Posted by: damonhd on April 14, 2007 at 02:04 AM

• I think publishing your own test harness is fine, as it does what you need to do already. If you are interested in seeing the features from the test harness adopted by other projects, in order to be able to migrate to them, I guess you could go with MIT/X license, and talk with JUnit/TestNG folks.

  For Mauve, Roman Kennke wrote a quick JUnit implementation/bridge for Mauve to be able to execute JUnit based tests, so that developers writing to JUnit APIs can write their tests that way, i.e. in particular, one could easily integrate the tests from Apache Harmony into Mauve that way. I assume that you could use a similar 'assimilation' approach if it turns out that you get test submissions from people who prefer JUnit/TestNG over Tonga.

  Having an XML format is attractive, I think.

  On a side note, was Tonga used, by chance, for the tests written in the Java Class Libraries books by Chen et. al.? They seemed to follow the Main.java -> main method format, and it wasn't hard to write a script to run them, afair. ;)

  Posted by: robilad on April 14, 2007 at 07:42 AM

• Hi David,

  I'm a bit surprised, because a lot of the features that Tonga has are already in TestNG (which, by the way, doesn't require your tests to extend a super class):

  
  Explicit list of cases in an external file (TestNG uses XML for that, text files are so... nineties :-))

  Groups are natively supported in the annotations.

  The output is HTML, XML and many other formats. It's also trivial to write your own reporter (e.g. there is a PDF reporter written by a user).

  Distribution is currently in the work via Gigaspaces.

  etc...

  
  Have you read the TestNG documentation? By the way, Glassfish is already using TestNG, you can also ask them for their feedback.

  --
  Cedric

  
  

  Posted by: cedric on April 14, 2007 at 11:25 PM

• A separate test framework is reasonable provided it adheres to the general guidelines for any test framework. Specifically:

  One button running of tests. Nothing more complex than "ant test". (It doesn't have to be ant, of course, but it can't be more complicated than that.)
  Passing tests generate no output. Only failing tests generate output.
  

  Posted by: elharo on April 17, 2007 at 04:29 AM

• Well, I'd say that no output should have to be eyeballed to check for errors; the return error code should unambiguously point of if something failed.

  A *small* amount of output from passign tests may be useful as a progress indication for example.

  Rgds

  Damon

  Posted by: damonhd on April 17, 2007 at 06:50 AM

This work is licensed under a Creative Commons License.