Test Harness for Product-Quality Test Suites

This article is oriented to developers of test suites of any kind. It provides some criteria that can be used when choosing a test harness for certain types of test suites. It describes requirements that are treated as most important for JT harness, the open source version of the JavaTest ^TM harness. The ME Framework is a JT harness plug-in for Java^TM Micro Edition (ME), driven by the same principles.

The requirements for the quality of test suites vary depending on the engineer's goal, the product being tested, how many and what kind of people will be using the test suite, how often and for how long, and how the test suite will be maintained. These factors may vary depending on whether you write unit tests for yourself or your project, or if, as a member of the SQEgroup on a large project, you are developing a test suite that will be executed by the SQA group. 

Sun Microsystems has a long successful story of development large test suites of various types. The development of the JavaTest  harness was highly influenced by its adoption in Java ^TM Technology Compatibility Kits (TCK). The majority, if not all of Sun's TCKs are built on the JavaTest harness. The key consequences of this:

• these test suites (TCKs) are products, developed by one group to be used outside of it on a constant basis for significant period of time.
• 'used in all TCKs from SUN' means that test suites built on JavaTest harness were run on all Java-compatible implementations in the world - just because implementations become Java-compatible after passing the TCK. It means a lot !
• though it is not the only area where JavaTest harness is used, the main focus is on testing of Java platform, not applications.

These test suites and a test harness have satisfy the requirements, listed below:

• Highly automated
• Documented
• Scalable
• Reliable
• Easy to debug
• Configurable

Highly automated

Test suites based on JT harness and ME Framework in Java ME case, are supposed to be executed on a continuous basis through the whole development cycle. In the case of TCKs, this means regression testing through development cycle and certification of the completed product. The more we can automate the better. The requirement for automation includes both providing a complete command line interface to the test harness and automation of the test frameworks..

Documented

Test harness and framework documentation

The importance of having good user documentation for tools like JT harness should be understood. In this case 'users' means users of the final test products, based on these tools.

Test documentation

Productized test suites are supposed to consists of well-documented tests, where all necessary information on each test can be obtained not just from the test code but from the test documentation. It is up to developers to decide whether to provide such kind of documentation for the test suite. The JT harness provides a mechanism to combine both formal test descriptions and user documentation in the same HTML document.

Scalable

Some test suites, that are built on JavaTest harness, are large. Ideally, the only impact from increasing the size of the test suite should be linear dependency on resources - either time or number of devices - that are being used for testing. There are multiple ways used to achieve this, some were described in the previous article.

Reliable

Each test suite must run to completion, which means not only that the framework code must run without errors but the test failures, if any, must not interrupt the process. Running test suites without interruption is an important quality criterion by itself but for large test suites it becomes a critical requirement. Execution of a really large test suite may take weeks, tests may run overnight. The test engineer must be provided with as many comprehensive results after the execution completes as possible.

Any resource leaks or synchronization problems in the framework code are very high priority issues.

Easy to debug

While the process of test execution is sometimes demanding, the bigger challenge is when tests fail. There are many features intended to simplify failure debugging, both in the JT harness and in the ME Framework.

Debugging-related features range from providing test sources linked to html test descriptions and browsable from JT harness, logging capabilities, possibility to execute tests standalone without a harness. The ME Framework provides the features that are most critical for Java ME, such as exporting each test to a standalone Java ME application.

Configurable

A complex test suite, that is built to test multiple products in multiple environments, may have hundreds of configuration options.  The configurability here is not just providing a way to set values for these options but a user-friendly mechanism, allowing optimization of the process and providing documentation for your test suite settings, sufficient to give a conscious answer to every configuration question.

The list is evidently not complete. Comments and additions are welcome.