Skip to main content

Test Harness for Product-Quality Test Suites

Posted by alexeyp on January 28, 2007 at 1:07 PM PST

This article is oriented to developers of test suites of any kind.
It provides some criteria that can be
used when choosing a test harness for certain types
of test suites. It describes
requirements that are treated as most important for href="">JT
harness, the open source version of the JavaTest TM
harness. The ME
is a JT harness plug-in for JavaTM
Micro Edition (ME), driven by the same principles.

The requirements for the quality of test suites vary depending
on the
engineer's goal, the product being tested, how many and what kind of
people will be using the test suite, how often and for how long, and
how the test suite will be maintained. These factors may vary depending
on whether you write unit tests for yourself or your project, or if, as
a member of the SQEgroup on a large
project, you are developing a test suite that will be executed by the
SQA group. 

Sun Microsystems has a long successful story of
test suites of various types.
The development of the JavaTest  harness
was highly
influenced by
its adoption in Java TM Technology
Compatibility Kits (TCK). The majority, if not all of Sun's
TCKs are built on the JavaTest harness. The key consequences of

  • these test suites (TCKs) are style="font-style: italic;">products,
    developed by one group to be
    used outside of it on a constant basis for significant
    period of time.
  • 'used in all
    TCKs from SUN
    ' means that test suites built on JavaTest
    harness were run on all Java-compatible implementations in the world -
    because implementations become Java-compatible after passing the TCK.
    It means a
    lot !
  • though it is not the only area where JavaTest harness is
    the main focus is on testing of Java platform, not applications.

These test suites and a test harness have satisfy the requirements,
listed below:


Test suites based on JT harness and ME Framework in Java ME
case, are
supposed to be
executed on a continuous basis through the whole development cycle. In
the case
of TCKs, this means regression testing through development cycle and
certification of the completed product. The more we can automate the
better. The requirement for automation includes both providing
a complete
command line interface to the test harness and automation of the test


Test harness and framework documentation

The importance of having good user documentation for tools like JT
harness should
be understood. In this case 'users' means users of the final test
based on these tools.

Test documentation

Productized test suites are supposed to consists of well-documented
tests, where all necessary information on each test can be
obtained not just from the test code but from the test
documentation. It is up to developers to decide whether to
provide such kind of
documentation for the test suite. The JT harness provides a
mechanism to combine both formal test descriptions and user
documentation in the same HTML document.


Some test suites, that are built on JavaTest harness, are large.
Ideally, the
only impact from increasing the size of the test suite should be linear
dependency on resources - either time or number of devices - that are
being used for testing. There are multiple ways used to achieve this,
some were
described in the href="">previous


Each test suite must run to completion, which means not only that the
framework code must run without errors but the test failures, if any,
must not interrupt the process. Running test suites without
interruption is an important quality criterion by itself but for large
test suites it becomes a critical requirement. Execution of a really
large test suite may take weeks, tests may run overnight. The
test engineer must be provided with as many comprehensive results after
the execution completes as possible.

Any resource
leaks or synchronization problems in the framework code are very high
priority issues.

Easy to debug

While the process of test execution is sometimes demanding, the bigger
challenge is when tests fail. There are many features intended to
failure debugging, both in the JT harness and in the ME Framework.

Debugging-related features range from providing test sources
linked to html test
descriptions and browsable from JT harness, logging capabilities,
possibility to execute tests standalone without a harness. The
ME Framework provides the features that are most critical for Java
ME, such as exporting each test to a standalone Java ME application.


A complex test suite, that is built to test multiple products
multiple environments, may have hundreds
of configuration options.  The configurability here is not
just providing a way to set values for these options but a
user-friendly mechanism, allowing optimization of the process and
providing documentation for your test suite settings, sufficient to
give a conscious answer to every
configuration question.

The list is evidently not complete. Comments and additions are

Related Topics >>