Test Harness for Product-Quality Test Suites
This article is oriented to developers of test suites of any kind.
It provides some criteria that can be
used when choosing a test harness for certain types
of test suites. It describes
requirements that are treated as most important for href="https://jtharness.dev.java.net/">JT
harness, the open source version of the JavaTest TM
harness. The ME
Framework is a JT harness plug-in for JavaTM
Micro Edition (ME), driven by the same principles.
The requirements for the quality of test suites vary depending
engineer's goal, the product being tested, how many and what kind of
people will be using the test suite, how often and for how long, and
how the test suite will be maintained. These factors may vary depending
on whether you write unit tests for yourself or your project, or if, as
a member of the SQEgroup on a large
project, you are developing a test suite that will be executed by the
Sun Microsystems has a long successful story of
test suites of various types.
The development of the JavaTest harness
its adoption in Java TM Technology
Compatibility Kits (TCK). The majority, if not all of Sun's
TCKs are built on the JavaTest harness. The key consequences of
- these test suites (TCKs) are
developed by one group to be
used outside of it on a constant basis for significant
period of time.
- 'used in all
TCKs from SUN' means that test suites built on JavaTest
harness were run on all Java-compatible implementations in the world -
because implementations become Java-compatible after passing the TCK.
It means a
- though it is not the only area where JavaTest harness is
the main focus is on testing of Java platform, not applications.
These test suites and a test harness have satisfy the requirements,
Test suites based on JT harness and ME Framework in Java ME
supposed to be
executed on a continuous basis through the whole development cycle. In
of TCKs, this means regression testing through development cycle and
certification of the completed product. The more we can automate the
better. The requirement for automation includes both providing
command line interface to the test harness and automation of the test
Test harness and framework documentation
The importance of having good user documentation for tools like JT
be understood. In this case 'users' means users of the final test
based on these tools.
Productized test suites are supposed to consists of well-documented
tests, where all necessary information on each test can be
obtained not just from the test code but from the test
documentation. It is up to developers to decide whether to
provide such kind of
documentation for the test suite. The JT harness provides a
mechanism to combine both formal test descriptions and user
documentation in the same HTML document.
Some test suites, that are built on JavaTest harness, are large.
only impact from increasing the size of the test suite should be linear
dependency on resources - either time or number of devices - that are
being used for testing. There are multiple ways used to achieve this,
described in the href="http://weblogs.java.net/blog/alexeyp/archive/2006/11/testing_java_me_1.html">previous
Each test suite must run to completion, which means not only that the
framework code must run without errors but the test failures, if any,
must not interrupt the process. Running test suites without
interruption is an important quality criterion by itself but for large
test suites it becomes a critical requirement. Execution of a really
large test suite may take weeks, tests may run overnight. The
test engineer must be provided with as many comprehensive results after
the execution completes as possible.
leaks or synchronization problems in the framework code are very high
While the process of test execution is sometimes demanding, the bigger
challenge is when tests fail. There are many features intended to
failure debugging, both in the JT harness and in the ME Framework.
Debugging-related features range from providing test sources
linked to html test
descriptions and browsable from JT harness, logging capabilities,
possibility to execute tests standalone without a harness. The
ME Framework provides the features that are most critical for Java
ME, such as exporting each test to a standalone Java ME application.
A complex test suite, that is built to test multiple products
multiple environments, may have hundreds
of configuration options. The configurability here is not
just providing a way to set values for these options but a
user-friendly mechanism, allowing optimization of the process and
providing documentation for your test suite settings, sufficient to
give a conscious answer to every
The list is evidently not complete. Comments and additions are