 |
|
 |
 |
|
|
 
Alexey Popov's Blog
«Testing Java ME Implementations - AMS |
Main
| Testing Command-Line Applications using Golden-File approach »
Test Harness for Product-Quality Test SuitesPosted by alexeyp on January 28, 2007 at 01:07 PM | Comments (0)This article is oriented to developers of test suites of any kind. It provides some criteria that can be used when choosing a test harness for certain types of test suites. It describes requirements that are treated as most important for JT harness, the open source version of the JavaTest TM harness. The ME Framework is a JT harness plug-in for JavaTM Micro Edition (ME), driven by the same principles. The requirements for the quality of test suites vary depending on the engineer's goal, the product being tested, how many and what kind of people will be using the test suite, how often and for how long, and how the test suite will be maintained. These factors may vary depending on whether you write unit tests for yourself or your project, or if, as a member of the SQEgroup on a large project, you are developing a test suite that will be executed by the SQA group. Sun Microsystems has a long successful story of development large test suites of various types. The development of the JavaTest harness was highly influenced by its adoption in Java TM Technology Compatibility Kits (TCK). The majority, if not all of Sun's TCKs are built on the JavaTest harness. The key consequences of this:
Highly automatedTest suites based on JT harness and ME Framework in Java ME case, are supposed to be executed on a continuous basis through the whole development cycle. In the case of TCKs, this means regression testing through development cycle and certification of the completed product. The more we can automate the better. The requirement for automation includes both providing a complete command line interface to the test harness and automation of the test frameworks..DocumentedTest harness and framework documentationThe importance of having good user documentation for tools like JT harness should be understood. In this case 'users' means users of the final test products, based on these tools.Test documentationProductized test suites are supposed to consists of well-documented tests, where all necessary information on each test can be obtained not just from the test code but from the test documentation. It is up to developers to decide whether to provide such kind of documentation for the test suite. The JT harness provides a mechanism to combine both formal test descriptions and user documentation in the same HTML document.ScalableSome test suites, that are built on JavaTest harness, are large. Ideally, the only impact from increasing the size of the test suite should be linear dependency on resources - either time or number of devices - that are being used for testing. There are multiple ways used to achieve this, some were described in the previous article.ReliableEach test suite must run to completion, which means not only that the framework code must run without errors but the test failures, if any, must not interrupt the process. Running test suites without interruption is an important quality criterion by itself but for large test suites it becomes a critical requirement. Execution of a really large test suite may take weeks, tests may run overnight. The test engineer must be provided with as many comprehensive results after the execution completes as possible.Any resource
leaks or synchronization problems in the framework code are very high
priority issues. Easy to debugWhile the process of test execution is sometimes demanding, the bigger challenge is when tests fail. There are many features intended to simplify failure debugging, both in the JT harness and in the ME Framework.Debugging-related features range from providing test sources linked to html test descriptions and browsable from JT harness, logging capabilities, possibility to execute tests standalone without a harness. The ME Framework provides the features that are most critical for Java ME, such as exporting each test to a standalone Java ME application. ConfigurableA complex test suite, that is built to test multiple products in multiple environments, may have hundreds of configuration options. The configurability here is not just providing a way to set values for these options but a user-friendly mechanism, allowing optimization of the process and providing documentation for your test suite settings, sufficient to give a conscious answer to every configuration question.The list is evidently not complete. Comments and additions are welcome.
Bookmark blog post:  del.icio.us  Digg  DZone  Furl  Reddit
CommentsComments are listed in date ascending order (oldest first) | Post Comment | |
 java.net RSS Feeds
|
