This blog is based on and is about my and my colleagues' experience in Java^TM ME testing tools and test suite development. It is inspired by SUN open-sourcing Java ME software stack and Java testing tools.

Implementation Testing vs. Application Testing

Java ME developers perform their application testing routinely with unit tests written as part of their development cycle. J2MEUnit and JMUnit work well for this: all you need to do is to package the test framework and all your tests with your application and provide an entry point to start test execution. These frameworks run your tests in a single MIDlet run and display results on the device screen. This is all what the most of the application developers need.

Things change when the object of your testing is a Java ME implementation on a mobile device (that is the entire Java ME software stack). We will talk below about the specific challenges that complicate the testing process. Some of these are specific to Java ME platform, others steam from the size and complexity of the technology.

Implementation Testing Tools

A number of tools were developed over the years to address the implementation testing needs. Some URLs:

• JCP TCK tools
• JT harness (an open source test harness)
• ME Framework (JT harness-based open source testing framework for Java ME platform)
• Java Device Test Suite (device quality test suite)

Dealing with Large Test Suites

The first issue to deal with is the test suite scalability. Due to the variety and complexity of Java ME technologies, the number of tests is large.

Let's say we wrote 10000 tests for the CLDC VM specification. If we tried to execute them in a JUnit-like test framework, we would quickly find out that

• 10000 tests do not fit into a single application, you have to split them in smaller chunks.
• some tests crash the VM. If we tried to execute all tests in a single MIDlet.startApp call, we would never see the result of any test (even those which don't crash). You need to be able to detect VM exits and catch any unexpected exceptions.
• some tests crash the VM on purpose. These are negative tests which verify that VM handles certain conditions properly (e.g. throws VerifyError).
• device runs out of memory because all class files of the test suite do not fit into it (garbage collector frees unused data from the memory, but doesn't free classes: these are garbage-collected only together with their class loader).
• tests running in the same VM interfere with each other causing failures which are hard to explain or debug. Or tests misbehave and lock up the VM (this is where you learn that shared resources and race conditions are a big deal in the testing world).
• you need an accurate report once the test run is complete: if 100 tests out of 10000 failed, you need detailed information on every failure and you don't want it displayed on the device screen. There should be a way to load test results from the device onto your server or desktop machine. In a perfect world test results for every test cycle are stored in an easily accessible database.

Since tests are typically run on a device with unstable software stack, execution of a large test suite there is an additional challenge. This is equally true for much more capable platforms like Java SE. One should expect all sorts of things to go wrong:

• Implementation bugs causing failures and crashes.
• Resource leaks resulting in memory overflow, unavailable file handlers and sockets.
• Incomplete implementation lacking networking or user interface support.

There are many other problems to be solved. For example, how do you automate testing of your Java-enabled Blu-ray player?

Let's talk about these and other problems on specific examples.

CDC and Java SE Testing Solutions

Java SE platform is where Java technology originated. Not surprisingly, this is where the testing solutions were initially worked out. CDC + Foundation Profile is a platform similar in many ways to Java SE (but without the myriad of recent extensions and features), the approach, described below, came from Java SE and works for CDC+FP.

CDC platform is richer compared with CLDC. The most important benefits we have is reflection APIs and user-defined class loaders. This makes many testing tasks easier.

The most robust testing set-up proved to be a client-server model for the test harness: the main harness application runs in a stable Java SE environment and a small agent is deployed on the device under test. The main harness application manages test suite configuration and execution, as well as the test result collection and storage. The agent connects to the harness application over the network. All the agent does is repeatedly download and execute tests (a separate class loader instance is created for each test download) as well as send test results back to the main application.

This effectively solves most of the problems outlined above:

• Small memory requirements. All you need is enough memory to hold the agent application and the largest test. Test classes are garbage-collected each time test class loader is garbage-collected, so you are not limited by the size of a test suite.
• Test interference is minimized. Since tests are loaded one-by-one, they do not affect each other and their resources are freed once execution is complete.
• No multiple application downloads during the test cycle. There is only one small agent application to download. The rest is handled automatically.
• No test results on device. You are not dependent on file system, you don't do detective work if the device dies. By the way, file system independence has the added benefit of nicely handling the security-constrained environments such as an applet.

There are several critical dependencies in this model: you need a (stable) network, class loader and a few extra security permissions. These may not be available everywhere by default, but this is topic for different discussion.

CLDC and MIDP Testing Solutions

Things get more difficult on a CLDC/MIDP stack. There is no class loader to rely on (and no reflection either). The only universal way to load test code on the device is via application download. While AMS (Application Management System) does not support automation by default, a very simple extension called "autotest" satisfies most testing needs.

The autotest feature is a special AMS mode which iteratively downloads, installs, runs and removes applications from a specified URL. Test harness makes new test applications available at this URL and the AMS picks them up.

10000 tests will be packaged into a number of test applications. Each application will include a small execution agent (to report test result back to the test harness) and the test classes. In a trivial case we will have 10000 different applications.

Shortening the Test Cycle

Test applications are downloaded over the network. The good news for MIDP is that all devices support HTTP download. The bad news is that some of these connections are as slow as 9 kbit/s. It will take 2.5 hours to download 10 MB of test classes.

And there are many other things which will affect the test run time. Even though execution agent is small, it is bundled with every application, wasting a lot of network traffic! Each test bundle download requires new a network connection; these will slow down the execution. Test data and test results are sent back and forth between the harness and the test application; extra network bandwidth is consumed. Each test needs time to execute: some tests do extensive computations and will take a few minutes to run.

Since the system is distributed, one must be minimally taking care of proper task scheduling. For example, test applications must  already be packaged and signed when request for the next application comes - if you start packaging task only on getting request for the next test application download, it may have surprisingly sad effect on execution time.

In the early days of CLDC TCK 1.0 (test suite, used for certification of implementations of CLDC 1.0 specification) test execution was taking up to 2 weeks in some environments! With a few improvements, one can pass CLDC TCK in a few hours:

• Pre-install shared agent code on the device (romize classes of the test agent).
• Package several tests per downloaded application to reduce the number of network connections and AMS operations. Each device has a different maximum application size so the total number of applications will vary.
• Use multiple devices in parallel. Let the harness assemble and sort test results.

What else?

Described test execution models have certain restrictions. As example: using Java ME application to run tests does not allow to test AMS behaviour, that is primary focus for some Java ME specs. This specific problem has more then one solution already, as well as there are solutions that allow to run interactive, distributed tests for Java ME.

Comments are welcome.