Skip to main content

Introduction to Java ME Testing Tools

Posted by alexeyp on November 27, 2006 at 1:21 AM PST

This blog is based on and is about my and my colleagues'
experience in JavaTM ME testing tools and test
suite development.
It is inspired by SUN open-sourcing href="https://phoneme.dev.java.net/">Java ME software stack and
Java testing tools.

Implementation Testing vs. Application Testing

Java ME developers perform their
application testing
routinely with unit tests written as part of their
development cycle.
J2MEUnit
and JMUnit
work well for this: all you need to do is
to package the test framework and all your tests with your application
and
provide an entry point to start test execution. These frameworks run
your tests in a single MIDlet run and display results on the device
screen.
This is all what the most of the application developers need.

Things change when the object of your testing is a Java ME
implementation on a mobile device (that is the entire Java ME software
stack). We will talk below about the specific challenges that
complicate the testing process.
Some of these are specific to Java ME platform, others steam from the
size and complexity of the technology.

Implementation Testing Tools

A number of tools were developed over the years to address
the implementation testing needs. Some URLs:

Dealing with Large Test Suites

The first issue to deal with is the test suite scalability.
Due to the variety and complexity of Java ME technologies,
the number of tests is large.

Let's say we wrote 10000 tests for the CLDC VM specification.
If we tried to execute them in a JUnit-like test framework,
we would quickly find out that

  • 10000 tests do not fit into a single application, you have
    to split them in smaller chunks.
  • some tests crash the VM. If we tried to execute all tests
    in a single MIDlet.startApp call, we would never see the result of any
    test (even those which don't crash). You need to be able to detect VM
    exits and catch any unexpected exceptions.
  • some tests crash the VM on purpose. These are negative
    tests which verify that VM handles certain conditions properly (e.g.
    throws VerifyError).
  • device runs out of memory because all class files of the
    test suite do not fit into it (garbage collector frees unused data from
    the memory, but doesn't free classes: these are garbage-collected only
    together with their class loader).
  • tests running in the same VM interfere with each other
    causing failures which are hard to explain or debug. Or tests misbehave
    and lock up the VM (this is where you learn that shared resources and
    race conditions are a big deal in the testing world).
  • you need an accurate report once the test run is complete:
    if 100 tests out of 10000 failed, you need detailed information on
    every failure and you don't want it displayed on the device screen.
    There should be a way to load test results from the device onto your
    server or desktop machine. In a perfect world test results for every
    test cycle are stored in an easily accessible database.

Since tests are typically run on a device with unstable
software
stack, execution of a large test suite there is an additional
challenge. This is equally true for much more capable platforms like
Java SE. One should expect all sorts of things to go wrong:

  • Implementation bugs causing failures and crashes.
  • Resource leaks resulting in memory overflow, unavailable
    file handlers and sockets.
  • Incomplete implementation lacking networking or user
    interface support.

There are many other problems to be solved.
For example, how do you automate testing of your Java-enabled Blu-ray
player?

Let's talk about these and other problems on specific examples.


CDC and Java SE Testing Solutions

Java SE platform is where Java technology originated. Not
surprisingly, this is where the testing solutions were initially worked
out.
CDC + Foundation Profile is a platform similar in many ways to Java SE
(but without the myriad of recent extensions and features), the
approach, described below, came from Java SE and works for CDC+FP.

CDC platform is richer compared with CLDC. The most important
benefits we have is reflection APIs and user-defined class loaders.
This makes many testing tasks easier.

The most robust testing set-up proved to be a client-server
model
for the test harness: the main harness application runs in a stable
Java SE environment and a small agent is deployed on the device under
test.
The main harness application manages test suite configuration and
execution,
as well as the test result collection and storage.
The agent connects to the harness application over the network. All the
agent does is repeatedly download and execute tests (a separate class
loader instance is created for each test download) as well
as send test results back to the main application.

This effectively solves most of the problems outlined above:

  • Small memory requirements. All you need is enough memory
    to hold the agent application and the largest test. Test classes are
    garbage-collected each time test class loader is garbage-collected, so
    you are not limited by the size of a test suite.
  • Test interference is minimized. Since tests are loaded
    one-by-one, they do not affect each other and their resources are freed
    once execution is complete.
  • No multiple application downloads during the test cycle.
    There is only one small
    agent application to download. The rest is handled automatically.
  • No test results on device. You are not dependent on file
    system, you don't do detective work if the device dies. By the way,
    file system independence has the added benefit of nicely handling the
    security-constrained environments such as an applet.

There are several critical dependencies in this model: you
need a (stable) network, class loader and a few extra security
permissions. These may not be available everywhere by default, but this
is topic for different discussion.

CLDC and MIDP Testing Solutions

Things get more difficult on a CLDC/MIDP stack.
There is no class loader to rely on (and no reflection either).
The only universal way to load test code on the device is via
application download. While AMS (Application Management System) does
not support automation
by default, a very simple extension called "autotest" satisfies
most testing needs.

The autotest feature is a special AMS mode which iteratively
downloads, installs, runs and removes applications from a specified
URL. Test harness makes new test applications available at this URL
and the AMS picks them up.

10000 tests will be packaged into a number of test
applications.
Each application will include a small execution agent (to report test
result back to the test harness) and the test classes.
In a trivial case we will have 10000 different applications.

Shortening the Test Cycle

Test applications are downloaded over the network.
The good news for MIDP is that all devices support HTTP download.
The bad news is that some of these connections are as slow
as 9 kbit/s. It will take 2.5 hours to download 10 MB of test classes.

And there are many other things which will affect the test run
time.
Even though execution agent is small, it is bundled with every
application, wasting a lot of network traffic!
Each test bundle download requires new a network connection; these will
slow down the execution.
Test data and test results are sent back and forth between the harness
and the test application; extra network bandwidth is consumed.
Each test needs time to execute: some tests do extensive computations
and will take a few minutes to run.

Since the system is distributed, one must be minimally taking care of
proper task scheduling. For example, test applications must 
already be packaged and signed when request for the next application
comes - if you start packaging task only on getting request
for
the next test application download, it may have surprisingly
sad
effect on execution time.

In the early days of CLDC TCK 1.0 (test suite, used for
certification of implementations of CLDC 1.0 specification) test
execution was taking up to 2
weeks in some environments!
With a few improvements, one can pass CLDC TCK in a few hours:

  • Pre-install shared agent code on the device (romize
    classes of the test agent).
  • Package several tests per downloaded application to reduce
    the number of network connections and AMS operations. Each device has a
    different maximum application size so the total number of applications
    will vary.
  • Use multiple devices in parallel. Let the harness assemble
    and sort test results.

What else?

Described test execution models have certain restrictions. As example:
using Java ME application to run tests does not allow to test AMS
behaviour, that is primary focus for some Java ME specs. This specific
problem has more then one solution already, as well as there are
solutions that allow to run interactive, distributed tests for Java ME.

Comments are welcome.

Related Topics >>