Alexey Popov's Blog: December 2006 Archive

'Server/Agent' approach, used for CDC implementations, where test code is downloaded by Agent application from the Server
'autotest' approach, where tests are packaged into sequence of applications, that are repeatedly downloaded/executed/removed.

Please make sure to familiarize yourself with these concepts, I will be referring to them later.

Today I would like to focus on limitations of these approaches, what types of functionality can not be tested this way and describe several solutions. Some of these ideas are implemented in ME Framework, there are variations, implemented in other Java ME testing products.

Talking about limitations, the main point is that using these mechanisms one can verify what is going inside of a single application when it is running. These mechanisms do not allow application restarts and crashes, assume fixed scenario and only single application running at any given time.
Note, that this is usually all that is needed to test of Java API behavior. Just it (API) is not the only focus of standards and testing in the Java ME world.

Examples of Java ME standards, that can not be tested with the above mentioned approaches, are as follows:

MIDP OTA specification, among other things, describes criteria, that result in an application installation failure. Tests for these requirements need to meet these criteria and verify that installation fails. API testing mechanisms work only if install&run steps pass successfully.
MIDP PushRegistry specification talks about static registration of the Push event handlers. If application is provided with special attributes, it gets registered in the PushRegistry to handle incoming WMA messages, for example. Testing of this functionality assumes the following steps are present in the test procedure:
- an application with Push registration is installed
-the push event is initiated
- application is launched to handle the event
The scenario does not fit into the Server/Agent scheme, because it requires an application installation to happen as part of the test, and does not fit into the autotest scheme, because it does not have automatic 'run/remove' steps, that are mandatory parts of the 'autotest' cycle.
CHAPI spec talks about communications between two applications, the ContentHandler and Invoker.

Overall, there is a considerable number of JCP and non-JCP standards that focus not only on API but on applications. Testing of these standards requires non-standard techniques.

The cases above are usually associated with the platform component, named JAM (Java Application Manager) or AMS (Application Management Software), therefore the testing techniques that involve AMS operations (install/update/run/remove) or application life cycle (start/pause/stop/destroy) will be referred to as 'AMS testing'.

OTA Testing Framework

Absolute Weapon

OTA Testing Framework is an established terminology for the technique, that was first time used to test the OTA specification. Below is the description of this specific case first, followed by some analysis and discussion of potential alternatives.

Typically, the OTA Testing Framework consists of the following components:

OTA test - the test scenario, that includes standard and custom AMS operations. It operates with applications associated with this test. It is executed on the server side of the system, main communication point responsible for reporting the whole test status.
Test application(s) - zero or more pre-packaged applications that may contain part of the test code that needs to be executed on the device. They may communicate with the main test scenario - OTA test

OTA test is executed in this framework on the test harness side and uses OTA server to publish applications:
OTA test and test applications

There is also an important component called OTA Server. In MIDP case it is HTTP server controlled by OTA test and responsible for provisioning of test applications. Nice artistic picture of the OTA Testing Framework in provided in the ME Framework Developer's Guide (link to PDF version), check Figure 3-7.

The approach described above is universal and powerful. It can be used to test the most exotic cases. At least, so far it stays as a last absolute weapon used if no other technique works.

Last Thing to Use

The only disadvantage of OTA Testing Framework is that it is terribly interactive. During the certification run TCKs do not allow to use any automation, every AMS operation must be performed manually. With a simple install/run/remove cycle, where one needs to read instructions, enter url, wait for download/launch/execution complete/remove, it takes 1 minute minimum for a test case.

Now imagine yourself a QA engineer, who has to run CHAPI TCK, that contains ~40 OTA tests, as a regression test suite on a regular basis during the whole development cycle without any automation.

Lets consider a variation to the above approach, that can be used when it is OK to narrow the scope of covered possibilities and simplify the usage.

XletManager

This is the approach, used initially in PBP & TV TCKs to verify parts of the specifications, related to an application life cycle. In PBP case it was also for testing Ixc - Inter Xlet Communication.

The idea is to specify the Java interface to the AMS that will allow one application to perform the life cycle operations on another application. It was created for PBP/TV, where testing is based on the Server/Agent model. The scheme works in the following way:

An implementation developer provides an implementation of XletManager interface, that would allow a test application, executed on the device to invoke the AMS operations through the Java API. In the general case, if Java interface to AMS is not available, the XletManager implementation may be interactive.
Test Suite user manually downloads and installs on the device the Agent application and a set of Xlet applications that will be used in tests
Tests that are executed within the Agent instantiate XletManager and operate with additional pre-installed test applications.

While in the PBP TCK this approach is used to initiate only life cycle operations from the test, executed on the device, it also can be extended to any AMS operations like install/update/run/remove.

Comparing with OTA Test Framework

Main benefit of XletManager-like approach is that it allows to use a single Server/Agent test execution model for the whole test suite.

With this specific approach, there is no dynamic application installation/removal, no application provisioning component, test applications are preinstalled before test execution begins.

There may be restrictions, that will complicate the use of this model or just make it impossible, like:

interactive implementation of XletManager may be complicated for devices with small screen
platform may not allow more than one application to run simultaneously.

Variations

There are several active components involved into execution of the test, there may be several places where to put the main scenario responsible for execution of AMS operations. It may be server (OTA Test Framework) or client side (XletManager-like) of the distributed test. It can be AMS itself - 'autotest' mode, that AMS is supposed to provide for an automated execution of test suites on CLDC/MIDP, is also a scenario of AMS operations, that can be used for test purpose.

Interface to the AMS may be implemented through plugins, that may be provided by implementation developers. These plugins may be interactive in general case or automated. Automated testing may be used for QA/regression testing or even for certification in some cases.

There are interesting possibilities of using standard Java ME APIs in order to communicate with AMS, for example CHAPI. As a minimum, it can be used to have the test code distributed across multiple MIDlets or MIDlet-suites that are registered as ContentHandler-s and invoked from the main test scenario.

Summary

Over time, the number and complexity of Java ME implementation tests dealing with AMS have increased. I believe this is a natural consequence of consistent focus on standards in the area of application management for consumer devices.

The interesting and important problem, that comes together with number of appearing and evolving AMS-testing frameworks that are often interactive is automation, it deserves separate article.

Merry Christmas and Happy New Year !

Powered by
Movable Type 3.01D

java.net RSS Feeds