Alexey Popov's Blog

Community: Embedded Java Archives

Interactive Tests for Java ME

Most of definitions and examples are given using the terminology of JT harness and ME Framework, but should be generic enough for the area. For details on what is TCK, JT harness and ME Framework refer to previous articles. Skip the background section if you are familiar with the subject.

Background

Why not everything is automated

For every specific implementation, testing of this functionality can be automated as well with different external tools or specific APIs. But first, not always - compatibility tests, for example, can not rely on any specifics because have to be correct for any compatible implementation, and second, this is a separate big topic, to be covered in another article.

What is meant by interactive tests

Requirements

Speaking from point of view of TCK development, the most important requirement is to make the test suite easiest for use. This means the less interactive tests it has, the better. If we have to have interactive tests, it is important to make their execution simple.

To achieve this we limited number of test types that we use in TCKs to very few. As a result, some features, that are not best suited to be tested by these test types, require multiple test cases to be written where we would write one 'custom' test case. The benefit is that TCK user has uniform interactive model, uniform interface to browse test instructions etc. This also allows for simpler external automation system development.

Types of TCK interactive tests

Note that these tests are not just set of user instructions 'do this - verify that', they include test code that can do most of work.

Java ME Specifics and Solutions

PJava story - first TCK alt bundle

Interactive tests created for Java SE TCKs are usually interactive applications, that run on the platform under test and show user instructions and test panel in a single window. This approach does not always work for Java ME for many reasons. First time we started doing interactive tests in the world of consumer devices, that was Personal Java, we found that tests we created can not be passed on PJava devices with single Frame limitation and small screen - test instructions and test panel were placed in the container without scrolling capabilities, these 'Done' and 'Yes/No' buttons may not appear on screen in some circumstances. To address this we issued first 'alternative TCK test bundle' that just enabled scrolling. After that passing of interactive tests became possible, though not convenient. Check how AgentFrame interface looks like on PJava with Truffle toolkit. The scheme of these interactive tests is the one that is still used in Java SE TCKs, can be described as follows:

Figure I. Simple Interactive Test.

MIDP TCK 1.0 interactive tests

Interactive tests for MIDP TCK 1.0 were executed using the same Autotest mechanism as regular automated tests, the difference was that interactive tests expected some user actions and were grouped together for convenience. These tests were developed using brand-new MIDP API and same approach that was used in JCK and PJCK. Instructions, test panel, all user interface components of these tests were displayed on the micro screen of MIDP 1.0 micro devices. Given big number of interactive tests, that were necessary to verify MIDP 1.0 GUI API, running MIDP 1.0 TCK on a regular basis during the development process, was a headache.

You can see the screen shot of the MIDP 1.0 emulator to the right. Click on it to see the sequence of screens that constituted the MIDP TCK 1.0 interactive test. As you can see, there is a big number of interactions, that are not related to execution of the test but for scrolling, switching controls etc.

Distributed Interactive test framework

MIDP TCK 2.x interactive tests

To address this problem of inconvenience of interactive tests on MIDP devices with small screen, the solution was very simple and usability improvement was huge. Briefly, these tests were rewritten to using Distributed Test library to have minimal functionality on the device and have user instructions and controls, related to the test logic, on the server part. Now to run tests on the device one need to stare to the desktop monitor, read instructions, press some buttons on the desktop, for example, to initiate tested process on the device, do some interaction with the device as necessary, state pass/fail result on the desktop if needed.

The important feature of Distributed Test framework, that lies underneath new Interactive Test Framework is that there are java components of the test, that reside on server and client sides and can work together to calculate test result. One can use server side technologies in the test, for example, for reference purpose to verify that same technology works properly at Java ME side.

The scheme of distributed interactive test can be described as follows

:

Figure II. Distributed Interactive Test.

In the example of the interactive test for sound, you can see all GUI of this test. Device part of this specific test does not have GUI at all, all that device does - produces sounds that are initiated from the server side, pass/fail criteria is specified on the server side of the test as well.

PBP TCK 1.0 interactive tests

Same solution was used for interactive tests for Personal Basis Profile TCK. Though PBP API is subset of J2SE API, reuse of JCK tests was not possible there - as I mentioned, these tests combine in a single application all instructions and controls, that were not available. PBP does not have Panel, Button - no any UI widgets, only Component, Container and Window Frame. To reproduce anything on the device screen we would have to draw it using graphics primitives from scratch and creating UI toolkit was not in the scope of the TCK.

Separating functionality and interaction between server and client parts of the tests, that were created using Distributed Test framework, worked well for PBP. In the example of the keyboard test you can see the same situation as in already referenced MIDP TCK 2.0 example, all GUI is on desktop screen, device does not have any GUI, just accepts key presses and pass them to the server side.

AGUI TCK 1.0 interactive tests

Yet another special solution to workaround small device screen was implemented for AGUI TCK 1.0.

AGUI stands for Advanced Graphics and User Interface, that assumes lots of interactive tests. For AGUI the execution model for interactive tests was also the same as for regular automated tests, all tests were executed in the same Agent, interactive tests grouped separately from automated. Read here about MIDP and CDC execution modes of ME Framework.

As AGUI API is a subset of Java SE API, specifically, Swing, our goal was to reuse as much of JCK interactive tests for this API as possible. The reason here is not only time saving, it is also an additional way to ensure compatibility across different Java platforms.

What we did is we again separated tests to different pieces that could be displayed separately and organized test UI as Tabs. It was relatively easy to do, since JCK interactive tests library assumed some structuring, there was implementation of this library done with Swing subset, that fit into AGUI API. Though it was still necessary to do extra clicks to switch between different tabs and sometimes scroll through the tab, the effect was a significant usability improvement comparing to having everything in the same window altogether.

You can see the screen shot of the AGUI 1.0 emulator to the left. Click on it to see the sequence of screens that constituted the AGUI TCK 1.0 interactive test.

It was a temporary solution, once we structured tests to independent pieces, it was easy to execute these components on distributed components. This was not a conversion of simple interactive tests to distributed tests but a special execution framework, that could execute simple interactive tests in the distributed way and gave some other minor usability improvements. Overall, Interactive tests in the AGUI TCK 1.0 when it was released looked exactly like all other distributed interactive tests for Java ME.

Tests with static image

This approach is natural one, it can be used to test 80% of functionality, related to user interaction. Some technologies use it as the only approach for testing, for example W3C SVG test suite. It can be used to test low level graphics and behavior of high-level user interface components.

Even when these tests are too primitive and require multiple test cases to be developed when few more sophisticated ones could be enough, the simplicity of development and execution, possibility to have consistency across many tests could be a reason to use this approach even when it is not the most convenient.

Static image tests with ME Framework

To simplify test execution we display multiple images, related to the same functionality, in a single window with test instructions. Every reference image is accompanied with a Test button, that initiates reproduction of this image on the device under test, ad verbal description. Check the example, the screenshot of the instructions dialog and device side for tests for Java Binding to Open GL ES API (JSR 239) .

Combining multiple related images that could be switched in any sequence allows not only for comparison of the result on the screen with reference, but also for checking of transition one image to another on the device. This, as well as verbal description of the scene, may be important when reference images were done on the implementation, that is radically different with one under test and comparing of test and reference image alone can not provide confidence that tested functionality behaves correctly.

One of the First Java Applications

Few words for a background.

The Java ^TM Compatibility Kit (JCK) is a test suite, used to verify if the Java standard is correctly implemented. The first JCK came out together with the first JDK from SUN, now this effort evolved into the industry-wide Java Community Process.

The JavaTest ^TM harness is a test monitor, used in first versions of JCK, then across multiple Java Technology Compatibility Kits (TCK), now evolved into a general purpose open source test harness.

How it started

With permission of Jonathan Gibbons, who was around when Java was born, here is his story of the JavaTest childhood:

JavaTest started round about JDK 1.0.2, JCK 1.0.2a was applet-based and did not use JavaTest;  JavaTest was introduced in JCK 1.0.2b.

It definitely was not the first application written in Java, but it is true that testing was important to Java from the beginning. JavaTest is sufficiently old that we had to develop many GUI widgets ourselves, and one of the early JavaTest developers (Tim Prinzing) went on to become a significant member in the Swing team. It is probably reasonable to say that Tim pioneered light-weight components.

Curious how it looked like back then at early JDK times ? Check here.

You can see that testing is one of keys that brought Java platform to where it is now. Java's way of compatibility testing, that is having high quality TCK required to pass to get Java logo,  is how Write Once Run Anywhere is achieved,  it is the cost of application portability and platform standardization.

On the road

Inspired by the conversation at David Herron's Blog, I was looking for analogy to illustrate the value of compatibility testing in the Java ecosystem, here it is - Java is a road, compatibility testing is its pavement. Applications are vehicles and the hard cover on the road makes it possible to drive fast. 

For Java ME platform, its fragmentation is a freedom to choose number of wheels, engine, use a bicycle or a truck. Another part, the 'dark side', is where the road cover has holes, caused by bugs in specification and implementations, test coverage problems. Services are built around the road to take care of vehicles and holes, like JDTS and Java Verified, while the road has its pavement (TCKs), differentiating it from deserts, forests and swamps.

Testing Java ME Implementations - AMS

API testing vs AMS testing

• 'Server/Agent' approach, used for CDC implementations, where test code is downloaded by Agent application from the Server
• 'autotest' approach, where tests are packaged into sequence of applications, that are repeatedly downloaded/executed/removed.

Today I would like to focus on limitations of these approaches, what types of functionality can not be tested this way and describe several solutions. Some of these ideas are implemented in ME Framework, there are variations, implemented in other Java ME testing products.

Talking about limitations, the main point is that using these mechanisms one can verify what is going inside of a single application when it is running. These mechanisms do not allow application restarts and crashes, assume fixed scenario and only single application running at any given time.
Note, that this is usually all that is needed to test  of Java API behavior. Just it (API) is not the only focus of standards and testing in the Java ME world.

Examples of Java ME standards, that can not be tested with the above mentioned approaches, are as follows:

• MIDP OTA specification, among other things, describes criteria, that result in an application installation failure. Tests for these requirements need to meet these criteria and verify that installation fails. API testing mechanisms work only if install&run steps pass successfully.
• MIDP PushRegistry specification talks about static registration of the Push event handlers. If application is provided with special attributes, it gets registered in the PushRegistry to handle incoming WMA messages, for example. Testing of this functionality assumes the following steps are present in the test procedure:
  - an application with Push registration is installed
  -the push event is initiated
  - application is launched to handle the event
  The scenario does not fit into the Server/Agent scheme, because it requires an application installation to happen as part of the test, and does not fit into the autotest scheme, because it does not have automatic 'run/remove' steps, that are mandatory parts of the 'autotest' cycle.
• CHAPI spec talks about communications between two applications, the ContentHandler and Invoker.

The cases above are usually associated with the platform component, named JAM (Java Application Manager) or AMS (Application Management Software), therefore the testing techniques that involve AMS operations (install/update/run/remove) or application life cycle (start/pause/stop/destroy) will be referred to as 'AMS testing'.

OTA Testing Framework

Absolute Weapon

Typically, the OTA Testing Framework consists of the following components:

• OTA test - the test scenario, that includes standard and custom AMS operations. It operates with applications associated with this test. It is executed on the server side of the system, main communication point responsible for reporting the whole test status.
  
• Test application(s) - zero or more pre-packaged applications that may contain part of the test code that needs to be executed on the device. They may communicate with the main test scenario - OTA test

There is also an important component called OTA Server. In MIDP case it is HTTP server controlled by OTA test and responsible for provisioning of test applications. Nice artistic picture of the OTA Testing Framework in provided in the ME Framework Developer's Guide (link to PDF version), check Figure 3-7.

The approach described above is universal and powerful. It can be used to test the most exotic cases. At least, so far it stays as a last absolute weapon used if no other technique works.

Last Thing to Use

The only disadvantage of OTA Testing Framework is that it is terribly interactive. During the certification run TCKs do not allow to use any automation, every AMS operation must be performed manually. With a simple install/run/remove cycle, where one needs to read instructions, enter url, wait for download/launch/execution complete/remove, it takes 1 minute minimum for a test case.

Now imagine yourself a QA engineer, who has to run CHAPI TCK, that contains ~40 OTA tests, as a regression test suite on a regular basis during the whole development cycle without any automation.

Lets consider a variation to the above approach, that can be used when it is OK to narrow the scope of covered possibilities and simplify the usage.

XletManager

The idea is to specify the Java interface to the AMS that will allow one application to perform the life cycle operations on another application. It was created for PBP/TV, where testing is based on the Server/Agent model. The scheme works in the following way:

• An implementation developer provides an implementation of XletManager interface, that would allow a test application, executed on the device to invoke the AMS operations through the Java API. In the general case, if Java interface to AMS is not available, the XletManager implementation may be interactive.
• Test Suite user manually downloads and installs on the device the Agent application and a set of Xlet applications that will be used in tests
• Tests that are executed within the Agent instantiate XletManager and operate with additional pre-installed test applications.

While in the PBP TCK this approach is used to initiate only life cycle operations from the test, executed on the device, it also can be extended to any AMS operations like install/update/run/remove.

Comparing with OTA Test Framework

Main benefit of XletManager-like approach is that it allows to use a single Server/Agent test execution model for the whole test suite.

With this specific approach, there is no dynamic application installation/removal, no application provisioning component, test applications are preinstalled before test execution begins.

There may be restrictions, that will complicate the use of this model or just make it impossible, like:

• interactive implementation of XletManager may be complicated for devices with small screen
• platform may not allow more than one application to run simultaneously.

Variations

Interface to the AMS may be implemented through plugins, that may be provided by implementation developers. These plugins may be interactive in general case or automated. Automated testing may be used for QA/regression testing or even for certification in some cases.

There are interesting possibilities of using standard Java ME APIs in order to communicate with AMS, for example CHAPI. As a minimum, it can be used to have the test code distributed across multiple MIDlets or MIDlet-suites that are registered as ContentHandler-s and invoked from the main test scenario.

Summary

Over time, the number and complexity of Java ME implementation tests dealing with AMS have increased. I believe this is a natural consequence of consistent focus on standards in the area of application management for consumer devices.

The interesting and important problem, that comes together with number of appearing and evolving AMS-testing frameworks that are often interactive is automation, it deserves separate article.

Merry Christmas and Happy New Year !

This work is licensed under a Creative Commons License.