This article is about interactive testing for Java ^TM and its ME specifics. It describes types of interactive tests that are being developed for Java Technology Compatibility Kits, testing of what functionality requires user interaction. What Java ME limitations cause problems for development of tests, that require user interaction and how these limitations can be worked out.

Most of definitions and examples are given using the terminology of JT harness and ME Framework, but should be generic enough for the area. For details on what is TCK, JT harness and ME Framework refer to previous articles. Skip the background section if you are familiar with the subject.

Background

Why not everything is automated

Interactive tests are needed to test API that produces output, that in general case can be only verified by human or requires human or require human input. That is API that draws something on the screen, plays sound, or reacts to key pressing or mouse dragging.

For every specific implementation, testing of this functionality can be automated as well with different external tools or specific APIs. But first, not always - compatibility tests, for example, can not rely on any specifics because have to be correct for any compatible implementation, and second, this is a separate big topic, to be covered in another article.

What is meant by interactive tests

From point of view of the test harness, all tests, automated and interactive, are discovered and executed in the same automated way. Interactive tests are those that show a dialog with some instructions and wait for user doing something. These tests are usually grouped together to be executed in a single session, separately from the rest. The rest are completely automated tests, you can run them nightly for regression testing. Execution of interactive tests takes hours of someones expensive time.

Requirements

Speaking from point of view of TCK development, the most important requirement is to make the test suite easiest for use. This means the less interactive tests it has, the better. If we have to have interactive tests, it is important to make their execution simple.

To achieve this we limited number of test types that we use in TCKs to very few. As a result, some features, that are not best suited to be tested by these test types, require multiple test cases to be written where we would write one 'custom' test case. The benefit is that TCK user has uniform interactive model, uniform interface to browse test instructions etc. This also allows for simpler external automation system development.

Types of TCK interactive tests

As mentioned before, such tests may require input from the user or require some output to be evaluated, or both. Test status may be calculated automatically or require user judgment. The interactive test library, that is included into ME Framework, provides Done and Yes/No interfaces to enable these two types of user interaction. There is also Info Only interface , that can be used if the end of test is known beforehand (event sequence is predefined). 

Note that these tests are not just set of user instructions 'do this - verify that', they include test code that can do most of work.

Java ME Specifics and Solutions

PJava story - first TCK alt bundle

Interactive tests created for Java SE TCKs are usually interactive applications, that run on the platform under test and show user instructions and test panel in a single window. This approach does not always work for Java ME for many reasons. First time we started doing interactive tests in the world of consumer devices, that was Personal Java, we found that tests we created can not be passed on PJava devices with single Frame limitation and small screen - test instructions and test panel were placed in the container without scrolling capabilities, these 'Done' and 'Yes/No' buttons may not appear on screen in some circumstances. To address this we issued first 'alternative TCK test bundle' that just enabled scrolling. After that passing of interactive tests became possible, though not convenient. Check how AgentFrame interface looks like on PJava with Truffle toolkit. The scheme of these interactive tests is the one that is still used in Java SE TCKs, can be described as follows:

Figure I. Simple Interactive Test.

MIDP TCK 1.0 interactive tests

Interactive tests for MIDP TCK 1.0 were executed using the same Autotest mechanism as regular automated tests, the difference was that interactive tests expected some user actions and were grouped together for convenience. These tests were developed using brand-new MIDP API and same approach that was used in JCK and PJCK. Instructions, test panel, all user interface components of these tests were displayed on the micro screen of MIDP 1.0 micro devices. Given big number of interactive tests, that were necessary to verify MIDP 1.0 GUI API, running MIDP 1.0 TCK on a regular basis during the development process, was a headache.

You can see the screen shot of the MIDP 1.0 emulator to the right. Click on it to see the sequence of screens that constituted the MIDP TCK 1.0 interactive test. As you can see, there is a big number of interactions, that are not related to execution of the test but for scrolling, switching controls etc.

Distributed Interactive test framework

MIDP TCK 2.x interactive tests

To address this problem of inconvenience of interactive tests on MIDP devices with small screen, the solution was very simple and usability improvement was huge. Briefly, these tests were rewritten to using Distributed Test library to have minimal functionality on the device and have user instructions and controls, related to the test logic, on the server part. Now to run tests on the device one need to stare to the desktop monitor, read instructions, press some buttons on the desktop, for example, to initiate tested process on the device, do some interaction with the device as necessary, state pass/fail result on the desktop if needed.

The important feature of Distributed Test framework, that lies underneath new Interactive Test Framework is that there are java components of the test, that reside on server and client sides and can work together to calculate test result. One can use server side technologies in the test, for example, for reference purpose to verify that same technology works properly at Java ME side.

The scheme of distributed interactive test can be described as follows

:

Figure II. Distributed Interactive Test.

In the example of the interactive test for sound, you can see all GUI of this test. Device part of this specific test does not have GUI at all, all that device does - produces sounds that are initiated from the server side, pass/fail criteria is specified on the server side of the test as well.

PBP TCK 1.0 interactive tests

Same solution was used for interactive tests for Personal Basis Profile TCK. Though PBP API is subset of J2SE API, reuse of JCK tests was not possible there - as I mentioned, these tests combine in a single application all instructions and controls, that were not available. PBP does not have Panel, Button - no any UI widgets, only Component, Container and Window Frame. To reproduce anything on the device screen we would have to draw it using graphics primitives from scratch and creating UI toolkit was not in the scope of the TCK.

Separating functionality and interaction between server and client parts of the tests, that were created using Distributed Test framework, worked well for PBP. In the example of the keyboard test you can see the same situation as in already referenced MIDP TCK 2.0 example, all GUI is on desktop screen, device does not have any GUI, just accepts key presses and pass them to the server side.

AGUI TCK 1.0 interactive tests

Yet another special solution to workaround small device screen was implemented for AGUI TCK 1.0.

AGUI stands for Advanced Graphics and User Interface, that assumes lots of interactive tests. For AGUI the execution model for interactive tests was also the same as for regular automated tests, all tests were executed in the same Agent, interactive tests grouped separately from automated. Read here about MIDP and CDC execution modes of ME Framework.

As AGUI API is a subset of Java SE API, specifically, Swing, our goal was to reuse as much of JCK interactive tests for this API as possible. The reason here is not only time saving, it is also an additional way to ensure compatibility across different Java platforms.

What we did is we again separated tests to different pieces that could be displayed separately and organized test UI as Tabs. It was relatively easy to do, since JCK interactive tests library assumed some structuring, there was implementation of this library done with Swing subset, that fit into AGUI API. Though it was still necessary to do extra clicks to switch between different tabs and sometimes scroll through the tab, the effect was a significant usability improvement comparing to having everything in the same window altogether.

You can see the screen shot of the AGUI 1.0 emulator to the left. Click on it to see the sequence of screens that constituted the AGUI TCK 1.0 interactive test.

It was a temporary solution, once we structured tests to independent pieces, it was easy to execute these components on distributed components. This was not a conversion of simple interactive tests to distributed tests but a special execution framework, that could execute simple interactive tests in the distributed way and gave some other minor usability improvements. Overall, Interactive tests in the AGUI TCK 1.0 when it was released looked exactly like all other distributed interactive tests for Java ME.

Tests with static image

This type of tests is very easy to create, execute and automate. Basically, the scenario of these tests is to show reference image and its verbal description somewhere, initiate drawing of the same image on the device and ask user to validate the output.

This approach is natural one, it can be used to test 80% of functionality, related to user interaction. Some technologies use it as the only approach for testing, for example W3C SVG test suite. It can be used to test low level graphics and behavior of high-level user interface components.

Even when these tests are too primitive and require multiple test cases to be developed when few more sophisticated ones could be enough, the simplicity of development and execution, possibility to have consistency across many tests could be a reason to use this approach even when it is not the most convenient.

Static image tests with ME Framework

To simplify test execution we display multiple images, related to the same functionality, in a single window with test instructions. Every reference image is accompanied with a Test button, that initiates reproduction of this image on the device under test, ad verbal description. Check the example, the screenshot of the instructions dialog and device side for tests for Java Binding to Open GL ES API (JSR 239) .

Combining multiple related images that could be switched in any sequence allows not only for comparison of the result on the screen with reference, but also for checking of transition one image to another on the device. This, as well as verbal description of the scene, may be important when reference images were done on the implementation, that is radically different with one under test and comparing of test and reference image alone can not provide confidence that tested functionality behaves correctly.