Skip to main content

Interactive Tests for Java ME

Posted by alexeyp on May 3, 2007 at 5:46 AM PDT

This article is about interactive testing for Java TM
and its ME
specifics. It describes types of interactive tests that are
being
developed for Java Technology Compatibility Kits, testing of
what functionality requires user interaction. What Java ME limitations
cause problems for development of
tests, that require user interaction and how these limitations can be
worked out.

Most of definitions and examples are given using the
terminology of JT
harness
and href="https://cqme.dev.java.net/framework.html">ME
Framework, but should be generic enough for the area.
For details on what is TCK, JT harness and ME Framework refer
to previous articles. Skip
the background section if you are familiar with the subject.

Background

Why not everything is automated

Interactive tests are needed to test API that produces output,
that in general case can be only verified by human or requires human or
require human input. That is API that draws something on the screen,
plays sound, or reacts to key pressing or mouse dragging.

For every specific implementation, testing of this
functionality can be automated as well with different external tools or
specific APIs. But first, not always - compatibility tests, for
example, can not rely on any specifics because have to be correct for
any compatible implementation, and second, this is a separate big
topic, to be covered in another article.

What is meant by interactive tests

From point of view of the test harness, all tests, automated and
interactive, are
discovered and executed in the same automated way. Interactive tests
are those that show a dialog with some instructions and wait for user
doing something.
These tests are usually grouped together to be executed in a single
session, separately from the rest. The rest are completely automated
tests, you can run them nightly for regression testing. Execution of
interactive tests takes hours of someones expensive time.

Requirements

Speaking from point of view of TCK
development, the most important
requirement is to make the test suite easiest for use. This means the
less interactive tests it has, the better. If we have to have
interactive tests, it is
important to make their execution simple.

To achieve this we limited number of test types that we use in
TCKs to very few. As a result, some features, that are not best suited
to be tested by these test types, require multiple test cases to be
written where we would write one 'custom' test case. The benefit is
that
TCK user has uniform interactive model, uniform interface to browse
test instructions etc. This also allows for simpler external automation
system development.

Types of TCK interactive tests

As mentioned before, such tests may require input from the user or
require some output to be evaluated, or both. Test status may be
calculated automatically or require user judgment. The interactive
test library, that is included into ME Framework, provides

Done

and

Yes/No

interfaces to enable these two types of user interaction. There is also
Info Only interface , that can be used if the end
of test is known beforehand (event sequence is predefined). 

Note that these tests are not just set of user instructions
'do this - verify that', they include test code that can do most of
work.

Java ME Specifics and Solutions

PJava story - first TCK alt bundle

Interactive tests created for Java SE TCKs are usually
interactive applications, that run on the platform under test and show
user instructions and test panel in a single window. This approach does
not always work for Java ME for many reasons. First time we started
doing interactive tests in the world of consumer devices, that was
Personal Java, we found that tests we created can not be
passed on PJava devices with single Frame limitation and small screen -
test instructions and test panel were placed in the container without
scrolling capabilities, these 'Done' and 'Yes/No' buttons may not
appear on screen in some circumstances. To address this we issued first
'alternative TCK test bundle' that just enabled scrolling. After that
passing of interactive tests became possible, though not convenient.
Check how

AgentFrame

interface looks like on
PJava
with Truffle toolkit. The scheme of these interactive tests is the one
that is still used in Java SE TCKs, can be described as follows:

style="width: 252px; height: 212px;"
alt="Siple interactive test"
src="http://weblogs.java.net/blog/alexeyp/archive/InteractiveImages/SimpleITestLogic.jpg">

style="font-weight: bold;">Figure I. Simple Interactive Test.

style="float: right;"
onclick='src="http://weblogs.java.net/blog/alexeyp/archive/InteractiveImages/midp10-"+i+".JPG";i++;i=i%6'
alt="MIDP10 test" src="http://weblogs.java.net/blog/alexeyp/archive/InteractiveImages/midp10-0.JPG"
hspace="40">

MIDP TCK 1.0 interactive tests

Interactive tests for MIDP TCK 1.0 were executed using the
same
href="http://weblogs.java.net/blog/alexeyp/archive/2006/11/testing_java_me_1.html#Autotest">Autotest
mechanism as regular
automated tests, the difference was that interactive tests expected
some user actions and were grouped together for convenience. These
tests were developed using
brand-new MIDP API and same approach that was used in JCK and PJCK.
Instructions, test panel, all user interface components
of these tests were displayed on the micro screen of MIDP 1.0 micro
devices. Given big number of interactive tests, that were necessary to
verify MIDP 1.0 GUI API, running MIDP 1.0 TCK on a regular
basis during the development process, was a headache.

You can see the screen shot of the MIDP 1.0 emulator to the
right. Click
on it to see the sequence of screens that constituted the
MIDP TCK 1.0 interactive test. As you can see, there is a big
number of
interactions, that are not related to execution of the test but for
scrolling, switching controls etc.

click here src="http://weblogs.java.net/blog/alexeyp/archive/InteractiveImages/clickhere.JPG">

Distributed Interactive test framework

MIDP TCK 2.x interactive tests

To address this problem of inconvenience of interactive tests
on MIDP devices with small screen, the solution was very simple and
usability improvement was huge. Briefly, these tests were rewritten to
using Distributed Test library to have minimal functionality on the
device and have user instructions and controls, related to the test
logic, on the server part. Now to run tests on the device one need to
stare to the desktop monitor, read instructions, press some
buttons on the desktop, for example, to initiate tested process on the
device, do some interaction with the device as necessary, state
pass/fail result on the desktop if needed.

The important feature of Distributed Test framework, that lies
underneath new Interactive Test Framework is that there are java
components of the test, that reside on server and client sides and can
work together to calculate test result. One can use server side
technologies in the test, for example, for reference purpose to verify
that same technology works properly at Java ME side.

The scheme of distributed interactive test can be described as
follows

: style="width: 579px; height: 349px;"
alt="Distributed interacrive test scheme"
src="http://weblogs.java.net/blog/alexeyp/archive/InteractiveImages/DistrITest1.JPG" align="middle">

style="font-weight: bold;">Figure II. Distributed
Interactive Test.

In the

example

of the interactive test for sound, you can see all GUI of this test.
Device part of this specific test does not have GUI at all, all that
device does - produces sounds that are initiated from the server side,
pass/fail criteria is specified on the server side of the test as well.

PBP TCK 1.0 interactive tests

Same solution was used for interactive tests for Personal
Basis Profile TCK. Though PBP API is subset of J2SE API, reuse of JCK
tests was not possible there - as I mentioned, these tests combine in a
single application all instructions and controls, that were not
available. PBP does not have Panel, Button - no any UI widgets, only
Component, Container and Window Frame. To reproduce anything on the
device screen we would have to draw it using graphics primitives from
scratch and creating UI toolkit was not in the scope of the TCK.

Separating functionality and interaction between server and
client parts of the tests, that were created using Distributed Test
framework, worked well for PBP. In the href="http://weblogs.java.net/blog/alexeyp/archive/InteractiveImages/PBP_KeyEvent_virtualkeyboard_fail."
onclick="window.open('http://weblogs.java.net/blog/alexeyp/archive/InteractiveImages/PBP_KeyEvent_virtualkeyboard_fail.','popup','width=643,height=608,scrollbars=no,resizable=no,toolbar=no,directories=no,location=no,menubar=no,status=no,left=0,top=0'); return false">example
of the keyboard test you can see the same situation as in already
referenced MIDP TCK 2.0 example,
all GUI is on desktop screen, device does not have any GUI, just
accepts key presses and pass them to the server side.

src="http://weblogs.java.net/blog/alexeyp/archive/InteractiveImages/agui10-0.JPG"
alt="AGUI interactive"
onclick='src="http://weblogs.java.net/blog/alexeyp/archive/InteractiveImages/agui10-"+j+".JPG";j++;j=j%4'
style="width: 312px; height: 694px; float: left;" hspace="40">

AGUI TCK 1.0 interactive tests

Yet another special solution to workaround small device screen
was implemented for AGUI TCK 1.0.

AGUI stands for Advanced Graphics and User Interface, that
assumes lots of interactive tests. For AGUI the execution model for
interactive tests was also the same as for regular automated tests, all
tests were executed in the same Agent, interactive tests grouped
separately from automated. Read href="http://weblogs.java.net/blog/alexeyp/archive/2006/11/testing_java_me_1.html">here
about MIDP and CDC execution modes of ME Framework.

As AGUI API is a subset of Java SE API, specifically, Swing,
our goal was to reuse
as much of JCK interactive tests for this API as possible. The reason
here is not only time saving, it is also an additional way to ensure
compatibility across different Java platforms.

Click there src="http://weblogs.java.net/blog/alexeyp/archive/InteractiveImages/clickthere.JPG">What
we did is we again separated tests to different pieces that could be
displayed separately and organized test UI as Tabs. It was relatively
easy to do, since JCK interactive tests library assumed some
structuring, there was implementation of this library done with Swing
subset, that fit into AGUI API. Though it was still
necessary to do extra clicks to switch between different tabs and
sometimes scroll through the tab, the effect was a significant
usability improvement comparing to having everything in the same window
altogether.

You can see the screen shot of the AGUI 1.0 emulator to the
left. Click
on it to see the sequence of screens that constituted the AGUI TCK 1.0
interactive test.

It was a temporary solution, once we structured tests to
independent pieces, it was easy to execute these components on
distributed components. This was not a conversion of simple interactive
tests to distributed tests but a special execution framework, that
could execute simple interactive tests in the distributed way and gave
some other minor usability improvements. Overall, Interactive tests in
the AGUI TCK 1.0 when it was released looked exactly like all other
distributed interactive tests for Java ME.

Tests with static image

This type of tests is very easy to create, execute and automate.
Basically, the scenario of these tests is to show reference image and
its verbal description somewhere, initiate drawing of the same image on
the device and ask user to validate the output.

This approach is natural one, it can be used to test 80% of
functionality, related to user interaction. Some technologies use it as
the only approach for testing, for example href="http://www.w3.org/Graphics/SVG/Test/">W3C SVG test
suite. It can be used to test low level graphics and
behavior of high-level user interface components.

Even when these tests are too primitive and require multiple
test cases to be developed when few more sophisticated ones could be
enough, the simplicity of development and execution,
possibility to have consistency across many tests could be a reason to
use this approach even when it is not the most convenient.

Static image tests with ME Framework

To simplify test execution we display multiple images, related
to the same functionality, in a single window with test
instructions. Every reference image is accompanied with a Test button,
that initiates reproduction of this image on the device under test, ad
verbal description. Check the example, the screenshot of the

instructions dialog

and

device side

for tests for Java Binding to Open GL ES API (JSR 239) .

Combining multiple related images that could be switched in
any sequence allows not only for comparison of the result on the screen
with reference, but also for checking of transition one image to
another on the device. This, as well as verbal description of the
scene, may be important when reference images were done on the
implementation, that is radically different with one under test and
comparing of test and reference image alone can not provide
confidence that tested functionality behaves correctly.

Related Topics >>