Debugging II - Hangups at Device Side

As a follow up to the past article about Debugging with ME Framework, here is the the guest post from Alexander Alexeev (aka Skavas) on the new feature
he has integrated into the ME Framework,
the Interactive MIDlet agent. The feature addresses some usability
issues of executing large test suites on mobile devices, provides
on-screen indication of the testing progress and allows to perform some
operations with test results on the device.

Background

The Introduction
article described the approaches, used for execution of large test
suites for Java ^TM ME implementations as well as some of techniques,
allowing to optimize test execution time. To restate the main
points, relevant to this debugging topic:

• test execution is managed at the server side
• the test execution process consists of sequential
  downloads of test MIDlet suites to the device
• during this process AMS and  MIDlet suites
  exchange control messages with test harness
• one of optimizations, allowing to minimize network traffic
  and number of downloads/installations/runs/removals, is to package
  multiple tests into a single bundle.

The diagram describing this autotest
cycle can be seen here.

The 'autotest'  approach allows to achieve the
sufficient level of automation, that is one of high priority requirements
for test suites we develop with JT harness
and ME Framework. 

User interactivity here still may be desired in few situations:

• 'sendTestResult' may not be executed for some reason. For
  example, because of the VM exit or test/VM hang up. Since this
  operation is executed at the Test level, and Test may consist of
  multiple test cases, all information from actually executed test cases
  may be lost. It will be unknown which test case caused the problem.
• when the device is slow and optimization is used (multiple
  tests in bundle), the whole test cycle will be shorter, but every
  individual test bundle will take time to execute. Without visual
  indication it may be hard to distinguish if tests are being executed
  that slow, or it is just device  stopped responding an hour
  ago.
• if test hung, it should be possible to cancel it
  with  minimal impact on the  execution process,
  without stop/restart the whole test run.

One solution for these debugging problems is using the Test
Export feature
to
run tests isolated from the test harness environment. Test results
from standalone execution are not sent to the JT harness, they are
displayed
on the device console, if there is such available.

Interactivity for Automated Tests

The proposed solution for all above problems is to introduce
interactivity at the device side. We added an option to use Interactive
MIDlet Agent for test execution, that provides the following features:

• shows the current status on the device display
• continuously saves
  the current status of the RMS
• provides a control for saving the current status of
  the RMS
• allows viewing results stored in the RMS
• provides a control for canceling the
  current test/test bundle and starting the next test/test bundle

These interactive features are also available in the Test Export mode.

User Interface

Additional set of standard configuration questions were added to the ME
Framework Configuration Editor. To make ME Framework to use this
on screen tracing functionality, see the screenshot
.

When the JT harness executes tests with these configuration
settings,
the user interface on the device side displays the following
information and commands:

• Information string with a common count of tests done, a
  count of failed tests, and a count of passed tests
• Information about the current test running
• Commands to cancel
  the test/test bundle and to save the results to the RMS

As you can see here
,
this new interface resembles the interface of device-side harnesses,
that are available for execution of mobile
variations of JUnit. 

To view results stored in the RMS, use the
RMSReader MIDlet application. It has a simple interface that uses
a Command to change the display either to "view log" or to "view ref"
output.

Open Issues

There should be better solutions then one that we chosen for canceling a hanging test.

The 'cancel test' functionality requires each test to be run in a
separate thread. Since Connected Limited Device Configuration has no
method to interrupt threads in order to break test execution and
proceed to the next test, a special flag is used to mark the test
thread as canceled. The canceled thread is set to minimum priority and
the agent starts to execute the next test.