Alexey Popov's Blog

Testing Archives

What is new in JT harness 4

As I wrote once,  development of this product is primarily driven by using it as a test harness for Technology Compatibility Kits. The next big step in the history of the product was its adoption  in the new area of Java ME quality test suites, specifically JDTS, the Java Device Test Suite. JDTS2.0 went out December 2006, see its DataSheet for more information. Among lots of other minor and major changes, this was first JDTS version based on JavaTest ^TM Harness, version 4.0.

The initial launch of the JT harness, the Open Source version of JavaTest Harness,  was based on the current stable version of the product, 3.2.2.  The version 4.0 was primarily driven and targeted to this release of JDTS 2.0, due to tight time line it was developed internally for some time and become available in open source only now since version 4.1.1

Major new JavaTest harness 4 Features

• Tutorial is now available with the binary.
  
• Backward compatibility with JavaTest harness 3.2.2
  
• Customization features.
  
  The requirement was to make JavaTest 4 a testing platform, that can be supplied with product-specific functionality and have customizable functions and appearance.
  
• Custom splash screen can be used to provide stronger identification of the JavaTest-based testing product.
• Custom tool bars, menus, pop-up menus, preferences can be used to associate functionality extensions with GUI
• Customizable GUI rendering of a test's result - e.g. show a graph of some data for a test. This is targeted for test categories, that may result in more then just pass/fail criteria, like performance tests. 
• Plug-in report mechanism. It can be used to fit the JavaTest-based test suite into the larger testing/certification system
  
  
• Configuration process improvements
  
  Configuration process is known as one of the most complex steps in usage of the large test suites, that are targeting multiple environments. JavaTest 4 made significant steps to improve usability in this area. New usage models, specific to quality vs compatibility testing area, were also addressed.
  
• Improved configuration templates mechanism to allow different levels of configuration tuning.
  
• Added new configuration interview question types
  
• Per-test configuration, accessible through test tree context menu. This allows development of the test suites, where user is required to configure only those parameters, that are minimally necessary for execution of specific test or group of tests and not pass through complete configuration process for the whole test suite.
  
  
• Management of test suite updates
  
  Ability to dynamically apply updates using existing installation of the configured testing product allows significant decrease in cost of applying updates to users, allowing more frequent updates and hence better product quality
  
  
• Reporting
  
  
• Merging report directories, generation of consolidated report to improve support to multi-user environment.
• XML formatted reports to simplify post-processing of test results using custom tools and integration into large testing/reporting infrastructure.
  
  
• Other
  
  Number of minor improvements, like: UI enhancements, logging

Mobile JUnit and absence of reflection.

In classical JUnit you have multiple test methods in a single class that match some pattern, test framework iterates through these methods using relection. CLDC/MIDP does not have reflection API, this is one of problems that need to be solved when adopting JUnit-like test frameworks to CLDC/MIDP. 

There are not too many solutions to this problem, well described in Vladimir's blog. I would like to add few words and will describe how we deal with this problem using ME Framework at Sun.

I like the elegant solution implemented in Sony Ericsson Mobile JUnit 1.0. Absence of reflection in CLDC (client side) sometimes can be dealt with at server side. In this case, when list of test methods to iterate through is static and not calculated at runtime, you can analyse class files or sources and generate this iterator-launcher method or class.

Another solution is to use preprocessors and code generation.

This is an approach that we use at Sun for compatibility test development for Java ME and Java SE. Test sources are described in a meta language that allows to combine meta data and code. Documentation and .java sources of tests are generated from this source file, as well as code for all calls to individual test methods. Here is the example of a generated launcher method for MultiTest API:

    protected void runTestCases() {
        if (isSelected("Connector2002")) {
            addStatus(Connector2002());
        }
        if (isSelected("Connector2003")) {
            addStatus(Connector2003());
        }
    }

As I wrote once, MultiTest has slightly different API then JUnit but similar principles. It's version for Java SE, that is natively supported by JT harness, uses reflection. Its CLDC-compatible variation, supported by ME Framework, requires this abstract 'runTestCases' method to be defined, in our case it is generated.

There other bonuses of code generation, like sharing compatibility tests across TCKs for Java platform API. We can have Java ME and Java SE versions of tests for the class that exists in both platforms generated from a single source.

Interactive Tests for Java ME

Most of definitions and examples are given using the terminology of JT harness and ME Framework, but should be generic enough for the area. For details on what is TCK, JT harness and ME Framework refer to previous articles. Skip the background section if you are familiar with the subject.

Background

Why not everything is automated

For every specific implementation, testing of this functionality can be automated as well with different external tools or specific APIs. But first, not always - compatibility tests, for example, can not rely on any specifics because have to be correct for any compatible implementation, and second, this is a separate big topic, to be covered in another article.

What is meant by interactive tests

Requirements

Speaking from point of view of TCK development, the most important requirement is to make the test suite easiest for use. This means the less interactive tests it has, the better. If we have to have interactive tests, it is important to make their execution simple.

To achieve this we limited number of test types that we use in TCKs to very few. As a result, some features, that are not best suited to be tested by these test types, require multiple test cases to be written where we would write one 'custom' test case. The benefit is that TCK user has uniform interactive model, uniform interface to browse test instructions etc. This also allows for simpler external automation system development.

Types of TCK interactive tests

Note that these tests are not just set of user instructions 'do this - verify that', they include test code that can do most of work.

Java ME Specifics and Solutions

PJava story - first TCK alt bundle

Interactive tests created for Java SE TCKs are usually interactive applications, that run on the platform under test and show user instructions and test panel in a single window. This approach does not always work for Java ME for many reasons. First time we started doing interactive tests in the world of consumer devices, that was Personal Java, we found that tests we created can not be passed on PJava devices with single Frame limitation and small screen - test instructions and test panel were placed in the container without scrolling capabilities, these 'Done' and 'Yes/No' buttons may not appear on screen in some circumstances. To address this we issued first 'alternative TCK test bundle' that just enabled scrolling. After that passing of interactive tests became possible, though not convenient. Check how AgentFrame interface looks like on PJava with Truffle toolkit. The scheme of these interactive tests is the one that is still used in Java SE TCKs, can be described as follows:

Figure I. Simple Interactive Test.

MIDP TCK 1.0 interactive tests

Interactive tests for MIDP TCK 1.0 were executed using the same Autotest mechanism as regular automated tests, the difference was that interactive tests expected some user actions and were grouped together for convenience. These tests were developed using brand-new MIDP API and same approach that was used in JCK and PJCK. Instructions, test panel, all user interface components of these tests were displayed on the micro screen of MIDP 1.0 micro devices. Given big number of interactive tests, that were necessary to verify MIDP 1.0 GUI API, running MIDP 1.0 TCK on a regular basis during the development process, was a headache.

You can see the screen shot of the MIDP 1.0 emulator to the right. Click on it to see the sequence of screens that constituted the MIDP TCK 1.0 interactive test. As you can see, there is a big number of interactions, that are not related to execution of the test but for scrolling, switching controls etc.

Distributed Interactive test framework

MIDP TCK 2.x interactive tests

To address this problem of inconvenience of interactive tests on MIDP devices with small screen, the solution was very simple and usability improvement was huge. Briefly, these tests were rewritten to using Distributed Test library to have minimal functionality on the device and have user instructions and controls, related to the test logic, on the server part. Now to run tests on the device one need to stare to the desktop monitor, read instructions, press some buttons on the desktop, for example, to initiate tested process on the device, do some interaction with the device as necessary, state pass/fail result on the desktop if needed.

The important feature of Distributed Test framework, that lies underneath new Interactive Test Framework is that there are java components of the test, that reside on server and client sides and can work together to calculate test result. One can use server side technologies in the test, for example, for reference purpose to verify that same technology works properly at Java ME side.

The scheme of distributed interactive test can be described as follows

:

Figure II. Distributed Interactive Test.

In the example of the interactive test for sound, you can see all GUI of this test. Device part of this specific test does not have GUI at all, all that device does - produces sounds that are initiated from the server side, pass/fail criteria is specified on the server side of the test as well.

PBP TCK 1.0 interactive tests

Same solution was used for interactive tests for Personal Basis Profile TCK. Though PBP API is subset of J2SE API, reuse of JCK tests was not possible there - as I mentioned, these tests combine in a single application all instructions and controls, that were not available. PBP does not have Panel, Button - no any UI widgets, only Component, Container and Window Frame. To reproduce anything on the device screen we would have to draw it using graphics primitives from scratch and creating UI toolkit was not in the scope of the TCK.

Separating functionality and interaction between server and client parts of the tests, that were created using Distributed Test framework, worked well for PBP. In the example of the keyboard test you can see the same situation as in already referenced MIDP TCK 2.0 example, all GUI is on desktop screen, device does not have any GUI, just accepts key presses and pass them to the server side.

AGUI TCK 1.0 interactive tests

Yet another special solution to workaround small device screen was implemented for AGUI TCK 1.0.

AGUI stands for Advanced Graphics and User Interface, that assumes lots of interactive tests. For AGUI the execution model for interactive tests was also the same as for regular automated tests, all tests were executed in the same Agent, interactive tests grouped separately from automated. Read here about MIDP and CDC execution modes of ME Framework.

As AGUI API is a subset of Java SE API, specifically, Swing, our goal was to reuse as much of JCK interactive tests for this API as possible. The reason here is not only time saving, it is also an additional way to ensure compatibility across different Java platforms.

What we did is we again separated tests to different pieces that could be displayed separately and organized test UI as Tabs. It was relatively easy to do, since JCK interactive tests library assumed some structuring, there was implementation of this library done with Swing subset, that fit into AGUI API. Though it was still necessary to do extra clicks to switch between different tabs and sometimes scroll through the tab, the effect was a significant usability improvement comparing to having everything in the same window altogether.

You can see the screen shot of the AGUI 1.0 emulator to the left. Click on it to see the sequence of screens that constituted the AGUI TCK 1.0 interactive test.

It was a temporary solution, once we structured tests to independent pieces, it was easy to execute these components on distributed components. This was not a conversion of simple interactive tests to distributed tests but a special execution framework, that could execute simple interactive tests in the distributed way and gave some other minor usability improvements. Overall, Interactive tests in the AGUI TCK 1.0 when it was released looked exactly like all other distributed interactive tests for Java ME.

Tests with static image

This approach is natural one, it can be used to test 80% of functionality, related to user interaction. Some technologies use it as the only approach for testing, for example W3C SVG test suite. It can be used to test low level graphics and behavior of high-level user interface components.

Even when these tests are too primitive and require multiple test cases to be developed when few more sophisticated ones could be enough, the simplicity of development and execution, possibility to have consistency across many tests could be a reason to use this approach even when it is not the most convenient.

Static image tests with ME Framework

To simplify test execution we display multiple images, related to the same functionality, in a single window with test instructions. Every reference image is accompanied with a Test button, that initiates reproduction of this image on the device under test, ad verbal description. Check the example, the screenshot of the instructions dialog and device side for tests for Java Binding to Open GL ES API (JSR 239) .

Combining multiple related images that could be switched in any sequence allows not only for comparison of the result on the screen with reference, but also for checking of transition one image to another on the device. This, as well as verbal description of the scene, may be important when reference images were done on the implementation, that is radically different with one under test and comparing of test and reference image alone can not provide confidence that tested functionality behaves correctly.

One of the First Java Applications

Few words for a background.

The Java ^TM Compatibility Kit (JCK) is a test suite, used to verify if the Java standard is correctly implemented. The first JCK came out together with the first JDK from SUN, now this effort evolved into the industry-wide Java Community Process.

The JavaTest ^TM harness is a test monitor, used in first versions of JCK, then across multiple Java Technology Compatibility Kits (TCK), now evolved into a general purpose open source test harness.

How it started

With permission of Jonathan Gibbons, who was around when Java was born, here is his story of the JavaTest childhood:

JavaTest started round about JDK 1.0.2, JCK 1.0.2a was applet-based and did not use JavaTest;  JavaTest was introduced in JCK 1.0.2b.

It definitely was not the first application written in Java, but it is true that testing was important to Java from the beginning. JavaTest is sufficiently old that we had to develop many GUI widgets ourselves, and one of the early JavaTest developers (Tim Prinzing) went on to become a significant member in the Swing team. It is probably reasonable to say that Tim pioneered light-weight components.

Curious how it looked like back then at early JDK times ? Check here.

You can see that testing is one of keys that brought Java platform to where it is now. Java's way of compatibility testing, that is having high quality TCK required to pass to get Java logo,  is how Write Once Run Anywhere is achieved,  it is the cost of application portability and platform standardization.

On the road

Inspired by the conversation at David Herron's Blog, I was looking for analogy to illustrate the value of compatibility testing in the Java ecosystem, here it is - Java is a road, compatibility testing is its pavement. Applications are vehicles and the hard cover on the road makes it possible to drive fast. 

For Java ME platform, its fragmentation is a freedom to choose number of wheels, engine, use a bicycle or a truck. Another part, the 'dark side', is where the road cover has holes, caused by bugs in specification and implementations, test coverage problems. Services are built around the road to take care of vehicles and holes, like JDTS and Java Verified, while the road has its pavement (TCKs), differentiating it from deserts, forests and swamps.

Testing session at Java ME track at SUN TechDays, Saint Petersburg

Slides and description for it were first created in the native language of Internet and IT, that we use at work and call English by inertia. I did a first pass of translation to Russian and found that for many notions I do not have adequate words. For example, does open source mean the same as открытый код ? From the other side - with mother tongue you can make your speech so alive !

Check blogs of my colleagues who are speakers at the Java ME track: Danila (CLDC), Petr (CDC), Alexander (Testing). Presentations in English, like one of Simon Ritter about ME Gaming, will be translated online.

I will also be there at the Java ME POD. See you at SUN TechDays in Saint Petersburg !

Test Harness and Test Format

JavaTest is a test harness, JUnit is a test format. JavaTest is created to manage test suites written in many different formats, JUnit is a specific format and API for test development. The comparison is invalid, it would be more appropriate to compare JUnit API with one of test APIs, included into JavaTest's core libraries or its extensions.

Test format that is traditionally associated with this harness is the one that is used in most of Java Technology Compatibility Kits (TCK), based on test meta data provided in html form and Test/MultiTest API. Check the previous article for example of how JavaTest's Test Description looks like in .html. Comparing to JUnit, the java code of the TCK test may look like this:

The choice of the test API with explicit return for both positive and negative test conditions was driven by its use in TCKs, test suites, that are used for certification. One reason is requirement to be able to provide maximum to evaluate test failures offline without access to the implementation, where test failure is reproduced. Because of this there 'log' API is used instead of 'System.out', there is a special class Status that holds additional information other then pass/fail indication. Another reason is that TCK tests verify specific assertions of the specification and action of qualifying test as Passed need to be concise.

The JTReg is a JavaTest with set of plugins tuned for JDK regression testing. As opposite to the primary JavaTest use as a test harness for certification test suites, like TCK, where main requirements come from the test suite end user, JTReg is targeted to JDK developers. Main goals here are easiness to write and execute a test, support for major test types, possibility to execute shell scripts, adoption to the JDK development process. As you can see, there API for JDK regression tests is even simpler. See also JTReg FAQ for comments about JUnit support in JavaTest and JTReg.

Overall, different needs and different uses result in different test formats, that can be supported with single test harness.

Debugging II - Hangups at Device Side

Background

The Introduction article described the approaches, used for execution of large test suites for Java ^TM ME implementations as well as some of techniques, allowing to optimize test execution time. To restate the main points, relevant to this debugging topic:

• test execution is managed at the server side
• the test execution process consists of sequential downloads of test MIDlet suites to the device
• during this process AMS and  MIDlet suites exchange control messages with test harness
• one of optimizations, allowing to minimize network traffic and number of downloads/installations/runs/removals, is to package multiple tests into a single bundle.

The diagram describing this autotest cycle can be seen here.

The 'autotest'  approach allows to achieve the sufficient level of automation, that is one of high priority requirements for test suites we develop with JT harness and ME Framework. 

User interactivity here still may be desired in few situations:

• 'sendTestResult' may not be executed for some reason. For example, because of the VM exit or test/VM hang up. Since this operation is executed at the Test level, and Test may consist of multiple test cases, all information from actually executed test cases may be lost. It will be unknown which test case caused the problem.
• when the device is slow and optimization is used (multiple tests in bundle), the whole test cycle will be shorter, but every individual test bundle will take time to execute. Without visual indication it may be hard to distinguish if tests are being executed that slow, or it is just device  stopped responding an hour ago.
• if test hung, it should be possible to cancel it with  minimal impact on the  execution process, without stop/restart the whole test run.

One solution for these debugging problems is using the Test Export feature to run tests isolated from the test harness environment. Test results from standalone execution are not sent to the JT harness, they are displayed on the device console, if there is such available.

Interactivity for Automated Tests

• shows the current status on the device display
• continuously saves the current status of the RMS
• provides a control for saving the current status of the RMS
• allows viewing results stored in the RMS
• provides a control for canceling the current test/test bundle and starting the next test/test bundle

These interactive features are also available in the Test Export mode.

User Interface

When the JT harness executes tests with these configuration settings, the user interface on the device side displays the following information and commands:

• Information string with a common count of tests done, a count of failed tests, and a count of passed tests
• Information about the current test running
• Commands to cancel the test/test bundle and to save the results to the RMS

To view results stored in the RMS, use the RMSReader MIDlet application. It has a simple interface that uses a Command to change the display either to "view log" or to "view ref" output.

Open Issues

There should be better solutions then one that we chosen for canceling a hanging test.

The 'cancel test' functionality requires each test to be run in a separate thread. Since Connected Limited Device Configuration has no method to interrupt threads in order to break test execution and proceed to the next test, a special flag is used to mark the test thread as canceled. The canceled thread is set to minimum priority and the agent starts to execute the next test.

Java ME Testing - Debugging Test Failures

What to do when tests of Java^TM ME implementation test suite fail ?

This article offers some suggestions for debugging test failures - with a special focus on the JT harness and ME Framework features that support debugging. My initial motivation for writing this article was to announce improvements in the Test Export feature, however the topic is just so entertaining that I couldn't stop there. :-)

Debugging in Java Micro Edition (ME) has number of challenges. Issues related to debugging problems in Java ME implementation can include any of the following items:

• The problem can reside at the Java layer or in the native code.
• Mobile device is usually a component of a larger system rather then isolated system.
  For example, this may be connectivity of different kinds, integration of mobile application with server-side components.
• Multiple technologies and standards are involved.
  For example, the CLDC/MIDP implementation may communicate with javacard using SATSA and with Java SE/EE using RPC API of jsr172
• The device might not provide access to the Java console or support of debugging protocols.

Problems, specific to automated Java ME test suites, have the following roots:

• Automated test execution on CLDC/MIDP requires special AMS mode, autotest.
  CDC/PBP execution mechanisms may be complex as well. Basic information on this topic can be found in the first article: Introduction to Java ME Testing Tools.
• The distributed test execution mechanism.
  This includes execution of a test with test harness (JT harness) running on the desktop and the test agent is running on the device. In addition the distributed tests themselves consist of multiple components (such as SE and ME components).
• A complex configuration mechanism.
  Before starting to run tests you need to configure test harness according to the environment and implementation under test. This complicates the debugging by making the whole system more integrated. Tighter integration makes it difficult to identify the problem by isolating the piece of the test code from the test harness.

Debugging Tips

Tip 1 - Browse the code, test specification, test output.

• recognized parameters of Test Description
• values of environment variables
• version
• time stamp
• exact command used for test execution
• test output itself
• lots of other stuff, useful and not

Tip 2 - Turn on logging

• Logger API used by the ME Framework implementation classes.
• 'Debug output' options for ME Framework components

Turn on Logger

To turn this type of logging on you need to specify the config file through java system property when starting JavaTest harness, like this:

java^
    -Djava.util.logging.config.file=D:/exec/0-configs/j2me-fw-log.properties^
    -jar javatest.jar

ME Framework Tracing Capabilities

• Test execution subsystem (CLDC/MIDP), Figure 3-4 at page 27.
  The trace lists control messages between the harness and the device that correspond to Autotest execution flow. This may be useful if test execution is very slow, like when there are multiple tests packaged per test bundle, to make sure that there is a progress, to find which exactly test crashed the VM, to see exact test parameters and test result outside of the harness. The real-time trace can be observed on the device console, if the console is available. See a snapshot from the Configuration Editor that shows which Interview questions are responsible for this option. Check server and device-side examples of trace output.
  
• Distributed test framework, Figure 3-5 at page 29.
  Here you can see all messages exchanged by this framework, who sends what. Needless to say, it is one of the important features used to investigate failures of distributed tests. Simple interactive test with 3 static images sends that many messages to pass.
  
• Agent (CDC/PBP).
  Passing '-trace' option to agent displays everything going inside of it at the console (if available). This is a standard feature of the JT harness Agent. The example of the Agent trace can serve as a good illustration to CDC-specific approaches to address test suite scalability. Note the data exchanges and test classes instantiations for every test.

Tip 3 - Execute the test standalone

Each individual test usually has an application entry point, allowing to execute it outside of the test harness. This is usually static main() taking an array of parameters (the same parameters as those passed to the test by the test harness). You can put test classes in the classpath and call the test by the class name.

To get values of test parameters you may need to dig into test environment. To execute standalone test corresponding to the following sample test description:

you must execute the following command:

export sampleValue=<find 'sampleValue' variable in the test environment>
java -classpath $TEST_HOME/classes^
    com.sun.tck.satsa.tests.api.java.security.KeyFactory.generatePublicTests^
    -envValue $sampleValue

Or even simpler - JT harness result file contains the exact command and values of parameters that must be executed. See section messages:(1/693) in the example.

Unfortunately this works well only in CDC and Java SE execution modes. This approach will not work for CLDC/MIDP, where you need the application to be packaged and deployed before you can execute it.

Tip 4 - store for offline analysis

Tip N - use common sense

• Vary configuration parameters
• Add trace printing into test itself or into tested code
• Execute the test on the another implementation to verify its correctness
• ...

Test Export

The ME Framework is a quite complex set of interacting components involved in test execution. When a test is reported as failed by the JT Harness it's not always evident where the bug is and how to track it down. Most often, when users are trying to track down a bug, they want to run the failed test standalone on the device under test. This is where the Test Export feature can help. Test Export converts an individual test into a small standalone Java ME application containing the test, which can be executed, debugged, or used in any way convenient for the user.

Getting Test Export as a side-effect of Autotest execution

A very simple, partial implementation of the test export was available in the ME Framework from the beginning. It was a hack of a standard CLDC/MIDP execution mechanism., described by the following scheme.

FIGURE X. Autotest mode.

In order to turn test into CLDC/MIDP application that executes separately from the harness, one must replace all these arrows, representing some message exchanges, with static data.

For the initial ME Framework implementation of the test export feature, test packages where supplied with all information necessary for test execution (that replaced links 3 and 4). The execution server was packaging test bundles without waiting for any requests from AMS (links 1 and 2). For the test harness, test status indicated success of packaging and not actual test execution (link 5). When executed, tests did not send result to the execution server but just printed them on the console (link 5). Tests were packaged with all test data but not automatically downloaded to the device and test packages not removed after execution cycle completed.

Even this simple and limited approach allowed to solve some troubleshooting-related tasks. For example, a QA engineer responsible for TCK execution on the development build of the implementation could use these exported test packages as attachments to bug reports. A development engineer not familiar with TCK execution could use it to reproduce the problem and verify the fix.

Test Export in ME Framework 1.2

The previous implementation of the Test Export was not complete. Evident limitations were that only simple non-distributed tests were exported and not test sources. For the user, this meant that a created application could not be modified and rebuilt. The previous implementation also exported only .jar files and not .jad files.

Implementing the full Test Export feature for all types of tests is a challenging task requiring very accurate tracking of all test dependencies. The following are examples of such dependencies:

• Custom jad/manifest entries
• Test specific security conditions
• Code and resource dependencies of test and test suite level

Identifying all test sources associated with a distributed network test is not a simple task. To make this feature really user-friendly, one should provide parts of the test execution infrastructure, such as the provisioning server for MIDP.

To initiate Test Export for a ME Framework 1.2-based test suite, the user should first open the Configuration Editor and, in the interview, set test export mode on. Then the user should specify the export directory (the location where exported test will be saved) and maybe some additional parameters, such as the 'Prefix of URLs to JAR files' parameter specific for the MIDP platform. See the following screen-shot of the JT harness Configuration Editor: Configuring Test Export.

Finally, the user should select the test or tests to export and run them just as in a regular test run. In test export mode, instead of executing tests on a device, the selected test or tests are saved in the specified test export directory.

What you get with test export?

When the test export is done, the following content is created in the export directory:

• Java ME applications in JAR files named test1.jar, test2.jar, etc. containing the exported tests.

Just as with any other Java application, these applications can be uploaded on any suitable device and launched. The exported tests are executed in the same way that they are executed by ME Framework but in standalone mode, printing the diagnostics and test result on the device screen.

• In MIDP mode, Java application descriptors (JAD files) are generated: test1.jad, test2.jad, etc.

These descriptors are properly configured:

• They contain the same attributes as in regular test runs.
• They are signed and include certificates if the test suite was configured for trusted MIDP security mode.
• They contain proper links to the exported JAR files.

You can use the generated JAD files for OTA provisioning of exported tests by putting them on any suitable HTTP server or by using the special provisioning server included with the ME Framework.

• Java sources of exported tests in src subdirectory.

These source files can be used for debugging or rebuilding the exported tests.

• Class files in classes subdirectory.

• Ant build script in build.xml file.

Yeah! Users can change the exported tests by modifying its sources and rebuilding them with this build script. It contains all necessary targets for compiling sources, updating JAR files from classes, and signing JAD files. In addition, build.properties is also generated in the export directory. It defines properties used by the build script and can be used to configure the build. For example, property trusted, if defined, triggers signing JAD files when updating exported tests.

• Additional libraries and tools in lib subdirectories.

These are not part of exported tests and are not required for using them, but can be also helpful. For example, exportSigner.jar is a tool for updating signatures and certificates of MIDlet suites with tests in those cases where the tests contained in it have been modified and its JAR file has changed.

Let's look at this new feature from the point of view of a Sun Java Wireless Toolkit (Wireless Toolkit) user. Before the test export feature was introduced, users could execute tests on the emulator by using its autotest mode:


$WTK_HOME/bin/emulator -Xautotest http://localhost:8080/test/getNextApp.jad


This command launches the emulator and specifies an URL at which the emulator should look for tests to download. At this URL resides the execution server of ME Framework. The execution server responds to 'getNextApp.jad' requests by sending new and new tests from a large test suite. The emulator fails in a loop repeating three steps (install, execute, remove) for each downloaded test until the execution server has no more tests to send. There was no way to get just one test (corresponding JAD and JAR files) for standalone execution. In addition, users of the Wireless Toolkit know that autotest mode doesn't allow debugging!

Now, with the test export feature, a user can select the test to debug and export it to any appropriate location. JAR and JAD files are generated and can be executed on emulator. The following is a typical command for launching the exported test on the Wireless Toolkit emulator:


$WTK_HOME/bin/emulator -Xdescriptor $EXPORT_DIR/test3.jad


During its execution, the test prints the output to stdout. Click here to view an example.

This time, debugging the test during its execution is easy. Just add the -Xdebug argument to last command with all necessary parameters. You can then use your preferred debugger with the IDE of your choice to debug the test with its sources which are also exported.

Future Plans

The next planned improvement for debugging support will come from the integration of  Skavas's GUI Agent. This will allow you to use GUI and RMS in addition to console output for execution of exported tests. Watch for more information about this in the future - it's a topic for another post.

The Test Export feature is under development right now and you can improve it by posting on ME Framework forums. Propose your improvement or new feature and you may see it in one of the next releases.

Testing Command-Line Applications using Golden-File approach

The story describes our experience of using JavaTest-style test format and JavaTest itself as a test harness for functional and regression tests. The tested item is a command-line application.

Background

The most frequently observed approach to functional test development for such simple cases is to have a set of shell scripts placed into a directory hierarchy. Each of these scripts performs one simple check and reports results to the console. Tests are run by a trivial test harness, usually another shell script, that performs test iteration and joint status reporting. Test specifications usually come separately from tests.

How it was done here

Every individual test description in this test suite contains information on how to launch the tool and how its output should be verified. All tests in this test suite are written using the "golden-file" approach. The golden data is:

• application output streams both stderr and stdout
• exit code
• set of output files

The test development scenario for the new test is as follows:

• Create test description, including:
• test case specification (textual, optional)
• parameters, comparison procedure to use
• Run test in the 'setup' mode
• Validate output and mark it as 'golden file'

Test Description example:

• Description - Contains verbal Test Case Specification
• The table describes actual test cases that will be executed. Description is done in the language of JavaTest's HTMLTestFinder. Below you can see how these specific descriptions are interpreted by the execution Script, which used for this test suite:
  
  

  title	Unique ID of the test case
  executeArgs	Command-line arguments to pass to the tool.
  keywords	This field is used by main test filtering engine of JavaTest harness. Not used in this test suite.
  executeClass	Which of predefined golden file comparators to use for this test

Pros and Cons 

Pros:

• All traditional prerequisites come from using a specialized test harness, like test execution and test result management, test specification, source, and test result browsing, parallel test execution, test exclusion and filtering, multiple environments support (JDK version, for example) etc etc. 
• We got highly automated and maintainable test suite, preserved high consistency through all tests
• Test development is fast and simple
• Application can be tested at multiple platforms by design. When testing Java applications that need to be verified in multiple environments, it is critical to avoid a dependency on a specific scripting language used for test development.
• We avoided information duplication. The test code is linked to the  test specification and a test report

• Good test harness is a complex tool that takes time to study. Infrastructure may need to be adjusted for it.
• The "golden file" approach is naturally limited, can not be enough to cover everything
• When output is unstable, this approach may not work or may require sophisticated comparators. The most obvious example is in processing timestamps.

Test Harness for Product-Quality Test Suites

The requirements for the quality of test suites vary depending on the engineer's goal, the product being tested, how many and what kind of people will be using the test suite, how often and for how long, and how the test suite will be maintained. These factors may vary depending on whether you write unit tests for yourself or your project, or if, as a member of the SQEgroup on a large project, you are developing a test suite that will be executed by the SQA group. 

Sun Microsystems has a long successful story of development large test suites of various types. The development of the JavaTest  harness was highly influenced by its adoption in Java ^TM Technology Compatibility Kits (TCK). The majority, if not all of Sun's TCKs are built on the JavaTest harness. The key consequences of this:

• these test suites (TCKs) are products, developed by one group to be used outside of it on a constant basis for significant period of time.
• 'used in all TCKs from SUN' means that test suites built on JavaTest harness were run on all Java-compatible implementations in the world - just because implementations become Java-compatible after passing the TCK. It means a lot !
• though it is not the only area where JavaTest harness is used, the main focus is on testing of Java platform, not applications.

• Highly automated
• Documented
• Scalable
• Reliable
• Easy to debug
• Configurable

Highly automated

Documented

Test harness and framework documentation

Test documentation

Scalable

Reliable

Any resource leaks or synchronization problems in the framework code are very high priority issues.

Easy to debug

Debugging-related features range from providing test sources linked to html test descriptions and browsable from JT harness, logging capabilities, possibility to execute tests standalone without a harness. The ME Framework provides the features that are most critical for Java ME, such as exporting each test to a standalone Java ME application.

Configurable

The list is evidently not complete. Comments and additions are welcome.

Testing Java ME Implementations - AMS

API testing vs AMS testing

• 'Server/Agent' approach, used for CDC implementations, where test code is downloaded by Agent application from the Server
• 'autotest' approach, where tests are packaged into sequence of applications, that are repeatedly downloaded/executed/removed.

Today I would like to focus on limitations of these approaches, what types of functionality can not be tested this way and describe several solutions. Some of these ideas are implemented in ME Framework, there are variations, implemented in other Java ME testing products.

Talking about limitations, the main point is that using these mechanisms one can verify what is going inside of a single application when it is running. These mechanisms do not allow application restarts and crashes, assume fixed scenario and only single application running at any given time.
Note, that this is usually all that is needed to test  of Java API behavior. Just it (API) is not the only focus of standards and testing in the Java ME world.

Examples of Java ME standards, that can not be tested with the above mentioned approaches, are as follows:

• MIDP OTA specification, among other things, describes criteria, that result in an application installation failure. Tests for these requirements need to meet these criteria and verify that installation fails. API testing mechanisms work only if install&run steps pass successfully.
• MIDP PushRegistry specification talks about static registration of the Push event handlers. If application is provided with special attributes, it gets registered in the PushRegistry to handle incoming WMA messages, for example. Testing of this functionality assumes the following steps are present in the test procedure:
  - an application with Push registration is installed
  -the push event is initiated
  - application is launched to handle the event
  The scenario does not fit into the Server/Agent scheme, because it requires an application installation to happen as part of the test, and does not fit into the autotest scheme, because it does not have automatic 'run/remove' steps, that are mandatory parts of the 'autotest' cycle.
• CHAPI spec talks about communications between two applications, the ContentHandler and Invoker.

The cases above are usually associated with the platform component, named JAM (Java Application Manager) or AMS (Application Management Software), therefore the testing techniques that involve AMS operations (install/update/run/remove) or application life cycle (start/pause/stop/destroy) will be referred to as 'AMS testing'.

OTA Testing Framework

Absolute Weapon

Typically, the OTA Testing Framework consists of the following components:

• OTA test - the test scenario, that includes standard and custom AMS operations. It operates with applications associated with this test. It is executed on the server side of the system, main communication point responsible for reporting the whole test status.
  
• Test application(s) - zero or more pre-packaged applications that may contain part of the test code that needs to be executed on the device. They may communicate with the main test scenario - OTA test

There is also an important component called OTA Server. In MIDP case it is HTTP server controlled by OTA test and responsible for provisioning of test applications. Nice artistic picture of the OTA Testing Framework in provided in the ME Framework Developer's Guide (link to PDF version), check Figure 3-7.

The approach described above is universal and powerful. It can be used to test the most exotic cases. At least, so far it stays as a last absolute weapon used if no other technique works.

Last Thing to Use

The only disadvantage of OTA Testing Framework is that it is terribly interactive. During the certification run TCKs do not allow to use any automation, every AMS operation must be performed manually. With a simple install/run/remove cycle, where one needs to read instructions, enter url, wait for download/launch/execution complete/remove, it takes 1 minute minimum for a test case.

Now imagine yourself a QA engineer, who has to run CHAPI TCK, that contains ~40 OTA tests, as a regression test suite on a regular basis during the whole development cycle without any automation.

Lets consider a variation to the above approach, that can be used when it is OK to narrow the scope of covered possibilities and simplify the usage.

XletManager

The idea is to specify the Java interface to the AMS that will allow one application to perform the life cycle operations on another application. It was created for PBP/TV, where testing is based on the Server/Agent model. The scheme works in the following way:

• An implementation developer provides an implementation of XletManager interface, that would allow a test application, executed on the device to invoke the AMS operations through the Java API. In the general case, if Java interface to AMS is not available, the XletManager implementation may be interactive.
• Test Suite user manually downloads and installs on the device the Agent application and a set of Xlet applications that will be used in tests
• Tests that are executed within the Agent instantiate XletManager and operate with additional pre-installed test applications.

While in the PBP TCK this approach is used to initiate only life cycle operations from the test, executed on the device, it also can be extended to any AMS operations like install/update/run/remove.

Comparing with OTA Test Framework

Main benefit of XletManager-like approach is that it allows to use a single Server/Agent test execution model for the whole test suite.

With this specific approach, there is no dynamic application installation/removal, no application provisioning component, test applications are preinstalled before test execution begins.

There may be restrictions, that will complicate the use of this model or just make it impossible, like:

• interactive implementation of XletManager may be complicated for devices with small screen
• platform may not allow more than one application to run simultaneously.

Variations

Interface to the AMS may be implemented through plugins, that may be provided by implementation developers. These plugins may be interactive in general case or automated. Automated testing may be used for QA/regression testing or even for certification in some cases.

There are interesting possibilities of using standard Java ME APIs in order to communicate with AMS, for example CHAPI. As a minimum, it can be used to have the test code distributed across multiple MIDlets or MIDlet-suites that are registered as ContentHandler-s and invoked from the main test scenario.

Summary

Over time, the number and complexity of Java ME implementation tests dealing with AMS have increased. I believe this is a natural consequence of consistent focus on standards in the area of application management for consumer devices.

The interesting and important problem, that comes together with number of appearing and evolving AMS-testing frameworks that are often interactive is automation, it deserves separate article.

Merry Christmas and Happy New Year !

Introduction to Java ME Testing Tools

Implementation Testing vs. Application Testing

Java ME developers perform their application testing routinely with unit tests written as part of their development cycle. J2MEUnit and JMUnit work well for this: all you need to do is to package the test framework and all your tests with your application and provide an entry point to start test execution. These frameworks run your tests in a single MIDlet run and display results on the device screen. This is all what the most of the application developers need.

Things change when the object of your testing is a Java ME implementation on a mobile device (that is the entire Java ME software stack). We will talk below about the specific challenges that complicate the testing process. Some of these are specific to Java ME platform, others steam from the size and complexity of the technology.

Implementation Testing Tools

A number of tools were developed over the years to address the implementation testing needs. Some URLs:

• JCP TCK tools
• JT harness (an open source test harness)
• ME Framework (JT harness-based open source testing framework for Java ME platform)
• Java Device Test Suite (device quality test suite)

Dealing with Large Test Suites

The first issue to deal with is the test suite scalability. Due to the variety and complexity of Java ME technologies, the number of tests is large.

Let's say we wrote 10000 tests for the CLDC VM specification. If we tried to execute them in a JUnit-like test framework, we would quickly find out that

• 10000 tests do not fit into a single application, you have to split them in smaller chunks.
• some tests crash the VM. If we tried to execute all tests in a single MIDlet.startApp call, we would never see the result of any test (even those which don't crash). You need to be able to detect VM exits and catch any unexpected exceptions.
• some tests crash the VM on purpose. These are negative tests which verify that VM handles certain conditions properly (e.g. throws VerifyError).
• device runs out of memory because all class files of the test suite do not fit into it (garbage collector frees unused data from the memory, but doesn't free classes: these are garbage-collected only together with their class loader).
• tests running in the same VM interfere with each other causing failures which are hard to explain or debug. Or tests misbehave and lock up the VM (this is where you learn that shared resources and race conditions are a big deal in the testing world).
• you need an accurate report once the test run is complete: if 100 tests out of 10000 failed, you need detailed information on every failure and you don't want it displayed on the device screen. There should be a way to load test results from the device onto your server or desktop machine. In a perfect world test results for every test cycle are stored in an easily accessible database.

Since tests are typically run on a device with unstable software stack, execution of a large test suite there is an additional challenge. This is equally true for much more capable platforms like Java SE. One should expect all sorts of things to go wrong:

• Implementation bugs causing failures and crashes.
• Resource leaks resulting in memory overflow, unavailable file handlers and sockets.
• Incomplete implementation lacking networking or user interface support.

There are many other problems to be solved. For example, how do you automate testing of your Java-enabled Blu-ray player?

Let's talk about these and other problems on specific examples.

CDC and Java SE Testing Solutions

Java SE platform is where Java technology originated. Not surprisingly, this is where the testing solutions were initially worked out. CDC + Foundation Profile is a platform similar in many ways to Java SE (but without the myriad of recent extensions and features), the approach, described below, came from Java SE and works for CDC+FP.

CDC platform is richer compared with CLDC. The most important benefits we have is reflection APIs and user-defined class loaders. This makes many testing tasks easier.

The most robust testing set-up proved to be a client-server model for the test harness: the main harness application runs in a stable Java SE environment and a small agent is deployed on the device under test. The main harness application manages test suite configuration and execution, as well as the test result collection and storage. The agent connects to the harness application over the network. All the agent does is repeatedly download and execute tests (a separate class loader instance is created for each test download) as well as send test results back to the main application.

This effectively solves most of the problems outlined above:

• Small memory requirements. All you need is enough memory to hold the agent application and the largest test. Test classes are garbage-collected each time test class loader is garbage-collected, so you are not limited by the size of a test suite.
• Test interference is minimized. Since tests are loaded one-by-one, they do not affect each other and their resources are freed once execution is complete.
• No multiple application downloads during the test cycle. There is only one small agent application to download. The rest is handled automatically.
• No test results on device. You are not dependent on file system, you don't do detective work if the device dies. By the way, file system independence has the added benefit of nicely handling the security-constrained environments such as an applet.

There are several critical dependencies in this model: you need a (stable) network, class loader and a few extra security permissions. These may not be available everywhere by default, but this is topic for different discussion.

CLDC and MIDP Testing Solutions

Things get more difficult on a CLDC/MIDP stack. There is no class loader to rely on (and no reflection either). The only universal way to load test code on the device is via application download. While AMS (Application Management System) does not support automation by default, a very simple extension called "autotest" satisfies most testing needs.

The autotest feature is a special AMS mode which iteratively downloads, installs, runs and removes applications from a specified URL. Test harness makes new test applications available at this URL and the AMS picks them up.

10000 tests will be packaged into a number of test applications. Each application will include a small execution agent (to report test result back to the test harness) and the test classes. In a trivial case we will have 10000 different applications.

Shortening the Test Cycle

Test applications are downloaded over the network. The good news for MIDP is that all devices support HTTP download. The bad news is that some of these connections are as slow as 9 kbit/s. It will take 2.5 hours to download 10 MB of test classes.

And there are many other things which will affect the test run time. Even though execution agent is small, it is bundled with every application, wasting a lot of network traffic! Each test bundle download requires new a network connection; these will slow down the execution. Test data and test results are sent back and forth between the harness and the test application; extra network bandwidth is consumed. Each test needs time to execute: some tests do extensive computations and will take a few minutes to run.

In the early days of CLDC TCK 1.0 (test suite, used for certification of implementations of CLDC 1.0 specification) test execution was taking up to 2 weeks in some environments! With a few improvements, one can pass CLDC TCK in a few hours:

• Pre-install shared agent code on the device (romize classes of the test agent).
• Package several tests per downloaded application to reduce the number of network connections and AMS operations. Each device has a different maximum application size so the total number of applications will vary.
• Use multiple devices in parallel. Let the harness assemble and sort test results.

What else?

Comments are welcome.

This work is licensed under a Creative Commons License.