Alexey Popov's Blog

Tools Archives

One of the First Java Applications

Few words for a background.

The Java ^TM Compatibility Kit (JCK) is a test suite, used to verify if the Java standard is correctly implemented. The first JCK came out together with the first JDK from SUN, now this effort evolved into the industry-wide Java Community Process.

The JavaTest ^TM harness is a test monitor, used in first versions of JCK, then across multiple Java Technology Compatibility Kits (TCK), now evolved into a general purpose open source test harness.

How it started

With permission of Jonathan Gibbons, who was around when Java was born, here is his story of the JavaTest childhood:

JavaTest started round about JDK 1.0.2, JCK 1.0.2a was applet-based and did not use JavaTest;  JavaTest was introduced in JCK 1.0.2b.

It definitely was not the first application written in Java, but it is true that testing was important to Java from the beginning. JavaTest is sufficiently old that we had to develop many GUI widgets ourselves, and one of the early JavaTest developers (Tim Prinzing) went on to become a significant member in the Swing team. It is probably reasonable to say that Tim pioneered light-weight components.

Curious how it looked like back then at early JDK times ? Check here.

You can see that testing is one of keys that brought Java platform to where it is now. Java's way of compatibility testing, that is having high quality TCK required to pass to get Java logo,  is how Write Once Run Anywhere is achieved,  it is the cost of application portability and platform standardization.

On the road

Inspired by the conversation at David Herron's Blog, I was looking for analogy to illustrate the value of compatibility testing in the Java ecosystem, here it is - Java is a road, compatibility testing is its pavement. Applications are vehicles and the hard cover on the road makes it possible to drive fast. 

For Java ME platform, its fragmentation is a freedom to choose number of wheels, engine, use a bicycle or a truck. Another part, the 'dark side', is where the road cover has holes, caused by bugs in specification and implementations, test coverage problems. Services are built around the road to take care of vehicles and holes, like JDTS and Java Verified, while the road has its pavement (TCKs), differentiating it from deserts, forests and swamps.

Testing session at Java ME track at SUN TechDays, Saint Petersburg

Slides and description for it were first created in the native language of Internet and IT, that we use at work and call English by inertia. I did a first pass of translation to Russian and found that for many notions I do not have adequate words. For example, does open source mean the same as открытый код ? From the other side - with mother tongue you can make your speech so alive !

Check blogs of my colleagues who are speakers at the Java ME track: Danila (CLDC), Petr (CDC), Alexander (Testing). Presentations in English, like one of Simon Ritter about ME Gaming, will be translated online.

I will also be there at the Java ME POD. See you at SUN TechDays in Saint Petersburg !

Java ME Testing - Debugging Test Failures

What to do when tests of Java^TM ME implementation test suite fail ?

This article offers some suggestions for debugging test failures - with a special focus on the JT harness and ME Framework features that support debugging. My initial motivation for writing this article was to announce improvements in the Test Export feature, however the topic is just so entertaining that I couldn't stop there. :-)

Debugging in Java Micro Edition (ME) has number of challenges. Issues related to debugging problems in Java ME implementation can include any of the following items:

• The problem can reside at the Java layer or in the native code.
• Mobile device is usually a component of a larger system rather then isolated system.
  For example, this may be connectivity of different kinds, integration of mobile application with server-side components.
• Multiple technologies and standards are involved.
  For example, the CLDC/MIDP implementation may communicate with javacard using SATSA and with Java SE/EE using RPC API of jsr172
• The device might not provide access to the Java console or support of debugging protocols.

Problems, specific to automated Java ME test suites, have the following roots:

• Automated test execution on CLDC/MIDP requires special AMS mode, autotest.
  CDC/PBP execution mechanisms may be complex as well. Basic information on this topic can be found in the first article: Introduction to Java ME Testing Tools.
• The distributed test execution mechanism.
  This includes execution of a test with test harness (JT harness) running on the desktop and the test agent is running on the device. In addition the distributed tests themselves consist of multiple components (such as SE and ME components).
• A complex configuration mechanism.
  Before starting to run tests you need to configure test harness according to the environment and implementation under test. This complicates the debugging by making the whole system more integrated. Tighter integration makes it difficult to identify the problem by isolating the piece of the test code from the test harness.

Debugging Tips

Tip 1 - Browse the code, test specification, test output.

• recognized parameters of Test Description
• values of environment variables
• version
• time stamp
• exact command used for test execution
• test output itself
• lots of other stuff, useful and not

Tip 2 - Turn on logging

• Logger API used by the ME Framework implementation classes.
• 'Debug output' options for ME Framework components

Turn on Logger

To turn this type of logging on you need to specify the config file through java system property when starting JavaTest harness, like this:

java^
    -Djava.util.logging.config.file=D:/exec/0-configs/j2me-fw-log.properties^
    -jar javatest.jar

ME Framework Tracing Capabilities

• Test execution subsystem (CLDC/MIDP), Figure 3-4 at page 27.
  The trace lists control messages between the harness and the device that correspond to Autotest execution flow. This may be useful if test execution is very slow, like when there are multiple tests packaged per test bundle, to make sure that there is a progress, to find which exactly test crashed the VM, to see exact test parameters and test result outside of the harness. The real-time trace can be observed on the device console, if the console is available. See a snapshot from the Configuration Editor that shows which Interview questions are responsible for this option. Check server and device-side examples of trace output.
  
• Distributed test framework, Figure 3-5 at page 29.
  Here you can see all messages exchanged by this framework, who sends what. Needless to say, it is one of the important features used to investigate failures of distributed tests. Simple interactive test with 3 static images sends that many messages to pass.
  
• Agent (CDC/PBP).
  Passing '-trace' option to agent displays everything going inside of it at the console (if available). This is a standard feature of the JT harness Agent. The example of the Agent trace can serve as a good illustration to CDC-specific approaches to address test suite scalability. Note the data exchanges and test classes instantiations for every test.

Tip 3 - Execute the test standalone

Each individual test usually has an application entry point, allowing to execute it outside of the test harness. This is usually static main() taking an array of parameters (the same parameters as those passed to the test by the test harness). You can put test classes in the classpath and call the test by the class name.

To get values of test parameters you may need to dig into test environment. To execute standalone test corresponding to the following sample test description:

you must execute the following command:

export sampleValue=<find 'sampleValue' variable in the test environment>
java -classpath $TEST_HOME/classes^
    com.sun.tck.satsa.tests.api.java.security.KeyFactory.generatePublicTests^
    -envValue $sampleValue

Or even simpler - JT harness result file contains the exact command and values of parameters that must be executed. See section messages:(1/693) in the example.

Unfortunately this works well only in CDC and Java SE execution modes. This approach will not work for CLDC/MIDP, where you need the application to be packaged and deployed before you can execute it.

Tip 4 - store for offline analysis

Tip N - use common sense

• Vary configuration parameters
• Add trace printing into test itself or into tested code
• Execute the test on the another implementation to verify its correctness
• ...

Test Export

The ME Framework is a quite complex set of interacting components involved in test execution. When a test is reported as failed by the JT Harness it's not always evident where the bug is and how to track it down. Most often, when users are trying to track down a bug, they want to run the failed test standalone on the device under test. This is where the Test Export feature can help. Test Export converts an individual test into a small standalone Java ME application containing the test, which can be executed, debugged, or used in any way convenient for the user.

Getting Test Export as a side-effect of Autotest execution

A very simple, partial implementation of the test export was available in the ME Framework from the beginning. It was a hack of a standard CLDC/MIDP execution mechanism., described by the following scheme.

FIGURE X. Autotest mode.

In order to turn test into CLDC/MIDP application that executes separately from the harness, one must replace all these arrows, representing some message exchanges, with static data.

For the initial ME Framework implementation of the test export feature, test packages where supplied with all information necessary for test execution (that replaced links 3 and 4). The execution server was packaging test bundles without waiting for any requests from AMS (links 1 and 2). For the test harness, test status indicated success of packaging and not actual test execution (link 5). When executed, tests did not send result to the execution server but just printed them on the console (link 5). Tests were packaged with all test data but not automatically downloaded to the device and test packages not removed after execution cycle completed.

Even this simple and limited approach allowed to solve some troubleshooting-related tasks. For example, a QA engineer responsible for TCK execution on the development build of the implementation could use these exported test packages as attachments to bug reports. A development engineer not familiar with TCK execution could use it to reproduce the problem and verify the fix.

Test Export in ME Framework 1.2

The previous implementation of the Test Export was not complete. Evident limitations were that only simple non-distributed tests were exported and not test sources. For the user, this meant that a created application could not be modified and rebuilt. The previous implementation also exported only .jar files and not .jad files.

Implementing the full Test Export feature for all types of tests is a challenging task requiring very accurate tracking of all test dependencies. The following are examples of such dependencies:

• Custom jad/manifest entries
• Test specific security conditions
• Code and resource dependencies of test and test suite level

Identifying all test sources associated with a distributed network test is not a simple task. To make this feature really user-friendly, one should provide parts of the test execution infrastructure, such as the provisioning server for MIDP.

To initiate Test Export for a ME Framework 1.2-based test suite, the user should first open the Configuration Editor and, in the interview, set test export mode on. Then the user should specify the export directory (the location where exported test will be saved) and maybe some additional parameters, such as the 'Prefix of URLs to JAR files' parameter specific for the MIDP platform. See the following screen-shot of the JT harness Configuration Editor: Configuring Test Export.

Finally, the user should select the test or tests to export and run them just as in a regular test run. In test export mode, instead of executing tests on a device, the selected test or tests are saved in the specified test export directory.

What you get with test export?

When the test export is done, the following content is created in the export directory:

• Java ME applications in JAR files named test1.jar, test2.jar, etc. containing the exported tests.

Just as with any other Java application, these applications can be uploaded on any suitable device and launched. The exported tests are executed in the same way that they are executed by ME Framework but in standalone mode, printing the diagnostics and test result on the device screen.

• In MIDP mode, Java application descriptors (JAD files) are generated: test1.jad, test2.jad, etc.

These descriptors are properly configured:

• They contain the same attributes as in regular test runs.
• They are signed and include certificates if the test suite was configured for trusted MIDP security mode.
• They contain proper links to the exported JAR files.

You can use the generated JAD files for OTA provisioning of exported tests by putting them on any suitable HTTP server or by using the special provisioning server included with the ME Framework.

• Java sources of exported tests in src subdirectory.

These source files can be used for debugging or rebuilding the exported tests.

• Class files in classes subdirectory.

• Ant build script in build.xml file.

Yeah! Users can change the exported tests by modifying its sources and rebuilding them with this build script. It contains all necessary targets for compiling sources, updating JAR files from classes, and signing JAD files. In addition, build.properties is also generated in the export directory. It defines properties used by the build script and can be used to configure the build. For example, property trusted, if defined, triggers signing JAD files when updating exported tests.

• Additional libraries and tools in lib subdirectories.

These are not part of exported tests and are not required for using them, but can be also helpful. For example, exportSigner.jar is a tool for updating signatures and certificates of MIDlet suites with tests in those cases where the tests contained in it have been modified and its JAR file has changed.

Let's look at this new feature from the point of view of a Sun Java Wireless Toolkit (Wireless Toolkit) user. Before the test export feature was introduced, users could execute tests on the emulator by using its autotest mode:


$WTK_HOME/bin/emulator -Xautotest http://localhost:8080/test/getNextApp.jad


This command launches the emulator and specifies an URL at which the emulator should look for tests to download. At this URL resides the execution server of ME Framework. The execution server responds to 'getNextApp.jad' requests by sending new and new tests from a large test suite. The emulator fails in a loop repeating three steps (install, execute, remove) for each downloaded test until the execution server has no more tests to send. There was no way to get just one test (corresponding JAD and JAR files) for standalone execution. In addition, users of the Wireless Toolkit know that autotest mode doesn't allow debugging!

Now, with the test export feature, a user can select the test to debug and export it to any appropriate location. JAR and JAD files are generated and can be executed on emulator. The following is a typical command for launching the exported test on the Wireless Toolkit emulator:


$WTK_HOME/bin/emulator -Xdescriptor $EXPORT_DIR/test3.jad


During its execution, the test prints the output to stdout. Click here to view an example.

This time, debugging the test during its execution is easy. Just add the -Xdebug argument to last command with all necessary parameters. You can then use your preferred debugger with the IDE of your choice to debug the test with its sources which are also exported.

Future Plans

The next planned improvement for debugging support will come from the integration of  Skavas's GUI Agent. This will allow you to use GUI and RMS in addition to console output for execution of exported tests. Watch for more information about this in the future - it's a topic for another post.

The Test Export feature is under development right now and you can improve it by posting on ME Framework forums. Propose your improvement or new feature and you may see it in one of the next releases.

Testing Command-Line Applications using Golden-File approach

The story describes our experience of using JavaTest-style test format and JavaTest itself as a test harness for functional and regression tests. The tested item is a command-line application.

Background

The most frequently observed approach to functional test development for such simple cases is to have a set of shell scripts placed into a directory hierarchy. Each of these scripts performs one simple check and reports results to the console. Tests are run by a trivial test harness, usually another shell script, that performs test iteration and joint status reporting. Test specifications usually come separately from tests.

How it was done here

Every individual test description in this test suite contains information on how to launch the tool and how its output should be verified. All tests in this test suite are written using the "golden-file" approach. The golden data is:

• application output streams both stderr and stdout
• exit code
• set of output files

The test development scenario for the new test is as follows:

• Create test description, including:
• test case specification (textual, optional)
• parameters, comparison procedure to use
• Run test in the 'setup' mode
• Validate output and mark it as 'golden file'

Test Description example:

• Description - Contains verbal Test Case Specification
• The table describes actual test cases that will be executed. Description is done in the language of JavaTest's HTMLTestFinder. Below you can see how these specific descriptions are interpreted by the execution Script, which used for this test suite:
  
  

  title	Unique ID of the test case
  executeArgs	Command-line arguments to pass to the tool.
  keywords	This field is used by main test filtering engine of JavaTest harness. Not used in this test suite.
  executeClass	Which of predefined golden file comparators to use for this test

Pros and Cons 

Pros:

• All traditional prerequisites come from using a specialized test harness, like test execution and test result management, test specification, source, and test result browsing, parallel test execution, test exclusion and filtering, multiple environments support (JDK version, for example) etc etc. 
• We got highly automated and maintainable test suite, preserved high consistency through all tests
• Test development is fast and simple
• Application can be tested at multiple platforms by design. When testing Java applications that need to be verified in multiple environments, it is critical to avoid a dependency on a specific scripting language used for test development.
• We avoided information duplication. The test code is linked to the  test specification and a test report

• Good test harness is a complex tool that takes time to study. Infrastructure may need to be adjusted for it.
• The "golden file" approach is naturally limited, can not be enough to cover everything
• When output is unstable, this approach may not work or may require sophisticated comparators. The most obvious example is in processing timestamps.

Test Harness for Product-Quality Test Suites

The requirements for the quality of test suites vary depending on the engineer's goal, the product being tested, how many and what kind of people will be using the test suite, how often and for how long, and how the test suite will be maintained. These factors may vary depending on whether you write unit tests for yourself or your project, or if, as a member of the SQEgroup on a large project, you are developing a test suite that will be executed by the SQA group. 

Sun Microsystems has a long successful story of development large test suites of various types. The development of the JavaTest  harness was highly influenced by its adoption in Java ^TM Technology Compatibility Kits (TCK). The majority, if not all of Sun's TCKs are built on the JavaTest harness. The key consequences of this:

• these test suites (TCKs) are products, developed by one group to be used outside of it on a constant basis for significant period of time.
• 'used in all TCKs from SUN' means that test suites built on JavaTest harness were run on all Java-compatible implementations in the world - just because implementations become Java-compatible after passing the TCK. It means a lot !
• though it is not the only area where JavaTest harness is used, the main focus is on testing of Java platform, not applications.

• Highly automated
• Documented
• Scalable
• Reliable
• Easy to debug
• Configurable

Highly automated

Documented

Test harness and framework documentation

Test documentation

Scalable

Reliable

Any resource leaks or synchronization problems in the framework code are very high priority issues.

Easy to debug

Debugging-related features range from providing test sources linked to html test descriptions and browsable from JT harness, logging capabilities, possibility to execute tests standalone without a harness. The ME Framework provides the features that are most critical for Java ME, such as exporting each test to a standalone Java ME application.

Configurable

The list is evidently not complete. Comments and additions are welcome.

Testing Java ME Implementations - AMS

API testing vs AMS testing

• 'Server/Agent' approach, used for CDC implementations, where test code is downloaded by Agent application from the Server
• 'autotest' approach, where tests are packaged into sequence of applications, that are repeatedly downloaded/executed/removed.

Today I would like to focus on limitations of these approaches, what types of functionality can not be tested this way and describe several solutions. Some of these ideas are implemented in ME Framework, there are variations, implemented in other Java ME testing products.

Talking about limitations, the main point is that using these mechanisms one can verify what is going inside of a single application when it is running. These mechanisms do not allow application restarts and crashes, assume fixed scenario and only single application running at any given time.
Note, that this is usually all that is needed to test  of Java API behavior. Just it (API) is not the only focus of standards and testing in the Java ME world.

Examples of Java ME standards, that can not be tested with the above mentioned approaches, are as follows:

• MIDP OTA specification, among other things, describes criteria, that result in an application installation failure. Tests for these requirements need to meet these criteria and verify that installation fails. API testing mechanisms work only if install&run steps pass successfully.
• MIDP PushRegistry specification talks about static registration of the Push event handlers. If application is provided with special attributes, it gets registered in the PushRegistry to handle incoming WMA messages, for example. Testing of this functionality assumes the following steps are present in the test procedure:
  - an application with Push registration is installed
  -the push event is initiated
  - application is launched to handle the event
  The scenario does not fit into the Server/Agent scheme, because it requires an application installation to happen as part of the test, and does not fit into the autotest scheme, because it does not have automatic 'run/remove' steps, that are mandatory parts of the 'autotest' cycle.
• CHAPI spec talks about communications between two applications, the ContentHandler and Invoker.

The cases above are usually associated with the platform component, named JAM (Java Application Manager) or AMS (Application Management Software), therefore the testing techniques that involve AMS operations (install/update/run/remove) or application life cycle (start/pause/stop/destroy) will be referred to as 'AMS testing'.

OTA Testing Framework

Absolute Weapon

Typically, the OTA Testing Framework consists of the following components:

• OTA test - the test scenario, that includes standard and custom AMS operations. It operates with applications associated with this test. It is executed on the server side of the system, main communication point responsible for reporting the whole test status.
  
• Test application(s) - zero or more pre-packaged applications that may contain part of the test code that needs to be executed on the device. They may communicate with the main test scenario - OTA test

There is also an important component called OTA Server. In MIDP case it is HTTP server controlled by OTA test and responsible for provisioning of test applications. Nice artistic picture of the OTA Testing Framework in provided in the ME Framework Developer's Guide (link to PDF version), check Figure 3-7.

The approach described above is universal and powerful. It can be used to test the most exotic cases. At least, so far it stays as a last absolute weapon used if no other technique works.

Last Thing to Use

The only disadvantage of OTA Testing Framework is that it is terribly interactive. During the certification run TCKs do not allow to use any automation, every AMS operation must be performed manually. With a simple install/run/remove cycle, where one needs to read instructions, enter url, wait for download/launch/execution complete/remove, it takes 1 minute minimum for a test case.

Now imagine yourself a QA engineer, who has to run CHAPI TCK, that contains ~40 OTA tests, as a regression test suite on a regular basis during the whole development cycle without any automation.

Lets consider a variation to the above approach, that can be used when it is OK to narrow the scope of covered possibilities and simplify the usage.

XletManager

The idea is to specify the Java interface to the AMS that will allow one application to perform the life cycle operations on another application. It was created for PBP/TV, where testing is based on the Server/Agent model. The scheme works in the following way:

• An implementation developer provides an implementation of XletManager interface, that would allow a test application, executed on the device to invoke the AMS operations through the Java API. In the general case, if Java interface to AMS is not available, the XletManager implementation may be interactive.
• Test Suite user manually downloads and installs on the device the Agent application and a set of Xlet applications that will be used in tests
• Tests that are executed within the Agent instantiate XletManager and operate with additional pre-installed test applications.

While in the PBP TCK this approach is used to initiate only life cycle operations from the test, executed on the device, it also can be extended to any AMS operations like install/update/run/remove.

Comparing with OTA Test Framework

Main benefit of XletManager-like approach is that it allows to use a single Server/Agent test execution model for the whole test suite.

With this specific approach, there is no dynamic application installation/removal, no application provisioning component, test applications are preinstalled before test execution begins.

There may be restrictions, that will complicate the use of this model or just make it impossible, like:

• interactive implementation of XletManager may be complicated for devices with small screen
• platform may not allow more than one application to run simultaneously.

Variations

Interface to the AMS may be implemented through plugins, that may be provided by implementation developers. These plugins may be interactive in general case or automated. Automated testing may be used for QA/regression testing or even for certification in some cases.

There are interesting possibilities of using standard Java ME APIs in order to communicate with AMS, for example CHAPI. As a minimum, it can be used to have the test code distributed across multiple MIDlets or MIDlet-suites that are registered as ContentHandler-s and invoked from the main test scenario.

Summary

Over time, the number and complexity of Java ME implementation tests dealing with AMS have increased. I believe this is a natural consequence of consistent focus on standards in the area of application management for consumer devices.

The interesting and important problem, that comes together with number of appearing and evolving AMS-testing frameworks that are often interactive is automation, it deserves separate article.

Merry Christmas and Happy New Year !

This work is licensed under a Creative Commons License.