Alexey Popov's Blog

May 2007 Archives

Simplest Test Suite built on ME Framework

It is targeted to people who use (or plan to use) ME Framework or JT harness or work in the area of Java ME testing.

Foreword

Before the open source release, we were developing and using the ME Framework internally for years, as long as Java ME exists. As it turned out, to start using it outside of the internal SUN environment and supporting infrastructure is not a trivial task. Unlikely these problems are unique to our project.

Some things  became too natural for internal users. Some approaches that we use internally when develop TCKs with ME Framework help to optimize the work of the big production organization, responsible for development and sustaining of a big number of products. It turns out that people who need to do a first step, like to create a single test suite once, need more simple ways to do simple things.

Simplest Test Suite

I would like to go into details of two issues, that were addressed in the ME Framework 1.2 3rd Development release and are demonstrated in this sample:

• Allowed alternative test formats, not limit to html test description, that have to be created for each test.
• Allowed not to provide explicit  information on what class and resource files are associated with test. This is necessary for automated packaging of test applications (bundles) and works now for simple tests.

Simple way to describe tests

TestFinder in JT harness is the class, responsible for providing array of TestDescription elements, that are representing test meta data in JT harness' model of the test suite. Support of different formats of tests is done in JT harness by plugging in corresponding extension of the TestFinder.

In short, the first change in ME Framework was just enabling TagTestFinder in the .jtt file. TagTestFinder uses javadoc tags to identify test and describe its meta information and it is a TestFinder, used in JTReg. HTMLTestFinder is more familiar to TCK users, it describes tests in specially formatted html table, check here for an example.

Simplest Test Suite uses tags to describe tests - it allows to decrease number of files, naturally supported by IDEs and somewhat familiar to people with JUnit experience. Check how the sample test looks like.

Other kinds of TestFinder

Other then Tag and Html finders, the standard set of TestFinder implementations, provided in JT harness, includes BinaryTestFinder, that reads information on tests to run from the binary file. It is the one that is most often used in TCKs, where there is a need to optimize time to read huge test tree. BinaryTestFinder is most applicable to cases, where test suite does not change after its development is done, because any change in any test description requires regeneration of the binary test data.. BinaryTestFinder is used in conjunction with another  TestFinder that reads test descriptions from tags or html files during the build of the test suite and stores them in the serialized form.

It's just sort of "extension" on top of other Finders, that"sits" on top of them, and gets the Descriptions from other Finders and serialize them to the binary data, and also allows to read the descriptions from the serialized data. It can be used with any other TestFinders that that do the actual job of parsing the TestDescriptions from HTML, javadocs, etc.

How to find what content to put into the .jar of the test application

This topic is more complex and hence more entertaining one.

The problem in more details. ME Framework allows for a great flexibility in how to turn huge pile of class and resource files of the test suite into set of MIDP application packages for iterative execution through autotest. Criteria that control creation of packages can be

• max size of the jar file
• number of tests to package into the single jar
• tests themselves may describe some conditions in their associated meta data that would regulate how these tests need to be packaged

Tests are packaged into .jar at runtime to allow the user to vary packaging parameters to achieve different goals, primarily to optimize network traffic and test execution time.

This all means - to form various application packages ME Framework needs to know which class and resource files are associated with each test.

These data can be calculated automatically using several methods, I will talk about them later. What we did for this sample: TestDescription in JT harness requires definition of the 'executeClass' parameter for every test. The value of this field is the name of the class that is test entry point, implementing one of supported interfaces.  If test consists from the single .java source file that defines single class - nothing to calculate, right  ? All class files of this test can be described by /$executeClass.*/ expression. Note that inner classes are OK here as well.

Other then this, nothing else happens in this sample automatically - if there are multiple .java files, that constitute a test, one need to list them manually in the 'resources' field.  Not that convenient as 'do nothing', but simple to explain :) Other then 'resources' field, the standard mechanism with special file named testClasses.lst still works, just generation of this file is not supported in the build of the Simplest Test Suite. The file testClasses.lst should be located in the special place in test suite's directory hierarchy and containing testURL-test classes pairs, check how it looks like in another sample.

Approaches to automatically calculate list of classes to package

Approach 1 - individual test compilation.

One way to do this is to put the load on the compiler. Compile tests individually, set own target dir for each test, not give and test library classes on the classpath but provide reference to their source directory instead. All class files, that are needed by the test, will be in the target directory of the compiler. The described approach is simple and reliable one, we used it for some TCK releases. Weak places of it is that it is slow, at least in our implementation where java compiler and preverifier were executed through the command line for each test. 

Approach 2 - parsing class files for dependencies.

 > java -cp jini2_1\lib\tools.jar;%JAVA_HOME%\lib\tools.jar com.sun.jini.tool.ClassDep ^
    -cp <ALL TEST AND FW CLASSES> <executeClass> ^
    -out com.sun.tck.cldc -out com.sun.tck.j2me -out java -out sun -out javax

In cases where all necessary classes can not be found through parsing class files from the entry point, we used static, manually created table, that is merged into the generated testClasses.lst. This can be just a table with the same format, or, as mentioned above, just a 'resources' field of the TestDescription.

Limitations

Yet more simplification

Further simplifying of this test suite and test development approach is possible by making it close to JTReg and xUnit. This can be:

• Minimizing set of mandatory meta data. For example, if a test consists from the single .java file, test entry point, that is now specified in the mandatory 'executeClass' parameter. is known.
• If this suite is used in the scenario, where tests are actively and constantly changing, it may be possible to eliminate 'build' stage, as it is done in JTReg,  implement test source compilation/preverification as a step in the test execution process. 

Acknowledgment

I would like to express my appreciation to the hosting provided by that allows to link sources for java.net projects from this blog a nice way that you can see above.

Interactive Tests for Java ME

Most of definitions and examples are given using the terminology of JT harness and ME Framework, but should be generic enough for the area. For details on what is TCK, JT harness and ME Framework refer to previous articles. Skip the background section if you are familiar with the subject.

Background

Why not everything is automated

For every specific implementation, testing of this functionality can be automated as well with different external tools or specific APIs. But first, not always - compatibility tests, for example, can not rely on any specifics because have to be correct for any compatible implementation, and second, this is a separate big topic, to be covered in another article.

What is meant by interactive tests

Requirements

Speaking from point of view of TCK development, the most important requirement is to make the test suite easiest for use. This means the less interactive tests it has, the better. If we have to have interactive tests, it is important to make their execution simple.

To achieve this we limited number of test types that we use in TCKs to very few. As a result, some features, that are not best suited to be tested by these test types, require multiple test cases to be written where we would write one 'custom' test case. The benefit is that TCK user has uniform interactive model, uniform interface to browse test instructions etc. This also allows for simpler external automation system development.

Types of TCK interactive tests

Note that these tests are not just set of user instructions 'do this - verify that', they include test code that can do most of work.

Java ME Specifics and Solutions

PJava story - first TCK alt bundle

Interactive tests created for Java SE TCKs are usually interactive applications, that run on the platform under test and show user instructions and test panel in a single window. This approach does not always work for Java ME for many reasons. First time we started doing interactive tests in the world of consumer devices, that was Personal Java, we found that tests we created can not be passed on PJava devices with single Frame limitation and small screen - test instructions and test panel were placed in the container without scrolling capabilities, these 'Done' and 'Yes/No' buttons may not appear on screen in some circumstances. To address this we issued first 'alternative TCK test bundle' that just enabled scrolling. After that passing of interactive tests became possible, though not convenient. Check how AgentFrame interface looks like on PJava with Truffle toolkit. The scheme of these interactive tests is the one that is still used in Java SE TCKs, can be described as follows:

Figure I. Simple Interactive Test.

MIDP TCK 1.0 interactive tests

Interactive tests for MIDP TCK 1.0 were executed using the same Autotest mechanism as regular automated tests, the difference was that interactive tests expected some user actions and were grouped together for convenience. These tests were developed using brand-new MIDP API and same approach that was used in JCK and PJCK. Instructions, test panel, all user interface components of these tests were displayed on the micro screen of MIDP 1.0 micro devices. Given big number of interactive tests, that were necessary to verify MIDP 1.0 GUI API, running MIDP 1.0 TCK on a regular basis during the development process, was a headache.

You can see the screen shot of the MIDP 1.0 emulator to the right. Click on it to see the sequence of screens that constituted the MIDP TCK 1.0 interactive test. As you can see, there is a big number of interactions, that are not related to execution of the test but for scrolling, switching controls etc.

Distributed Interactive test framework

MIDP TCK 2.x interactive tests

To address this problem of inconvenience of interactive tests on MIDP devices with small screen, the solution was very simple and usability improvement was huge. Briefly, these tests were rewritten to using Distributed Test library to have minimal functionality on the device and have user instructions and controls, related to the test logic, on the server part. Now to run tests on the device one need to stare to the desktop monitor, read instructions, press some buttons on the desktop, for example, to initiate tested process on the device, do some interaction with the device as necessary, state pass/fail result on the desktop if needed.

The important feature of Distributed Test framework, that lies underneath new Interactive Test Framework is that there are java components of the test, that reside on server and client sides and can work together to calculate test result. One can use server side technologies in the test, for example, for reference purpose to verify that same technology works properly at Java ME side.

The scheme of distributed interactive test can be described as follows

:

Figure II. Distributed Interactive Test.

In the example of the interactive test for sound, you can see all GUI of this test. Device part of this specific test does not have GUI at all, all that device does - produces sounds that are initiated from the server side, pass/fail criteria is specified on the server side of the test as well.

PBP TCK 1.0 interactive tests

Same solution was used for interactive tests for Personal Basis Profile TCK. Though PBP API is subset of J2SE API, reuse of JCK tests was not possible there - as I mentioned, these tests combine in a single application all instructions and controls, that were not available. PBP does not have Panel, Button - no any UI widgets, only Component, Container and Window Frame. To reproduce anything on the device screen we would have to draw it using graphics primitives from scratch and creating UI toolkit was not in the scope of the TCK.

Separating functionality and interaction between server and client parts of the tests, that were created using Distributed Test framework, worked well for PBP. In the example of the keyboard test you can see the same situation as in already referenced MIDP TCK 2.0 example, all GUI is on desktop screen, device does not have any GUI, just accepts key presses and pass them to the server side.

AGUI TCK 1.0 interactive tests

Yet another special solution to workaround small device screen was implemented for AGUI TCK 1.0.

AGUI stands for Advanced Graphics and User Interface, that assumes lots of interactive tests. For AGUI the execution model for interactive tests was also the same as for regular automated tests, all tests were executed in the same Agent, interactive tests grouped separately from automated. Read here about MIDP and CDC execution modes of ME Framework.

As AGUI API is a subset of Java SE API, specifically, Swing, our goal was to reuse as much of JCK interactive tests for this API as possible. The reason here is not only time saving, it is also an additional way to ensure compatibility across different Java platforms.

What we did is we again separated tests to different pieces that could be displayed separately and organized test UI as Tabs. It was relatively easy to do, since JCK interactive tests library assumed some structuring, there was implementation of this library done with Swing subset, that fit into AGUI API. Though it was still necessary to do extra clicks to switch between different tabs and sometimes scroll through the tab, the effect was a significant usability improvement comparing to having everything in the same window altogether.

You can see the screen shot of the AGUI 1.0 emulator to the left. Click on it to see the sequence of screens that constituted the AGUI TCK 1.0 interactive test.

It was a temporary solution, once we structured tests to independent pieces, it was easy to execute these components on distributed components. This was not a conversion of simple interactive tests to distributed tests but a special execution framework, that could execute simple interactive tests in the distributed way and gave some other minor usability improvements. Overall, Interactive tests in the AGUI TCK 1.0 when it was released looked exactly like all other distributed interactive tests for Java ME.

Tests with static image

This approach is natural one, it can be used to test 80% of functionality, related to user interaction. Some technologies use it as the only approach for testing, for example W3C SVG test suite. It can be used to test low level graphics and behavior of high-level user interface components.

Even when these tests are too primitive and require multiple test cases to be developed when few more sophisticated ones could be enough, the simplicity of development and execution, possibility to have consistency across many tests could be a reason to use this approach even when it is not the most convenient.

Static image tests with ME Framework

To simplify test execution we display multiple images, related to the same functionality, in a single window with test instructions. Every reference image is accompanied with a Test button, that initiates reproduction of this image on the device under test, ad verbal description. Check the example, the screenshot of the instructions dialog and device side for tests for Java Binding to Open GL ES API (JSR 239) .

Combining multiple related images that could be switched in any sequence allows not only for comparison of the result on the screen with reference, but also for checking of transition one image to another on the device. This, as well as verbal description of the scene, may be important when reference images were done on the implementation, that is radically different with one under test and comparing of test and reference image alone can not provide confidence that tested functionality behaves correctly.

This work is licensed under a Creative Commons License.