Skip to main content

Simplest Test Suite built on ME Framework

Posted by alexeyp on May 14, 2007 at 7:40 PM PDT

This article is about some changes in the href="https://cqme.dev.java.net/framework.html">ME Framework
that
were made in response to the feedback that we got from early adopters
and are available in the third development release of ME Framework 1.2
that Mikhail
just href="https://cqme.dev.java.net/servlets/NewsItemView?newsItemID=4991">announced.

It is targeted to people who use (or plan to
use) ME
Framework or JT
harness
or work in the area of Java ME testing.

Foreword

Before the open source release, we were developing and using the ME
Framework internally for years, as long as Java ME exists. As it turned
out, to start using it outside of the internal SUN environment and
supporting infrastructure is not
a trivial task. Unlikely these problems are unique to our project.

Some things  became too natural for
internal users. Some approaches
that we
use internally when develop TCKs with ME Framework help to
optimize the work of the big production organization, responsible for
development and sustaining of a big number of products. It turns out
that people who need to do
a first step, like to
create a single test suite once, need more simple ways to do simple
things.

Simplest Test Suite

The href="https://cqme.dev.java.net/source/browse/cqme/trunk/samples/meframework/SimplestTestSuite/?rev=667#dirlist">Simplest
Test Suite
example,
that Alexander (aka Skavas) has recently checked into
repository, is what its name means - simplest possible test suite, that
we found reasonable for this product. It consists from three files -
ant build script, test suite .jtt parameter file and a sample test, If
you follow the instructions from
the href="https://cqme.dev.java.net/source/browse/cqme/trunk/samples/meframework/SimplestTestSuite/readme.txt?rev=667&view=markup">readme
file, it will take 5 min
to build and launch it plus time to download.

I would like to go into details of two issues, that were
addressed
in the ME Framework 1.2 3rd Development release and are demonstrated in
this
sample:

  • Allowed alternative test formats, not limit to html test
    description, that have to be created for each test.
  • Allowed not to provide explicit  information on
    what class and
    resource files are
    associated with test. This is necessary for automated packaging of test
    applications
    (bundles) and works now for simple tests.

Simple way to describe tests

The ME Framework is a plugin to href="https://jtharness.dev.java.net">JT harness
(aka JavaTest TM harness), that enables Java ME
test development and execution. This specific change was to correct
usage of JT harness features by ME Framework, specifically, TestFinder.

href="http://fisheye4.cenqua.com/browse/jtharness/trunk/code/src/com/sun/javatest/TestFinder.java?r=49">TestFinder
in JT harness is
the class, responsible for providing array of href="http://fisheye4.cenqua.com/browse/jtharness/trunk/code/src/com/sun/javatest/TestDescription.java?r=49">TestDescription
elements, that are representing test meta data in JT harness' model of
the test suite. Support of different formats of tests is done in JT
harness by plugging in corresponding extension of the TestFinder.

In short, the first change in ME Framework was just enabling href="http://fisheye4.cenqua.com/browse/jtharness/trunk/code/src/com/sun/javatest/finder/TagTestFinder.java?r=49">TagTestFinder
in the .jtt file.
TagTestFinder uses javadoc
tags to
identify test and describe its meta information and it is a TestFinder,
used in JTReg href="http://fisheye4.cenqua.com/browse/jtharness/trunk/code/src/com/sun/javatest/finder/HtmlTestFinder.java?r=49">HTMLTestFinder
is more familiar to TCK users,
it describes tests in specially formatted html table, check href="http://weblogs.java.net/blog/alexeyp/archive/2007/02/testing_command_1.html">here
for an example.

Simplest Test Suite uses tags to describe tests - it allows to
decrease number of files, naturally supported by IDEs and somewhat
familiar to people with JUnit experience. Check how the href="http://fisheye4.cenqua.com/browse/cqme/trunk/samples/SimplestTestSuite/tests/example/SimpleTests.java?r=664">sample
test looks like.

Other kinds of TestFinder

Other then Tag and Html finders, the standard set of
TestFinder implementations, provided in JT harness, includes
href="http://fisheye4.cenqua.com/browse/jtharness/trunk/code/src/com/sun/javatest/finder/BinaryTestFinder.java?r=49">BinaryTestFinder,
that reads information on tests to run from the
binary file. It is the one that is most often used in TCKs, where there
is a need to optimize
time to read huge test tree. BinaryTestFinder is most applicable to
cases, where test suite does not change after its development is done,
because any change in any test description requires regeneration of the
binary test data.. BinaryTestFinder is used in conjunction with
another  TestFinder that
reads test descriptions from tags or html files during the build of the
test suite and stores them in the
serialized form.

It's just sort of "extension" on top of other Finders,
that"sits" on
top of them, and gets the Descriptions from other Finders and serialize
them to the binary data, and also allows to read the descriptions from
the serialized data. It can be used with any other TestFinders that
that
do the actual job of parsing the TestDescriptions from HTML, javadocs,
etc.

How to find what content to put into the .jar of the test
application

This topic is more complex and hence more entertaining one.

The problem in more details. ME Framework allows for a great
flexibility in how to turn huge pile of class and resource files of the
test suite into set of MIDP application packages for iterative
execution through href="http://weblogs.java.net/blog/alexeyp/archive/2006/11/testing_java_me_1.html#Autotest">autotest.
Criteria that control
creation of packages can be

  • max size
    of the jar file
  • number of tests to package into the single jar
  • tests
    themselves may describe some conditions in their associated meta data
    that
    would regulate how these tests need to be packaged

Tests are packaged
into .jar at runtime to allow the user to vary packaging parameters to
achieve different goals, primarily to optimize network traffic and test
execution time.

This all means - to form various application packages ME
Framework needs to know which class and resource files are associated
with each test.

These data can be calculated automatically using several
methods, I will talk about them later. What we did for this sample:
TestDescription in JT harness requires definition of the
'executeClass' parameter for every test. The value of this field is the
name of the class that is test
entry point, implementing one of supported interfaces.  If
test
consists from the single .java source
file that defines single class - nothing to
calculate, right  ? All class files of this test can be
described by /$executeClass.*/ expression. Note that inner classes are
OK here as well.

Other then this, nothing else happens in this sample
automatically - if there are multiple .java files, that constitute a
test, one need to
list them manually in the
'resources' field.  Not that convenient as 'do nothing', but
simple to explain :) Other then 'resources' field, the standard
mechanism with
special file named testClasses.lst
still works, just generation of this file is not supported in the build
of the Simplest Test Suite.
The file testClasses.lst should be located in the special place in test
suite's directory hierarchy and containing testURL-test classes
pairs, check href="https://cqme.dev.java.net/source/browse/cqme/trunk/samples/meframework/SampleTestSuite/build/testclasses.lst?rev=333&view=markup">how
it looks like in another sample.

Approaches to automatically calculate list of classes to
package

Content of the testClasses.lst is calculated statically during the
build of
the test suite.

Approach 1 - individual test compilation.

One way to
do this is to put the load on the compiler. Compile tests individually,
set own target dir for each test, not give and test library classes on
the classpath but provide reference to their source directory instead.
All class files, that are needed by the test, will be in the target
directory of the compiler. The described approach is simple and
reliable one, we used it
for
some TCK releases. Weak places of it is that it is slow, at least in
our implementation where java compiler and preverifier were executed
through the command line for each test. 

Approach 2 - parsing class files for dependencies.

We do this with a simple tool based on JINI's href="http://java.sun.com/products/jini/2.0/doc/api/com/sun/jini/tool/ClassDep.html">ClassDep
API.



It does static class analysis and should calculate all dependencies
correctly except for cases like Class.forName
is used in the tests
(which is rarely
needed).
There are probably other tools which can do the same thing.




Here is the example of the command, that can be used in the test suite
build script:

 >
java -cp
jini2_1\lib\tools.jar;%JAVA_HOME%\lib\tools.jar
com.sun.jini.tool.ClassDep ^

    -cp <ALL TEST AND FW
CLASSES> <executeClass> ^

    -out com.sun.tck.cldc -out
com.sun.tck.j2me -out java -out sun -out
javax

In cases where all necessary classes can not be found through parsing
class files from the entry point, we used static, manually created
table, that is merged into the generated testClasses.lst. This can be
just a table with the same format, or, as mentioned above, just a
'resources' field of the TestDescription.

Limitations

Build script  for the sample will work only with simple
automated
tests for MIDP. Support of distributed, interactive, OTA or CDC tests
will require more sophisticated build procedure, that will take care of
compiling SE-side components, for example.

Yet more simplification

Further simplifying of this test suite and test development
approach
is possible by making it close to JTReg and xUnit. This can be:

  • Minimizing set of mandatory meta data. For example, if a
    test
    consists from the single .java file, test entry point, that is now
    specified in the mandatory 'executeClass' parameter. is known.
  • If this suite is used in the scenario, where tests are
    actively
    and constantly changing, it may be possible to eliminate 'build' stage,
    as it is done in JTReg,  implement test source
    compilation/preverification as a step in the test execution
    process. 

Acknowledgment

I would like to express my appreciation to the href="http://www.cenqua.com/fisheye"> src="http://www.cenqua.com/images/fisheyed.gif"
alt="Project Supported by FishEye" border="0" height="33"
width="89"> hosting provided by href="http://www.cenqua.com/"> src="http://www.cenqua.com/images/cenquad.gif"
alt="Supported by Cenqua" border="0" height="33"
width="89"> that allows to link sources
for java.net projects from this blog a nice way that you can
see above.

Related Topics >>