Simplest Test Suite built on ME Framework
This article is about some changes in the
were made in response to the feedback that we got from early adopters
and are available in the third development release of ME Framework 1.2
It is targeted to people who use (or plan to
Framework or JT
harness or work in the area of Java ME testing.
Before the open source release, we were developing and using the ME
Framework internally for years, as long as Java ME exists. As it turned
out, to start using it outside of the internal SUN environment and
supporting infrastructure is not
a trivial task. Unlikely these problems are unique to our project.
Some things became too natural for
internal users. Some approaches
use internally when develop TCKs with ME Framework help to
optimize the work of the big production organization, responsible for
development and sustaining of a big number of products. It turns out
that people who need to do
a first step, like to
create a single test suite once, need more simple ways to do simple
Simplest Test Suite
that Alexander (aka Skavas) has recently checked into
repository, is what its name means - simplest possible test suite, that
we found reasonable for this product. It consists from three files -
ant build script, test suite .jtt parameter file and a sample test, If
you follow the instructions from
file, it will take 5 min
to build and launch it plus time to download.
I would like to go into details of two issues, that were
in the ME Framework 1.2 3rd Development release and are demonstrated in
- Allowed alternative test formats, not limit to html test
description, that have to be created for each test.
- Allowed not to provide explicit information on
what class and
resource files are
associated with test. This is necessary for automated packaging of test
(bundles) and works now for simple tests.
Simple way to describe tests
The ME Framework is a plugin to
(aka JavaTest TM harness), that enables Java ME
test development and execution. This specific change was to correct
usage of JT harness features by ME Framework, specifically, TestFinder.
in JT harness is
the class, responsible for providing array of href="http://fisheye4.cenqua.com/browse/jtharness/trunk/code/src/com/sun/javatest/TestDescription.java?r=49">TestDescription
elements, that are representing test meta data in JT harness' model of
the test suite. Support of different formats of tests is done in JT
harness by plugging in corresponding extension of the TestFinder.
In short, the first change in ME Framework was just enabling
in the .jtt file.
TagTestFinder uses javadoc
identify test and describe its meta information and it is a TestFinder,
used in JTReg. href="http://fisheye4.cenqua.com/browse/jtharness/trunk/code/src/com/sun/javatest/finder/HtmlTestFinder.java?r=49">HTMLTestFinder
is more familiar to TCK users,
it describes tests in specially formatted html table, check href="http://weblogs.java.net/blog/alexeyp/archive/2007/02/testing_command_1.html">here
for an example.
Simplest Test Suite uses tags to describe tests - it allows to
decrease number of files, naturally supported by IDEs and somewhat
familiar to people with JUnit experience. Check how the href="http://fisheye4.cenqua.com/browse/cqme/trunk/samples/SimplestTestSuite/tests/example/SimpleTests.java?r=664">sample
test looks like.
Other kinds of TestFinder
Other then Tag and Html finders, the standard set of
TestFinder implementations, provided in JT harness, includes
that reads information on tests to run from the
binary file. It is the one that is most often used in TCKs, where there
is a need to optimize
time to read huge test tree. BinaryTestFinder is most applicable to
cases, where test suite does not change after its development is done,
because any change in any test description requires regeneration of the
binary test data.. BinaryTestFinder is used in conjunction with
another TestFinder that
reads test descriptions from tags or html files during the build of the
test suite and stores them in the
It's just sort of "extension" on top of other Finders,
top of them, and gets the Descriptions from other Finders and serialize
them to the binary data, and also allows to read the descriptions from
the serialized data. It can be used with any other TestFinders that
do the actual job of parsing the TestDescriptions from HTML, javadocs,
How to find what content to put into the .jar of the test
This topic is more complex and hence more entertaining one.
The problem in more details. ME Framework allows for a great
flexibility in how to turn huge pile of class and resource files of the
test suite into set of MIDP application packages for iterative
execution through href="http://weblogs.java.net/blog/alexeyp/archive/2006/11/testing_java_me_1.html#Autotest">autotest.
Criteria that control
creation of packages can be
- max size
of the jar file
- number of tests to package into the single jar
themselves may describe some conditions in their associated meta data
would regulate how these tests need to be packaged
Tests are packaged
into .jar at runtime to allow the user to vary packaging parameters to
achieve different goals, primarily to optimize network traffic and test
This all means - to form various application packages ME
Framework needs to know which class and resource files are associated
with each test.
These data can be calculated automatically using several
methods, I will talk about them later. What we did for this sample:
TestDescription in JT harness requires definition of the
'executeClass' parameter for every test. The value of this field is the
name of the class that is test
entry point, implementing one of supported interfaces. If
consists from the single .java source
file that defines single class - nothing to
calculate, right ? All class files of this test can be
described by /$executeClass.*/ expression. Note that inner classes are
OK here as well.
Other then this, nothing else happens in this sample
automatically - if there are multiple .java files, that constitute a
test, one need to
list them manually in the
'resources' field. Not that convenient as 'do nothing', but
simple to explain :) Other then 'resources' field, the standard
special file named testClasses.lst
still works, just generation of this file is not supported in the build
of the Simplest Test Suite.
The file testClasses.lst should be located in the special place in test
suite's directory hierarchy and containing testURL-test classes
pairs, check href="https://cqme.dev.java.net/source/browse/cqme/trunk/samples/meframework/SampleTestSuite/build/testclasses.lst?rev=333&view=markup">how
it looks like in another sample.
Approaches to automatically calculate list of classes to
Content of the testClasses.lst is calculated statically during the
the test suite.
Approach 1 - individual test compilation.
One way to
do this is to put the load on the compiler. Compile tests individually,
set own target dir for each test, not give and test library classes on
the classpath but provide reference to their source directory instead.
All class files, that are needed by the test, will be in the target
directory of the compiler. The described approach is simple and
reliable one, we used it
some TCK releases. Weak places of it is that it is slow, at least in
our implementation where java compiler and preverifier were executed
through the command line for each test.
Approach 2 - parsing class files for dependencies.
We do this with a simple tool based on JINI's
It does static class analysis and should calculate all dependencies
correctly except for cases like
is used in the tests
(which is rarely
There are probably other tools which can do the same thing.
Here is the example of the command, that can be used in the test suite
-cp <ALL TEST AND FW
CLASSES> <executeClass> ^
-out com.sun.tck.cldc -out
com.sun.tck.j2me -out java -out sun -out
In cases where all necessary classes can not be found through parsing
class files from the entry point, we used static, manually created
table, that is merged into the generated testClasses.lst. This can be
just a table with the same format, or, as mentioned above, just a
'resources' field of the TestDescription.
Build script for the sample will work only with simple
tests for MIDP. Support of distributed, interactive, OTA or CDC tests
will require more sophisticated build procedure, that will take care of
compiling SE-side components, for example.
Yet more simplification
Further simplifying of this test suite and test development
is possible by making it close to JTReg and xUnit. This can be:
- Minimizing set of mandatory meta data. For example, if a
consists from the single .java file, test entry point, that is now
specified in the mandatory 'executeClass' parameter. is known.
- If this suite is used in the scenario, where tests are
and constantly changing, it may be possible to eliminate 'build' stage,
as it is done in JTReg, implement test source
compilation/preverification as a step in the test execution
I would like to express my appreciation to the
alt="Project Supported by FishEye" border="0" height="33"
width="89"> hosting provided by href="http://www.cenqua.com/"> src="http://www.cenqua.com/images/cenquad.gif"
alt="Supported by Cenqua" border="0" height="33"
width="89"> that allows to link sources
for java.net projects from this blog a nice way that you can