Alexey Popov's Blog: March 2007 Archive

Posted by alexeyp on March 31, 2007 at 11:37 PM | Permalink | Comments (1)

If you speak Russian, check the agenda and descriptions of Java ME sessions. 'Java ME Testing' session description is in the end of the list, it concludes the Java ME track and Day 2.

Slides and description for it were first created in the native language of Internet and IT, that we use at work and call English by inertia. I did a first pass of translation to Russian and found that for many notions I do not have adequate words. For example, does open source mean the same as открытый код ? From the other side - with mother tongue you can make your speech so alive !

Check blogs of my colleagues who are speakers at the Java ME track: Danila (CLDC), Petr (CDC), Alexander (Testing). Presentations in English, like one of Simon Ritter about ME Gaming, will be translated online.

I will also be there at the Java ME POD. See you at SUN TechDays in Saint Petersburg !

Test Harness and Test Format

Posted by alexeyp on March 26, 2007 at 10:58 AM | Permalink | Comments (4)

It is true that the most popular test format supported by many Java ^TM IDEs is JUnit and its variations like TestNG or J2MEUnit. Interested to compare JavaTest ^TM harness and JUnit ?

JavaTest is a test harness, JUnit is a test format. JavaTest is created to manage test suites written in many different formats, JUnit is a specific format and API for test development. The comparison is invalid, it would be more appropriate to compare JUnit API with one of test APIs, included into JavaTest's core libraries or its extensions.

Test format that is traditionally associated with this harness is the one that is used in most of Java Technology Compatibility Kits (TCK), based on test meta data provided in html form and Test/MultiTest API. Check the previous article for example of how JavaTest's Test Description looks like in .html. Comparing to JUnit, the java code of the TCK test may look like this:

public void testadd() throws AssertionFailedException {
    System.out.println("add");
    math.Arithmetic instance = Arithmetic.getInstance();
    for (int a=0;a<=10;a++)
    for (int b=0;b<=10;b++) {
        int expectedResult = a+b;
        int result = instance.add(a,b);
        assertEquals(expectedResult, result);
    }
}

public Status testadd() {
    log.println("add");
    math.Arithmetic instance = Arithmetic.getInstance();
    for (int a=0;a<=10;a++)
    for (int b=0;b<=10;b++) {
        int expectedResult = a+b;
        int result = instance.add(a,b);
        if (expectedResult != result)
            return Status.failed("Expected: "+expectedResult+
" Received: "+ result);
    }
    return Status.passed("OK");
}

The choice of the test API with explicit return for both positive and negative test conditions was driven by its use in TCKs, test suites, that are used for certification. One reason is requirement to be able to provide maximum to evaluate test failures offline without access to the implementation, where test failure is reproduced. Because of this there 'log' API is used instead of 'System.out', there is a special class Status that holds additional information other then pass/fail indication. Another reason is that TCK tests verify specific assertions of the specification and action of qualifying test as Passed need to be concise.

The JTReg is a JavaTest with set of plugins tuned for JDK regression testing. As opposite to the primary JavaTest use as a test harness for certification test suites, like TCK, where main requirements come from the test suite end user, JTReg is targeted to JDK developers. Main goals here are easiness to write and execute a test, support for major test types, possibility to execute shell scripts, adoption to the JDK development process. As you can see, there API for JDK regression tests is even simpler. See also JTReg FAQ for comments about JUnit support in JavaTest and JTReg.

JDTS Framework is a set of JavaTest harness plugins, that provide support for their own format, that is specially focused at functional and quality testing for Java ME.

Overall, different needs and different uses result in different test formats, that can be supported with single test harness.

Debugging II - Hangups at Device Side

Posted by alexeyp on March 01, 2007 at 11:06 AM | Permalink | Comments (0)

As a follow up to the past article about Debugging with ME Framework, here is the the guest post from Alexander Alexeev (aka Skavas) on the new feature he has integrated into the ME Framework, the Interactive MIDlet agent. The feature addresses some usability issues of executing large test suites on mobile devices, provides on-screen indication of the testing progress and allows to perform some operations with test results on the device.

Background

The Introduction article described the approaches, used for execution of large test suites for Java ^TM ME implementations as well as some of techniques, allowing to optimize test execution time. To restate the main points, relevant to this debugging topic:

test execution is managed at the server side
the test execution process consists of sequential downloads of test MIDlet suites to the device
during this process AMS and MIDlet suites exchange control messages with test harness
one of optimizations, allowing to minimize network traffic and number of downloads/installations/runs/removals, is to package multiple tests into a single bundle.

The diagram describing this autotest cycle can be seen here.

The 'autotest' approach allows to achieve the sufficient level of automation, that is one of high priority requirements for test suites we develop with JT harness and ME Framework.

User interactivity here still may be desired in few situations:

'sendTestResult' may not be executed for some reason. For example, because of the VM exit or test/VM hang up. Since this operation is executed at the Test level, and Test may consist of multiple test cases, all information from actually executed test cases may be lost. It will be unknown which test case caused the problem.
when the device is slow and optimization is used (multiple tests in bundle), the whole test cycle will be shorter, but every individual test bundle will take time to execute. Without visual indication it may be hard to distinguish if tests are being executed that slow, or it is just device stopped responding an hour ago.
if test hung, it should be possible to cancel it with minimal impact on the execution process, without stop/restart the whole test run.

One solution for these debugging problems is using the Test Export feature to run tests isolated from the test harness environment. Test results from standalone execution are not sent to the JT harness, they are displayed on the device console, if there is such available.

Interactivity for Automated Tests

The proposed solution for all above problems is to introduce interactivity at the device side. We added an option to use Interactive MIDlet Agent for test execution, that provides the following features:

shows the current status on the device display
continuously saves the current status of the RMS
provides a control for saving the current status of the RMS
allows viewing results stored in the RMS
provides a control for canceling the current test/test bundle and starting the next test/test bundle

These interactive features are also available in the Test Export mode.

User Interface

Additional set of standard configuration questions were added to the ME Framework Configuration Editor. To make ME Framework to use this on screen tracing functionality, see the screenshot .

When the JT harness executes tests with these configuration settings, the user interface on the device side displays the following information and commands:

Information string with a common count of tests done, a count of failed tests, and a count of passed tests
Information about the current test running
Commands to cancel the test/test bundle and to save the results to the RMS

As you can see here , this new interface resembles the interface of device-side harnesses, that are available for execution of mobile variations of JUnit.

To view results stored in the RMS, use the RMSReader MIDlet application. It has a simple interface that uses a Command to change the display either to "view log" or to "view ref" output.

Open Issues

There should be better solutions then one that we chosen for canceling a hanging test.

The 'cancel test' functionality requires each test to be run in a separate thread. Since Connected Limited Device Configuration has no method to interrupt threads in order to break test execution and proceed to the next test, a special flag is used to mark the test thread as canceled. The canceled thread is set to minimum priority and the agent starts to execute the next test.

Powered by
Movable Type 3.01D

java.net RSS Feeds