Alexey Popov's Blog: Test Harness and Test Format

My pages	Projects	Communities	java.net

Get Involved

java-net Project

Request a Project

Project Help Wanted Ads

Publicize your Project

Get Informed

java.net Online Books

java.net Archives

Get Connected

java.net Forums

Wiki and Javapedia

People, Partners, and Jobs

Java User Groups

RSS Feeds

Search

Online Books:

java.net on MarkMail:

Advanced Search

Alexey Popov's Blog

«Debugging II - Hangups at Device Side | Main | Testing session at Java ME track at SUN TechDays, Saint Petersburg »

Test Harness and Test Format

Posted by alexeyp on March 26, 2007 at 10:58 AM | Comments (4)

It is true that the most popular test format supported by many Java ^TM IDEs is JUnit and its variations like TestNG or J2MEUnit. Interested to compare JavaTest ^TM harness and JUnit ?

JavaTest is a test harness, JUnit is a test format. JavaTest is created to manage test suites written in many different formats, JUnit is a specific format and API for test development. The comparison is invalid, it would be more appropriate to compare JUnit API with one of test APIs, included into JavaTest's core libraries or its extensions.

Test format that is traditionally associated with this harness is the one that is used in most of Java Technology Compatibility Kits (TCK), based on test meta data provided in html form and Test/MultiTest API. Check the previous article for example of how JavaTest's Test Description looks like in .html. Comparing to JUnit, the java code of the TCK test may look like this:

public void testadd() throws AssertionFailedException {
    System.out.println("add");
    math.Arithmetic instance = Arithmetic.getInstance();
    for (int a=0;a<=10;a++)
    for (int b=0;b<=10;b++) {
        int expectedResult = a+b;
        int result = instance.add(a,b);
        assertEquals(expectedResult, result);
    }
}

public Status testadd() {
    log.println("add");
    math.Arithmetic instance = Arithmetic.getInstance();
    for (int a=0;a<=10;a++)
    for (int b=0;b<=10;b++) {
        int expectedResult = a+b;
        int result = instance.add(a,b);
        if (expectedResult != result)
            return Status.failed("Expected: "+expectedResult+
" Received: "+ result);
    }
    return Status.passed("OK");
}

The choice of the test API with explicit return for both positive and negative test conditions was driven by its use in TCKs, test suites, that are used for certification. One reason is requirement to be able to provide maximum to evaluate test failures offline without access to the implementation, where test failure is reproduced. Because of this there 'log' API is used instead of 'System.out', there is a special class Status that holds additional information other then pass/fail indication. Another reason is that TCK tests verify specific assertions of the specification and action of qualifying test as Passed need to be concise.

The JTReg is a JavaTest with set of plugins tuned for JDK regression testing. As opposite to the primary JavaTest use as a test harness for certification test suites, like TCK, where main requirements come from the test suite end user, JTReg is targeted to JDK developers. Main goals here are easiness to write and execute a test, support for major test types, possibility to execute shell scripts, adoption to the JDK development process. As you can see, there API for JDK regression tests is even simpler. See also JTReg FAQ for comments about JUnit support in JavaTest and JTReg.

JDTS Framework is a set of JavaTest harness plugins, that provide support for their own format, that is specially focused at functional and quality testing for Java ME.

Overall, different needs and different uses result in different test formats, that can be supported with single test harness.

Bookmark blog post:

del.icio.us

Digg

DZone

Furl

Comments

Comments are listed in date ascending order (oldest first) | Post Comment

I think it'd be just fair to mention about Tonga, which effectively run JavaTest, JTReg, and JUnit. Plus it has a lot of other interesting things like distributed tests execution and reproduction, etc.

Some more on Tonga can be found here http://weblogs.java.net/blog/cos/archive/2006/05/java_is_getting.html

Cheers,
Cos

Posted by: cos on March 28, 2007 at 07:04 PM

Thanks for the comment ! Tonga is a good example of what big QA system may evolve into. What I like in it is that features most useful for QA, like possibility to store and compare program output, require
plugin development for Javatest, and are natural features of Tonga. Its another strength, as I understand, is an integration with other JDK QA tools and processes.

Posted by: alexeyp on March 29, 2007 at 01:36 AM

Alexey, your example is quite incorrect: in fact, assertionFailedException may also hold any additional information as well as Status, just write assertEquals("Expected: "+expectedResult+ " Received: "+ result, expectedResult, result);. So, I think Status is remnant of procedure-oriented programming times..

Posted by: romanmak on May 15, 2007 at 02:59 AM

Hi romanmak,
I don't think introduction of exceptions into the language replaces the traditional rules of structured/procedural programming such as returning a result at the end of a method.
Reporting status through an exception is, of course, technically possible, but arguably leads to less clean code. Use of exceptions should generally be restricted to a recovery from abnormal situations where the flow of control breaks.
In case of a unit test, both "pass" and "fail" branches can be considered "normal" control flow (unlike other code, tests are designed to detect and trigger problems in the code).
This, in my view, is a good justification for the choice of Status return type for JT Harness tests.

Posted by: m_gorshenev on May 15, 2007 at 02:10 PM

Powered by
Movable Type 3.01D

java.net RSS Feeds