Skip to main content

J2SE Compatibility Test Sources Released

Posted by kgh on December 13, 2004 at 9:01 PM PST

We've just released the complete J2SE compatibility test
sources under a read-only license, so that the community can
see what's really going on with Java compatibility testing.


The core strength of the Java platform is compatibility.
There can be all kinds of implementations from all kinds
of vendors, but they all have to be compatible. This
means developers can write portable Java applications
that will reliably execute on any available run-time.
And vendors can invest their energy in clever implementation
techniques to build the best compatible implementations.

The Java team at Sun takes compatibility really seriously.
Personally, I'm happy to be a deranged lunatic on this issue. Other
people have their own particular obsessions about software
licensing, but our obsession is around preserving compatibility.

This very strong focus on compatibility is partly driven
by history. We've known since the very early days of Java
that cross-platform compatibility is one of the most
appreciated benefits of the Java platform. We're
regularly told by developers that it really matters to them.
So you would think it would "just happen" by market forces,
right? Well, alas, no!
Our experience has been that we have had to
fight vigorously to preserve compatibility in the face of
both accidental back-sliding and deliberate, hostile, forking.

But right now the compatibility test program is working.
We have very strong
acceptance from the Java developer community and from
the Java vendor community about the importance of
compatibility, and we have an extremely strong testing
and branding program.

The core of the J2SE compatibility program is the Java
technlology Compatibility Kit (the "JCK"). Right now it is
roughly 2.5 million lines
of source code and around 76,000 distinct test cases.
Since the JCK is so important, we wanted to make it easier for
developers to get their hands on it and see what it is really
about. There aren't any tricks, or back-doors, or weird
secrets here. Just zillions of details!

What we've released

We've released the complete J2SE technology Compatibility Kit (JCK)
sources under a read-only license, including the sources for all
the tests and the sources for the test harness. This is intended
to let people read and evaluate the tests.

In order to keep the license simple, we have restricted
it to be strictly "read only". I want to
emphasize this. The license does not allow you to
compile or run the tests (If you want to do that, you can get
a different, more complicated, license, see below).
We did this because otherwise the license would have blossomed
into a twenty page legal document. But we have tried to make
sure the license meets the reasonable needs of developers who want to
evaluate the JCK sources:

  • It's available at zero cost, through a click-through license.

  • There is no "tainting". Once you delete your copy of the JCK,
    you aren't constrained in your future actions. To try to make
    this really clear, we included a section explicitly granting
    what the lawyers call "residual rights", which basically means
    that stuff that sticks in your head is OK to use in the future.

  • You can publish feedback and comments publicly.

The Tests

If you study the JCK, you will find that test coverage
varies by area. That reflects pragmatic choices about
which areas need most testing. We tend to put more test
energy into areas where (a) compatibility is most important
and (b) where there is greater risk of incompatibility.

So for example, you will find there is extremely detailed test
coverage on the language specification and the core
JVM specification. These are areas where tight convergence
is very important for developers and where there are competing
product variants, so more vigilance is needed.

But there are other API areas where there seems comparatively
little risk of accidental incompatibilities and these areas
may get comparatively light JCK coverage. This is a continual
balancing act and the JCK team adjusts depending on what
is happening in the external world. We will start
adding greater test coverage to an area if there starts to
be evidence for more risk of actual incompatibilities.

One last word on the tests: I want to emphasize that these
are tests aimed at interface compatibility, not at traditional
product quality assurance testing. In addition to the 2.5
million lines of JCK tests that we use for compatibility
testing, we have an additional 10.4 million lines of SQA test
code that we use inside of Sun for product level testing of
Sun's J2SE implementation. SQA and JCK testing have some
overlap, but they have very different goals.

The Rules

Now, passing the JCK means more than just passing the individual
tests. You also need to satisfy the Rules. The Rules
are defined in the JCK User's Guide, which is part of the
download bundle.

So what's going on here? Well, the tests can make sure that
people are meeting the main positive requirements of the
specifications, but it is impossible to write tests that can
check for every possible violation of the specs.
For example, we can write tests to make sure that
a Java compiler supports every standard Java language keyword
correctly, but it is is very hard to write a test that would
detect a new keyword that some clever vendor has snuck in to their
version of the language. (Not that anyone would ever do anything
like that. No way. Inconceivable!)

So the Rules are used to define some higher level standards
for compatibility which may not currently be captured in
explicit tests. One example is that there is a Rule that
requires that a product be compatible in "all configurations".
You can't have a special configuration that you use to pass
the tests, but then encourage your customers to use other
configurations that are actually subtly incompatible. Yes,
someone tried that trick once. Nice try, but this kind of
thing is why we have the Rules as well as the Tests.

Getting a full JCK license

If you want to move beyond reading the tests to actually
compiling or running tests, you will need to move to one of
the fancier and more complicated licenses such as
or the upcoming J2SE Java Distribution License (JDL).

These licenses are intended to support doing full-scale
compatibility testing on your own J2SE implementation.
These are serious commercial licenses and they cover
all kinds of wonderful details about
exactly what you can and cannot do with the tests, and
what trademark rights and obligations you have, and support
licensing agreements and much, much more. It's the kind
of stuff that lawyers love. For commercial use, these
licenses start at about $5OK, including some minimal support.

If you are serious about using the J2SE JCK to do compatibility
testing then you probably do need to understand all that stuff.
And you will probably want to buy extra support, as full
compatibility testing is complicated and most serious users
find they do need help.

But if all you want to do is look at the tests, these
licenses are definitely total overkill, which is why we've created
the new read-only license to allow simple evaluation access.


We think the JCK is the greatest thing since sliced bread.

It isn't necessarily perfect, but it's been a great vehicle for
driving real compatibility. We hope you'll enjoy reading it!

- Graham

Related Topics >>