J2SE Compatibility Test Sources Released

We've just released the complete J2SE compatibility test sources under a read-only license, so that the community can see what's really going on with Java compatibility testing. See:    http://jck.dev.java.net

Background

The core strength of the Java platform is compatibility. There can be all kinds of implementations from all kinds of vendors, but they all have to be compatible. This means developers can write portable Java applications that will reliably execute on any available run-time. And vendors can invest their energy in clever implementation techniques to build the best compatible implementations.

The Java team at Sun takes compatibility really seriously. Personally, I'm happy to be a deranged lunatic on this issue. Other people have their own particular obsessions about software licensing, but our obsession is around preserving compatibility.

This very strong focus on compatibility is partly driven by history. We've known since the very early days of Java that cross-platform compatibility is one of the most appreciated benefits of the Java platform. We're regularly told by developers that it really matters to them. So you would think it would "just happen" by market forces, right? Well, alas, no! Our experience has been that we have had to fight vigorously to preserve compatibility in the face of both accidental back-sliding and deliberate, hostile, forking.

But right now the compatibility test program is working. We have very strong acceptance from the Java developer community and from the Java vendor community about the importance of compatibility, and we have an extremely strong testing and branding program.

The core of the J2SE compatibility program is the Java technlology Compatibility Kit (the "JCK"). Right now it is roughly 2.5 million lines of source code and around 76,000 distinct test cases. Since the JCK is so important, we wanted to make it easier for developers to get their hands on it and see what it is really about. There aren't any tricks, or back-doors, or weird secrets here. Just zillions of details!

What we've released

We've released the complete J2SE technology Compatibility Kit (JCK) sources under a read-only license, including the sources for all the tests and the sources for the test harness. This is intended to let people read and evaluate the tests.

In order to keep the license simple, we have restricted it to be strictly "read only". I want to emphasize this. The license does not allow you to compile or run the tests (If you want to do that, you can get a different, more complicated, license, see below). We did this because otherwise the license would have blossomed into a twenty page legal document. But we have tried to make sure the license meets the reasonable needs of developers who want to evaluate the JCK sources:

• It's available at zero cost, through a click-through license.

• There is no "tainting". Once you delete your copy of the JCK, you aren't constrained in your future actions. To try to make this really clear, we included a section explicitly granting what the lawyers call "residual rights", which basically means that stuff that sticks in your head is OK to use in the future.

• You can publish feedback and comments publicly.

The Tests

If you study the JCK, you will find that test coverage varies by area. That reflects pragmatic choices about which areas need most testing. We tend to put more test energy into areas where (a) compatibility is most important and (b) where there is greater risk of incompatibility.

So for example, you will find there is extremely detailed test coverage on the language specification and the core JVM specification. These are areas where tight convergence is very important for developers and where there are competing product variants, so more vigilance is needed.

But there are other API areas where there seems comparatively little risk of accidental incompatibilities and these areas may get comparatively light JCK coverage. This is a continual balancing act and the JCK team adjusts depending on what is happening in the external world. We will start adding greater test coverage to an area if there starts to be evidence for more risk of actual incompatibilities.

One last word on the tests: I want to emphasize that these are tests aimed at interface compatibility, not at traditional product quality assurance testing. In addition to the 2.5 million lines of JCK tests that we use for compatibility testing, we have an additional 10.4 million lines of SQA test code that we use inside of Sun for product level testing of Sun's J2SE implementation. SQA and JCK testing have some overlap, but they have very different goals.

The Rules

Now, passing the JCK means more than just passing the individual tests. You also need to satisfy the Rules. The Rules are defined in the JCK User's Guide, which is part of the download bundle.

So what's going on here? Well, the tests can make sure that people are meeting the main positive requirements of the specifications, but it is impossible to write tests that can check for every possible violation of the specs. For example, we can write tests to make sure that a Java compiler supports every standard Java language keyword correctly, but it is is very hard to write a test that would detect a new keyword that some clever vendor has snuck in to their version of the language. (Not that anyone would ever do anything like that. No way. Inconceivable!)

So the Rules are used to define some higher level standards for compatibility which may not currently be captured in explicit tests. One example is that there is a Rule that requires that a product be compatible in "all configurations". You can't have a special configuration that you use to pass the tests, but then encourage your customers to use other configurations that are actually subtly incompatible. Yes, someone tried that trick once. Nice try, but this kind of thing is why we have the Rules as well as the Tests.

Getting a full JCK license

If you want to move beyond reading the tests to actually compiling or running tests, you will need to move to one of the fancier and more complicated licenses such as SCSL or the upcoming J2SE Java Distribution License (JDL).

These licenses are intended to support doing full-scale compatibility testing on your own J2SE implementation. These are serious commercial licenses and they cover all kinds of wonderful details about exactly what you can and cannot do with the tests, and what trademark rights and obligations you have, and support licensing agreements and much, much more. It's the kind of stuff that lawyers love. For commercial use, these licenses start at about $5OK, including some minimal support.

If you are serious about using the J2SE JCK to do compatibility testing then you probably do need to understand all that stuff. And you will probably want to buy extra support, as full compatibility testing is complicated and most serious users find they do need help.

But if all you want to do is look at the tests, these licenses are definitely total overkill, which is why we've created the new read-only license to allow simple evaluation access.

Enjoy!

We think the JCK is the greatest thing since sliced bread.

It isn't necessarily perfect, but it's been a great vehicle for driving real compatibility. We hope you'll enjoy reading it!

                                                                  - Graham