Graham Hamilton's Blog

J2SE Compatibility Test Sources Released

We've just released the complete J2SE compatibility test sources under a read-only license, so that the community can see what's really going on with Java compatibility testing. See:    http://jck.dev.java.net

Background

The core strength of the Java platform is compatibility. There can be all kinds of implementations from all kinds of vendors, but they all have to be compatible. This means developers can write portable Java applications that will reliably execute on any available run-time. And vendors can invest their energy in clever implementation techniques to build the best compatible implementations.

The Java team at Sun takes compatibility really seriously. Personally, I'm happy to be a deranged lunatic on this issue. Other people have their own particular obsessions about software licensing, but our obsession is around preserving compatibility.

This very strong focus on compatibility is partly driven by history. We've known since the very early days of Java that cross-platform compatibility is one of the most appreciated benefits of the Java platform. We're regularly told by developers that it really matters to them. So you would think it would "just happen" by market forces, right? Well, alas, no! Our experience has been that we have had to fight vigorously to preserve compatibility in the face of both accidental back-sliding and deliberate, hostile, forking.

But right now the compatibility test program is working. We have very strong acceptance from the Java developer community and from the Java vendor community about the importance of compatibility, and we have an extremely strong testing and branding program.

The core of the J2SE compatibility program is the Java technlology Compatibility Kit (the "JCK"). Right now it is roughly 2.5 million lines of source code and around 76,000 distinct test cases. Since the JCK is so important, we wanted to make it easier for developers to get their hands on it and see what it is really about. There aren't any tricks, or back-doors, or weird secrets here. Just zillions of details!

What we've released

We've released the complete J2SE technology Compatibility Kit (JCK) sources under a read-only license, including the sources for all the tests and the sources for the test harness. This is intended to let people read and evaluate the tests.

In order to keep the license simple, we have restricted it to be strictly "read only". I want to emphasize this. The license does not allow you to compile or run the tests (If you want to do that, you can get a different, more complicated, license, see below). We did this because otherwise the license would have blossomed into a twenty page legal document. But we have tried to make sure the license meets the reasonable needs of developers who want to evaluate the JCK sources:

• It's available at zero cost, through a click-through license.

• There is no "tainting". Once you delete your copy of the JCK, you aren't constrained in your future actions. To try to make this really clear, we included a section explicitly granting what the lawyers call "residual rights", which basically means that stuff that sticks in your head is OK to use in the future.

• You can publish feedback and comments publicly.

The Tests

If you study the JCK, you will find that test coverage varies by area. That reflects pragmatic choices about which areas need most testing. We tend to put more test energy into areas where (a) compatibility is most important and (b) where there is greater risk of incompatibility.

So for example, you will find there is extremely detailed test coverage on the language specification and the core JVM specification. These are areas where tight convergence is very important for developers and where there are competing product variants, so more vigilance is needed.

But there are other API areas where there seems comparatively little risk of accidental incompatibilities and these areas may get comparatively light JCK coverage. This is a continual balancing act and the JCK team adjusts depending on what is happening in the external world. We will start adding greater test coverage to an area if there starts to be evidence for more risk of actual incompatibilities.

One last word on the tests: I want to emphasize that these are tests aimed at interface compatibility, not at traditional product quality assurance testing. In addition to the 2.5 million lines of JCK tests that we use for compatibility testing, we have an additional 10.4 million lines of SQA test code that we use inside of Sun for product level testing of Sun's J2SE implementation. SQA and JCK testing have some overlap, but they have very different goals.

The Rules

Now, passing the JCK means more than just passing the individual tests. You also need to satisfy the Rules. The Rules are defined in the JCK User's Guide, which is part of the download bundle.

So what's going on here? Well, the tests can make sure that people are meeting the main positive requirements of the specifications, but it is impossible to write tests that can check for every possible violation of the specs. For example, we can write tests to make sure that a Java compiler supports every standard Java language keyword correctly, but it is is very hard to write a test that would detect a new keyword that some clever vendor has snuck in to their version of the language. (Not that anyone would ever do anything like that. No way. Inconceivable!)

So the Rules are used to define some higher level standards for compatibility which may not currently be captured in explicit tests. One example is that there is a Rule that requires that a product be compatible in "all configurations". You can't have a special configuration that you use to pass the tests, but then encourage your customers to use other configurations that are actually subtly incompatible. Yes, someone tried that trick once. Nice try, but this kind of thing is why we have the Rules as well as the Tests.

Getting a full JCK license

If you want to move beyond reading the tests to actually compiling or running tests, you will need to move to one of the fancier and more complicated licenses such as SCSL or the upcoming J2SE Java Distribution License (JDL).

These licenses are intended to support doing full-scale compatibility testing on your own J2SE implementation. These are serious commercial licenses and they cover all kinds of wonderful details about exactly what you can and cannot do with the tests, and what trademark rights and obligations you have, and support licensing agreements and much, much more. It's the kind of stuff that lawyers love. For commercial use, these licenses start at about $5OK, including some minimal support.

If you are serious about using the J2SE JCK to do compatibility testing then you probably do need to understand all that stuff. And you will probably want to buy extra support, as full compatibility testing is complicated and most serious users find they do need help.

But if all you want to do is look at the tests, these licenses are definitely total overkill, which is why we've created the new read-only license to allow simple evaluation access.

Enjoy!

We think the JCK is the greatest thing since sliced bread.

It isn't necessarily perfect, but it's been a great vehicle for driving real compatibility. We hope you'll enjoy reading it!

                                                                  - Graham

Comments

• Note that the link to the SCSL licensing page is actually: http://wwws.sun.com/software/communitysource/j2se/java2/download.html.

  Posted by: johnm on December 13, 2004 at 09:33 PM

• Graham,
  Thanks to you and all the folks involved in this move, especially for you clear explanation of the read-only license (this removes a lot of the guessing, especially concerning the "tainting" aspect).

  I was wondering about the compile-and-run versions of the JCK which are under SCSL (in the future JDL?): you say that commercial use means paying 50K... does the "commercial" attribute mean, that non-commercial projects could use the JCK (use as in compile-and-run) without that price tag? I

  Posted by: murphee on December 14, 2004 at 04:29 AM

• ... damn, why was that post cut off... anyway here the rest:
  I

  Posted by: murphee on December 14, 2004 at 04:31 AM

• OK... the comment service doesn

  Posted by: murphee on December 14, 2004 at 04:32 AM

• OK...I give up...

  Posted by: murphee on December 14, 2004 at 04:34 AM

• to murphee - The trick is not to use quotes or double quotes. Using things like -I am- instead of the compressed form. It seems to be a feature of the parser used. If you use the preview feature you will see if you have sliped off...

  - eduard/o

  Posted by: pelegri on December 14, 2004 at 07:24 AM

• @pelegri:
  Thanks for the tip (my problem was that I did look at the preview, which looked as I had intended it to, but the end result was messed up...odd);

  Posted by: murphee on December 14, 2004 at 11:46 AM

• So, the only way to get the TCK and certify my JVM and class libraries is to stick to a complicated commercial license... this means no way Kaffe, GCJ or any other nice free software / open source software JVM can be certified. :-(

  I hoped Sun would be willing to do best, but this was very disapointing.

  Posted by: flozano on December 14, 2004 at 02:23 PM

• 
  One point I probably ought to have mentioned is that there
  are TCK scholarships available for legitimate not-for-profit
  groups trying to pass the JCK. So, yes, people wanting to run
  the JCK do have to sign complicated licenses (sorry, but this
  is complex stuff!), but it is free for legitimate not-for-profits.
  And typically we also provide basic support.

  For more details on the TCK scholarship program see:

  http://java.sun.com/scholarship/

  Posted by: kgh on December 14, 2004 at 02:38 PM

• I find it kind of sad that a (good) post about about releasing some testing code is 50% about licensing. Does Sun have the highest Lawyer per Employee cost in the word?

  Also kind of funny that you actually allow me to use the information that is -In my head- . Hilarious if you think about it..

  Cheers,
  Mikael Grev

  Posted by: mgrev on December 14, 2004 at 03:04 PM

• Graham,

  Thanks for letting people see the JCK. It is a good start. Now, let's write some tests examining various use-cases.

  Use the JCK to...

  clarify specs to the developer. -- Pass
  clarify specs to Sun by writing additional tests. -- Fail
  test virtual machines and libraries for compliance. -- Fail
  find gaps in the JCK using coverage or mutation testing tools. -- Fail

  
  Also, could you clarify something for me? Of course I promise not to approach the JCK with the Java Language Spec, the Java VM Spec, and a pad of graph paper unless I have explicit permission. Unfortunately, learning to program without a debugger has given me a tendency to "play the computer" subconsciously. Some are far worse off.

  - Curt

  Posted by: coxcu on December 14, 2004 at 04:19 PM

• Mikael: Yes, licensing can definitely be tiresome. But people
  often seem to have licensing questions, so I wanted to try and
  explain the key terms up front.

  By the way, on the "using information in your head" issue, the
  main reason we're saying this is that in the past some people
  have expressed concern about somehow becoming tainted and not
  being able to keep residual knowledge. The reason for putting
  this in the license was to try to reassure people that this isn't
  an issue here!

  regards - Graham
  

  Posted by: kgh on December 14, 2004 at 05:23 PM

• Well, congratulations on making this step, Graham.

  
  Being a passionate free runtime developer, I'm very interested in seeing Kaffe/gcj/GNU Classpath/GNU JAXP/JESSIE/GNU inetlib/Tritonus/GNU regex/JacORB/gcjx/gjdoc etc. evolve into a fully compatible set of implementations of the J2SE specs, so I'm naturally interested in details about the (up-till-now) mythical TCK, and the whole non-profit TCK scholarship program thing.

  So far, we're using Mauve, JACKS and Japitools as free software substitutes for the real thing with good success. Recently we've put Kaffe using GNU Classpath into the Apache Gump, to make it build and run software that cordially matters to many people: free and open source software.

  There is some nice progress on that front, as GNU Classpath quickly moves forward and missing methods and classes get implemented by some really great hackers. We went from 10% to 30% success in the last two weeks, which is quite encouraging. It's still a good shot (15%) away from JDK 1.5, but we'll get there, too in a few weeks ;)

  The great thing about that is, that I can talk about Kaffe and the sucesses and failures on the Gump freely to you. I can look at the failures in the Gump, talk to my peers from various open source projects, and fix the bugs in Kaffe, GNU Classpath or those projects, respectively. That's what a real open community of peers is like. All of that without the stone-walling that's apparently necessary for the TCK.

  So, that's the part I really don't get: what's so confidential about a test suite? I mean, what sort of an earth-shattering trade secret could there be hiding in the TCK's tests for, say, java.lang.Object.toString()? The veil of secrecy in this particular license is not as thick as I had imagined it to be, as one can provide feedback about the JCK, but it still would create a very frustrating fence between people who accept a read-only license, and those who don't, and prohibit us to efficiently cooperate.

  So, why does the license prohibit compiling and running tests? I mean, seriously, that sounds like the first thing I'd try to do, being seriously interested in compatibilty. Let me explain my reasoning a bit:

  Compiling the test suite would be a nice way to find and shake out bugs and compatibility problems in Tom's gcjx, the next-generation of gcj. When (and that's no longer an if, as I'm pretty much ready to go for it) I apply for a TCK scholarship for J2SE 1.5, gcjx is going to be a part of the free software stack that I'd like to see become a compatible J2SE 1.5 implementation. So making gcjx robust enough to deal with the actual TCK tests would be a cool thing to do before we find out that we have to spend a few months fixing the compiler first.

  What does prohibiting that buy Sun? How much support cost would Sun have to bear for me running a

  find /path-to-tck/ -name "*.java" | xargs -n 1 gcjx

  script on my own machine and talking to Tom Tromey and other gcj devs about the resulting failures and bugs in gcjx without bothering you or Neil?

  We're not even talking about the fearsome TCK challenges here, that I talked about yesterday with Enrico, just about using the 2.5 million lines of 1.5-Java-certified source code in the TCK to find bugs in free software that would very, very much like to be compatible with Sun's reference implementation (and would be the cornerstone of a scholarship application for J2SE, just like GNU Classpath).

  In my opinion, making it easy for free software implementations to fix their bugs before they go the way of a scholarship would help shorten the certification period, potential costly TCK challenges, and all that. I really, really, want a scholarship application for the free runtime stack I described above to work out, and that means to me, that it shouldn't be a frustratingly long, legally cumbersome process.

  It's been taking both JOnAS and Geronimo a good deal of time to get their code to work with the TCK and to fix the bugs they found, because the task for J2EE is pretty huge. I guess it's a similarly massive task ahead of us for J2SE 1.5 as well. In order to not waste Sun's money and your (and Calvin's) time during the scholarship process, it would be good to be able to start fixing our bugs and moving towards compliance right now.

  All you have to do for that is to fix this 'look, but don't touch' license to be more permissive (and I applaud the changes you already mentioned, and the move to resove potential problems like taintedness) and this JCK release would be way more useful to everyone, including Sun.

  cheers,
  dalibor topic

  Posted by: robilad on December 14, 2004 at 05:57 PM

• Graham: It could also be that a very small group, with interests of their own, are very loud in the community. (Not at all like the -Fix Swing- group, which is very nice and just wants the best for every one...)

  Cheers,
  Mikael Grev

  Posted by: mgrev on December 14, 2004 at 08:47 PM

• Dalibor,
  running the JCK for external companies is usual big business, so if you fail the test and have to do it 3 times, that means a higher income for Sun.
  The opening of the test suite only has as effect that bugs in the test suite might be found by others (Sun gains more eyeballs) and that you as a person trying to follow the test do not have any excuses when you fail, and really have to pay a second time to run it a second time.

  All in all; 100% gain for Sun :)

  Posted by: zander on December 15, 2004 at 02:27 AM

• Zander,

  Are you sure? I bet that Sun just charges for the JCK to cover costs--not to generate revenue.

  On the other hand, Graham has a compatibility lunacy and Jonathan Schwartz has a desire to grow the number of Java developers to 10 million. Both goals would be better served by a JCK that could be run at no cost. The process of finding bugs in the JCK will be hampered by not being able to run it.

  The two best techniques for finding holes in test suites (code coverage and mutation testing) both require running the tests. Until the JCK can be run at no cost, Graham's claim to be a deranged lunatic is open to debate.

  - Curt
  

  Posted by: coxcu on December 15, 2004 at 11:55 AM

• My take on this is that this is a Bait and Switch.

  Posted by: johnm on December 15, 2004 at 01:02 PM

• John,

  I understand you do not like the license. I think many people may well find it useful, but I understand we may disagree on that.

  However, I am rather surprised at the implication that we are being deceptive. I deliberately spent a large chunk of my blog text on explaining the licensing issues, because I wanted to AVOID people being confused about what the license meant.

  regards - Graham

  
  

  Posted by: kgh on December 15, 2004 at 01:13 PM

• Graham, thanks for responding. As I noted in my blog (and the comments therewith), the very point of source code is to be used. The fact that you've been very clear about the read-only license is irrelevant to the fact that the only way to be able to use the JCK is to submit to onerous licensing terms. That fact that you're clear about that up front rather than hiding that fact does mean that you're not acting e.g., fraudulently. However, from a moral standpoint, I'll stand firm.
  

  Posted by: johnm on December 15, 2004 at 01:22 PM

• The jck is there for jdk compatibilty. But does anyone know of any project/product (free or comercial) out there that has test suites for performance metrics? I'd like to be able to have a test suite to run to get metics for different VM's (for different versions of the jdk) rather than believing the vendor's "Our version is faster" line.

  Posted by: mandersen94 on January 09, 2005 at 03:41 PM

• Can anyone answer a simple question... is JCK open source, or is it not?

  Posted by: vpatryshev on February 26, 2005 at 08:57 PM

• vpatryshev: it's not open source. It's quite far from it.

  cheers,
  dalibor topic

  Posted by: robilad on May 05, 2005 at 12:14 AM

This work is licensed under a Creative Commons License.