Skip to main content

Tough and Trustworthy Tigers

Posted by mreinhold on September 1, 2004 at 3:26 PM PDT

The Tiger Release Candidate shipped
earlier today
.

Even more amazingly, our QA team is happy with it!

Our hard-working QA team recently presented a summary of their results
based on the near-final builds of Tiger. Overall this is looking to be the
most stable and reliable JDK that we've ever shipped. Here are the highlights
of their report:

  • Applet
    compatibility
         We test a set of over 400 applets,
    nearly all of which are external, to make sure that they run as well on Tiger
    as they do on any other popular VM, most especially the old and somewhat
    quirky VM from Microsoft. 97% of these tests pass, which is a much higher
    fraction than for any previous JDK release. The few failures are mostly due
    to applets that are relying on behavior that's outside the scope of the J2SE
    specification.
  • Reliability     We
    run a set of five large server-class applications, including Sun's own
    application server, another well-known application server, and Tomcat, on
    some big iron under heavy load to see how long they stay up. As of this
    writing they've been up and running continuously for 28 days -- at some point
    we'll have to decide when to shut them off. This is a much longer uptime
    than we've achieved in previous releases.
  • Conformance    
    The 1.5 JCK (Java Compatibility Kit) contains a whopping 45,194 tests (for
    comparison, the 1.4.2 JCK had a mere 27,309 tests). Tiger passes all of
    them.
  • Regression
    tests
         These are tests written by development
    engineers to make sure that a bug gets fixed and stays fixed. 99.7% of the
    tests in the regression-test suite pass. We've carefully reviewed the few
    failures, and in all cases we decided that the risk of fixing them at this
    late stage outweighs their relatively small end-user impact.
  • Functional tests     These are tests written
    by quality engineers to test the overall functionality of the JDK. 99.7% of
    the tests in the functional-test suite pass. As with the failing regression
    tests, fixing the few remaining failures is just not worth the risk right
    now.

A total of 8,002 features, enhancements, and bug fixes were integrated into
Tiger, so when I step back and think about it this way I'm fairly amazed that
it's working so well.

Tiger is the highest-quality JDK that we've ever built. Is it perfect?
No, of course not. There are no doubt still some bugs lurking, but hopefully
none is too serious. If Tiger quality is important to you then please download
and test the release candidate and
let us know
right away if something's wrong. The next couple of weeks are our last chance
to fix any thermonuclear, hair-on-fire, sky-is-falling showstopper bugs.

Related Topics >>