Peter Kessler's Blog

TechDays in Chennai

TechDays in Chennai was fun. I can't get over the energy of the attendees, and the city around the conference. Not to mention the energy of the Sun Technical Outreach folks that put these events on.

TechDays are just like JavaOne, except instead of taking BART into San Francisco, I flew halfway around the world to get here. There were 3 tracks: roughly JavaME and JavaSE, JavaEE, and Community activities. It was wonderful, for me at least, of having the focus on just developers. It also made the event much more humane than JavaOne; with many fewer people around attendees who wanted to talk one-on-one could find the speakers and corner them for blocks of time. I also got to go to a lot of sessions I usually miss at JavaOne, so that was educational, too.

I got a lot ofinteresting questions, and I hope I encouraged some people to put their ideas up as java.net projects. The most impressive business card I got was from someone at the Indian Department of Atomic Energy, who wants to use real-time Java to control nuclear power plants.

I'm looking forward to Pune, Mumbai, Kolkata, and Delhi. If any of you jdk.Researchers are at the events, please come and introduce yourselves! The point of these trips is to meet with developers and build the java.net community, and face-to-face time is a great way to do that.

Where's Peter?

I'm off to India for the Sun Tech Day in Chennai, February 7th and 8th. Two days of bringing people up to speed on all the cool technology we have out there.

After that I'll be participating in the Sun Developer Day on February 9th in Pune, and one in Kolkata on February 14th. Then it's off to the Sun University day at Delhi University on February 16th. A chance for me to meet a lot of really smart developers, and see if we can't drum up some exciting new java.net JDK projects.

If you are at any of those events, please come by and introduce yourself.

I hope to point a lot of people at java.net and the JDK community. That will mean more competition for the Mustang Regression Challenge, but competition is good.

Chat with some HotSpot Virtual Machine Experts

On March 15, 2005 11:00 A.M. PST (19:00 UTC), Ross Knippel and I will be in a chat room answering questions about Sun's HotSpot virtual machine. I'll handle questions about storage allocation and garbage collection, while Ross handles questions about the runtime compilers. Between us we'll answer questions about performance and how to improve it (and how not to improve it). Since the Mustang sources are available, we'll also field any questions you have about the structure of the code, how things work, and all that stuff that goes on below the level of Java code.

We invite you send in questions. That's what will make the chat interesting, both for you and for us. I've started a forum topic to collect questions to seed the chat.

Welcome to the Lab

Walk this way. Welcome to the JDK lab. Here we've set up an environment in which people can safely do research under the Java Research License.

There's a light (over at the Frankenstein place) by Richard O'Brien
Riff Raff:
Let the sun and light come streaming
Into my life.
Into my life.
Brad & Janet:
There's a light.
All:
Over at the Frankenstein place.

If you want to experiment on the sources for the JDK, it's as easy as 1-2-3:

Request the "Researcher" role in the JDK project, confirming that you accept the terms of JRL. That will give you access to the jdk-research project. (We wouldn't want randoms to wander through the lab, sticking their fingers in the sockets, jiggling the more delicate experiments, etc.)
Request a project. That gets you a clean lab bench at which to work and lets you advertise your project to potential collaborators. (Leave the "Choose a community you'd like to join" field blank, and then in the "Other Notes and Comments" section request that this project be added to the JDK collection of projects.)
Have at it. You can set up forums for discussions about your project, mailing lists for your colleagues, CVS repositories, issue tracking for that last lingering bug, etc.

(If you aren't quite sure which end of the JDK to grab to start a new project, you might want to just do step 1, look around at some of the existing projects (when there are some), and help out on one of those until you get your bearings in this wonderful new world.)

Putting your project under the jdk-research project assures you that the only people who can see your project are those who have signed the JRL. Once in the lab you can talk freely about the JDK sources, trade snippets of code, discuss bug fixes, and share other ideas about the JDK.

When you have something really cool, please let us know! If you come up with a flying car, I definitely want to hear about it. If you have a bug fix or performance improvement that you think should be included in our next release, we encourage you to contribute it. We'll make sure your fix works on all of our supported platforms (40 of them for JDK 5.0!), in all our supported locales, passes the JCK tests, doesn't break backward compatibility, and all those other things about which we care deeply. If your contribution proposes API changes, we'll run those by our usual experts. (Of course, if you want to propose major new API's you should do that through the Java Community Process.) If your contribution is accepted for the next release of the platform, we'll also make sure the change gets documented in the release notes as needed, at least to acknowledge you as a contributor.

If you want ideas of things to work on, look at the bug database, or look at some of the suggestions on the the Mustang forum. Or maybe you already know your favorite, most annoying bug that for some reason Sun doesn't have the resources to fix, but on which the future of your startup company depends: get to work.

Maybe you want to try something a little less technical? Translate the manual pages or javadocs and be a local hero. We're working on a pluggable locale framework for the JRE, so you could be ready with a your favorite locale.

You could set up a class at your university on the internals of the JVM. Have a bake-off for alternate synchronization primitives. Or write a new garbage collector (there are already three of them in there, so some framework is all set up). Teach a class on software engineering practices, using the JDK as a case in point. We might even send out guest lecturers to get you started.

Or think of something I haven't. In fact, I hope you do. That's the whole point of making the JDK sources available: so that people can try whatever they want and contribute the good bits back to the community.

Where are the flying cars?

We've published 5 Mustang builds so far. But there hasn't been much to write home about. We've made some steady progress, but no sweeping language changes, no major new library packages, no ports of the virtual machine to new platforms. We haven't even implemented all of the suggestions from this forum. What's taking us so long?

It's probably worth giving people a heads up on how the Mustang release is going to progress between now and when we declare it done.

Right now we are mostly fixing bugs we didn't get into Tiger. (You'll see, this will happen again at the end of Mustang, with fixes we'll push off to Dolphin.) We're also cleaning up code and other small chores while we work out what new things to get into Mustang. There are lots of sources of good new ideas: while we were working on Tiger we talked to a lot of people about what we should do next, the Java community process has some big ticket items, our licensee partners have porting and tuning concerns, this forum is full of smart people, etc. We are also starting to field feedback from people who are using the new features we put into Tiger. All that goes into planning for Mustang. There's no shortage of bug reports, RFE's, suggestions for tuning, good ideas (and some not so good ones). We have a long list of things we want to do; the trick is figuring out what we can get done well in Mustang. Soon the planning will die down and we engineers will be heads down writing code.

But for you watching the snapshot builds come out: a note of caution. Things will probably get rough before they get better. All that new code will come with unit tests, but those don't catch all the bugs. We have a whole quality engineering team to write additional tests, but they are coming up to speed on the new features too, and sometimes they read the specs more carefully than the developers do. There are the JCK tests to write, too. QE is always behind when whole new API's are in flux. We have a quality assurance team to run all the tests on all the instruction sets and operating systems we support, in various locales, on all sorts of weird graphics cards, etc., but it takes QA a while (weeks, months) to cover the more obscure variations.

Different groups have different strategies to balance working on new features and fixing bugs. Some teams have some designated bug fixers for a release, to leave other people free to focus on new features. Some groups have everyone fix bugs in and around working on new things. Some groups have people now working on features that we know won't be ready until Dolphin. And it's not like we have a set list of bugs to be fixed that we just work our way through. Sometimes we introduce new bugs in spite of our code reviews and unit testing, and you might trip across bugs those in the snapshot releases. (And you'll report them, I hope, so that we can get them fixed.)

Probably what you'll see in the snapshots -- unfortunately -- is a temporary decrease in quality. And steps backwards in overall performance, footprint, startup time, etc. Oh, the snapshots are not as bad as all that. Those builds have been through individual engineer and group testing, integration testing with other groups, and as much testing as we can get in while still getting the snapshots out to you in less than a week. Mustang might be different because we are trying some new ways of managing things: e.g., putting the snapshots out so people can try them earlier and file reports on bugs they find. And, we'll be accepting -- real soon now -- bug fixes and features from you java.net folks. But I thought I should warn you that you might not see continuous improvement. The advantage of the snapshots is that you get to see the new features first, try them out, exploit them in your code, and give us feedback and suggestions for improvement. Things will get better.

At some point the axe will fall, and new features won't be allowed into Mustang. (Well, it's not as sharp a line as that. There are business reasons, and late JSR's, and pressure from one or another of our valued licensees, etc., that make this process a "slush" rather than a "freeze".) The idea of drawing a line is to give the QE folks time to finish writing tests for the features, and to give the QA folks time to run all their combinations. The documentation folks come around and figure out what's new and how to explain it in the release notes and web pages. New development doesn't stop, but it does slow down. People who didn't get their features into Mustang switch to getting those features into Dolphin. But finding and fixing bugs becomes a higher priority. And the review and testing bar gets higher for any changes. Also, once the release settles down we can do integrated performance analysis and work on other aspects of quality we can't do until we know what we are working with. So later in the release cycle you'll see a marked improvement in quality.

At some point, the bar for putting back changes gets so high that we'll turn down even obvious bug fixes that wouldn't have been given a second thought early in the release. Those will go into Dolphin, and then get back-ported to the updates to Mustang. (How can I be thinking, now, about updates to Mustang? Because the big wheel goes around, with the little wheel inside it.) And we'll be asking around again about priorities and features and what we should do next.

I hope that gives you a sense of what to expect in the snapshot releases and how we might deal with suggestions -- or code -- you might contribute.

Don't look for flying cars in the first few snapshot releases. They'll be along later.

Why are there two of everything?

You might have noticed that in addition to the Tiger source snapshots, we have just posted a Mustang source snapshot under the Java Research license.

So now we have two code lines being actively worked on: we continue to find bugs in Tiger and fix them in update releases, and we have on-going development in Mustang. That's bound to be confusing. I'm a HotSpot virtual machine engineer so I'm used to having two (or more) code lines in progress. But if you are going to rummage around in the HotSpot virtual machine sources (under hotspot/src), I'd like to explain why we seem to have two of a lot of things. I'll gradually work my way down through the layers of this particular onion.

We have to ship releases. We have Tiger and Mustang (and all the other releases) because we want to get stuff out into the hands of our users. But we're never really "done", so development is continuous while releases are periodic. What you see by looking at the current Tiger and Mustang source snapshots is that early in a release (Mustang) there isn't really that much difference between from the previous release (Tiger). But new development stopped on Tiger months ago. At any given time, we have (at least) two releases in progress, one in active development, the other(s) for bug fix updates.

Backward compatibility. Once you get into the sources for the virtual machine, you'll sometimes find that we often have two implementations of things. One reason for this is because we think backward compatibility is really important. While we are working on some new thing, we have to keep the old thing working, and the easiest way to do that is to keep the old thing around. While browsing through the sources, you'll find a lot of code guarded by command-line switches you probably didn't know about. (Look at all those command line switches in hotspot/src/share/runtime/globals.hpp!) Those are there so we can do A-B comparisons for functionality, conformance, performance, footprint, etc. Only when an new implementation shows itself to be compatible and substantially better than the old one do we throw the switch to use the new one. And we usually leave the switch around for a release or two in case someone wants to revert to the old behavior.

Dessert topping or floor wax. One of the problems with being a successful Java virtual machine is that people want to use you for everything, even things you didn't exactly anticipate. While the Java platform might have burst on the scene as a way of executing content for small applets in web browsers (and people still use it for that), people now also use it for running gigantic high-throughput applications on big multiprocessors. Of course, we want to make everyone happy, but that often means having alternate implementations inside the virtual machine. You can see this in the choice of the client versus server runtime compiler: the client runtime compiler gives good startup and modest performance, while the server runtime compiler is not as fast to start up, but the code it generates runs significantly faster. You can't use them both at the same time (yet), but if you are looking around the code base, you'll find both runtime compilers in there.

One size does not fit all. In that same style of offering different qualities of service, we offer something like 3 different garbage collection algorithms. The concurrent mark sweep algorithm provides lower pause times at some cost in performance, while the parallel collector offers better performance with occasional longer pauses. We're not going make that choice for our users. If you go looking for "the garbage collector", you'll be disappointed (or maybe pleasantly surprised) to find at least three of them in there.

We learn, but slowly. The HotSpot Java virtual machine is a work in progress. Ideas that we had a while ago might have been appropriate for then, but things change. So the source base changes too. We learn things about the interactions of the different parts of the virtual machine (basically: compiler(s), garbage collector(s), and runtime system) and we try to clean up the code. In some parts of the virtual machine, that means changing the interfaces, but our desire not to be disruptive means we often leave the old interface and implementation in place for a release or two. For example, the older garbage collectors use a "generation framework" that is extremely flexible, but has some overhead. The newest collector uses a less flexible interface that is more efficient. We won't gratuitously convert the older collectors (we'd risk breaking things, for no benefit to you, our customers), so you'll find both programming styles in there if you look.

It depends on your point of view. Sometimes you'll be prowling through the source and come across things that look to be two of the same thing. For example, hotspot/src/share/oops/oopsHierarchy.hpp shows what appear to be similar hierarchies for oops and klasses. But those are not alternate implementations, or us evolving the interface, or anything like that. They are two faces of the virtual machine's view of the data structures used to represent Java objects (and a few VM internal data structures). Simplifying somewhat: an oop is the Java reference to an object, whereas the klass is the way we manipulate that object from the C++ code inside the virtual machine. That's an example of where you have to be able to hold both ideas in your head at the same time, instead of looking at only the one you think you are interested in.

The HotSpot virtual machine is a collection of engineering tradeoffs and compromises. As such you will often find more than one way of doing things when you look through the sources. I hope I've clarified some of the reasons for that. If not, ask questions and I'll try to come up with answers.