Kelly O'Hair's Blog

September 2006 Archives

Teamware, Mercurial, and SCCS revs that go bump in the night

As seen in Mark's recent blog on the JDK and Source Code Management (SCM) solutions at http://blogs.sun.com/mr/entry/openjdk_scm, you can see that we are finally looking at converting to a new SCM.

Inside Sun the historic and most common SCM used has been Teamware. The Sun Workshop product had delivered Teamware with it's compilers and tools for many releases, but the newer Sun Studio product dropped Teamware from it's feature list many years ago. Of course this didn't stop the developers from continuing to use it. The best high-level description of Teamware I could find is at http://docs.sun.com/source/806-3573/intro.html.

Besides all the basic SCM features you'd want, Teamware provided two extremely valuable features in my view. The primary feature was the ability to have any number of workspaces (repositories) with a parent/child relationship, there is no single repository. The second feature was it's extremely powerful merging tool called filemerge. Luckily filemerge can work with any SCM, and over the years many source merging tools have come into the picture, so as much as I loved filemerge, it's not that critical a SCM feature anymore. But that first feature of having essentially a tree of repositories allowed us to do a sort of 'divide & conquer' or distributed approach to development. Any group of developers could spin off a team workspace and a child workspace per team member, completely independent from the master workspace. This distributed development model allowed teams to work in complete isolation, yet have all the SCM tools available to them at any number of levels. Well, bottom line is, we just can't give this up very easily.

At the core of Teamware was SCCS which managed source file changes on a one by one basis. A Teamware workspace is essentially a set of SCCS controlled source files, with a bit of frosting. If you really want the nitty gritty details, take a look at http://docs.sun.com/source/806-3573/underhood.html It's these SCCS files that contain the various revisions of source files, along with comments and details from the developer when a revision of a source file was created.

As Mark stated in his blog, we are strongly leaning toward Mercurial, but we need to study the impact of the conversion. Martin Englund has been looking into the SCCS migration into Mecurial so we have a feeling of the size and performance differences. Just copying files into a new SCM isn't the issue here, it's the conversion of all the file revisions and the information that goes along with each revision. As an example, the Hotspot workspace alone contains roughly 3,300 files, but has a total of 140,000 individual file revisions. And Hotspot is one of the younger workspaces in our JDK workspace corral. We know (thanks to Martin) that Mercurial can handle the 3,300 files with amazing speed, and Martin even tried 40,000 files. It was very fast. Now we need to see what happens when it gets all the revisions too.

Teamware did not provide a very good 'change set' model, the workspaces had individual history of putbacks or commits, but the history of these actions didn't propagate up the tree of repositories. So we lack any good Teamware history data to help us group the individual SCCS file revisions together into logical 'change sets'. So for the most part we are left with the individual SCCS revision data for thousands of files. So we are brainstorming how we can group individual file changes together into change sets with just the:

• SCCS comments (in theory the files of a change set might have the same comment)
• possible bug ids (7 digit numbers) in the comments (in theory the files of a change set might have the same bug id)
• user id's (in theory a change set would have the same user id)
• times with no time zone :^( (maybe in theory, the files of a change set would have been changed within 24 hours of each other?)

Why not just do a blind 'one file revision' to 'one change set'? I guess it's part of being an engineer, we can do better than that. :^) And it looks like a fun graph project too, humm, maybe I can get some use out of that Mathematics degree I have?

Of course, if we figure this out, and I don't get fired for doing a horrible job of it (just kidding), this SCCS revision conversion work could map to any of SCM's with a change set model, but we will be focusing on Mercurial for the time being.

The Open Solaris team did a fairly detailed study on SCMs which is available at http://opensolaris.org/os/community/tools/scm/.

Mercurial is an open source project and details about Mercurial can be found at http://www.selenic.com/mercurial/wiki/index.cgi. Martin Englund has also been nice enough to provide a Teamware to Mercurial mapping for the existing Teamware users at http://blogs.sun.com/martin/entry/mercurial_for_teamware_users.

No transition will be easy, and we know that some of you might be upset that the choice wasn't SubVersion, but for what we need to do in the JDK, with our JDK developers all over the world, our answer wasn't SubVersion.

JPRT: Build/Test System for the JDK

I did a little blogging on JPRT at http://blogs.sun.com/kto/entry/jprt_sun_hardware_is_so but that was mostly to talk about the COOL rack of Sun hardware that I used. Now I want to talk a little more about why we need something like JPRT, and what it does for us. I've been working on this JPRT project for quite some time now, so I've kind of lost touch with the real world lately. Ronald Reagan is still the President, isn't he? ;^) Anyway....

JPRT ("JDK Putback Reliablity Testing", but ignore what the letters stand for, I change what they mean every day, just to annoy people :^) is a build and test system for the JDK, or any source base that has been configured for JPRT. As I mentioned in the above blog, JPRT is a major modification to a system called PRT that the HotSpot VM development team has been using for many years, very successfully I might add. Keeping the source base always buildable and reliable is the first step in the 12 steps of dealing with your product quality... or was the 12 steps from Alcoholics Anonymous... oh well, anyway, it's the first of many steps. ;^)

Internally when we make changes to any part of the JDK, there are certain procedures we are required to perform prior to any putback or commit of the changes. The procedures often vary from team to team, depending on many factors, such as whether native code is changed, or if the change could impact other areas of the JDK. But a common requirement is a verification that the source base with the changes (and merged with the very latest source base) will build on many of not all 8 platforms, and a full 'from scratch' build, not an incremental build, which can hide full build problems. The testing needed varies, depending on what has been changed.

Anyone that was worked on a project where multiple engineers or groups are submitting changes to a shared source base knows how disruptive a 'bad commit' can be on everyone. How many times have you heard:
"So And So made a bunch of changes and now I can't build!".
But multiply the number of platforms by 8, and make all the platforms old and antiquated OS versions with bizarre system setup requirements and you have a pretty complicated situation (see http://download.java.net/jdk6/docs/build/README-builds.html).

We don't tolerate bad commits, but our enforcement is somewhat lacking, usually it's an 'after the fact' correction. Luckily the Source Code Management system we use (another antique called TeamWare) allows for a tree of repositories and 'bad commits' are usually isolated to a small team. Punishment to date has been pretty drastic, the Queen of Hearts in 'Alice in Wonderland' said 'Off With Their Heads', well trust me, you don't want to be the engineer doing a 'bad commit' to the JDK. With JPRT, hopefully this will become a thing of the past, not that we have had many 'bad commits' to the master source base, in general the teams doing the integrations know how important their jobs are and they rarely make 'bad commits'. So for these JDK integrators, maybe what JPRT does is keep them from chewing their finger nails at night. ;^)

Over the years each of the teams have accumulated sets of machines they use for building, or they use some of the shared machines available to all of us. But the hunt for build machines is just part of the job, or has been. And although the issues with consistency of the build machines hasn't been a horrible problem, often you never know if the Solaris build machine you are using has all the right patches, or if the Linux machine has the right service pack, or if the Windows machine has it's latest updates. Hopefully the JPRT system can solve this problem. When we ship the binary JDK bits, it is SO very important that the build machines are correct, and we know how difficult it is to get them setup. Sure, if you need to debug a JDK problem that only shows up on Windows XP or Solaris 9, you'll still need to hunt down a machine, but not as a regular everyday occurance.

I'm a big fan of a regular nightly build and test system, constantly verifying that a source base builds and tests out. There are many examples of automated build/tests, some that trigger on any change to the source base, some that just run every night. Some provide a protection gateway to the 'golden' source base which only gets changes that the nightly process has verified are good. The JPRT (and PRT) system is meant to guard the source base before anything is sent to it, guarding all source bases from the evil developer, well maybe 'evil' isn't the right word, I haven't met many 'evil' developers, more like 'error prone' developers. ;^) Humm, come to think about it, I may be one from time to time. :^{ But the point is that by spreading the build up over a set of machines, and getting the turnaround down to under an hour, it becomes realistic to completely build on all platforms and test it, on every putback. We have the technology, we can build and rebuild and rebuild, and it will be better than it was before, ha ha... Anybody remember the Six Million Dollar Man? Man, I gotta get out more often.. Anyway, now the nightly build and test can become a 'fetch the latest JPRT build bits' and start extensive testing (the testing not done by JPRT, or the platforms not tested by JPRT).

Is it Open Source? No, not yet. Would you like to be? Let me know. Or is it more important that you have the ability to use such a system for JDK changes?

So enough blabbering on about this JPRT system, tell me what you think.
And let me know if you want to hear more about it or not.

Stay tuned for the next episode, same Bloody Bat time, same Bloody Bat channel. ;^)

-kto

This work is licensed under a Creative Commons License.