Teamware, Mercurial, and SCCS revs that go bump in the night

As seen in Mark's recent blog on the JDK and Source Code Management (SCM) solutions at http://blogs.sun.com/mr/entry/openjdk_scm, you can see that we are finally looking at converting to a new SCM.

Inside Sun the historic and most common SCM used has been Teamware. The Sun Workshop product had delivered Teamware with it's compilers and tools for many releases, but the newer Sun Studio product dropped Teamware from it's feature list many years ago. Of course this didn't stop the developers from continuing to use it. The best high-level description of Teamware I could find is at http://docs.sun.com/source/806-3573/intro.html.

Besides all the basic SCM features you'd want, Teamware provided two extremely valuable features in my view. The primary feature was the ability to have any number of workspaces (repositories) with a parent/child relationship, there is no single repository. The second feature was it's extremely powerful merging tool called filemerge. Luckily filemerge can work with any SCM, and over the years many source merging tools have come into the picture, so as much as I loved filemerge, it's not that critical a SCM feature anymore. But that first feature of having essentially a tree of repositories allowed us to do a sort of 'divide & conquer' or distributed approach to development. Any group of developers could spin off a team workspace and a child workspace per team member, completely independent from the master workspace. This distributed development model allowed teams to work in complete isolation, yet have all the SCM tools available to them at any number of levels. Well, bottom line is, we just can't give this up very easily.

At the core of Teamware was SCCS which managed source file changes on a one by one basis. A Teamware workspace is essentially a set of SCCS controlled source files, with a bit of frosting. If you really want the nitty gritty details, take a look at http://docs.sun.com/source/806-3573/underhood.html It's these SCCS files that contain the various revisions of source files, along with comments and details from the developer when a revision of a source file was created.

As Mark stated in his blog, we are strongly leaning toward Mercurial, but we need to study the impact of the conversion. Martin Englund has been looking into the SCCS migration into Mecurial so we have a feeling of the size and performance differences. Just copying files into a new SCM isn't the issue here, it's the conversion of all the file revisions and the information that goes along with each revision. As an example, the Hotspot workspace alone contains roughly 3,300 files, but has a total of 140,000 individual file revisions. And Hotspot is one of the younger workspaces in our JDK workspace corral. We know (thanks to Martin) that Mercurial can handle the 3,300 files with amazing speed, and Martin even tried 40,000 files. It was very fast. Now we need to see what happens when it gets all the revisions too.

Teamware did not provide a very good 'change set' model, the workspaces had individual history of putbacks or commits, but the history of these actions didn't propagate up the tree of repositories. So we lack any good Teamware history data to help us group the individual SCCS file revisions together into logical 'change sets'. So for the most part we are left with the individual SCCS revision data for thousands of files. So we are brainstorming how we can group individual file changes together into change sets with just the:

• SCCS comments (in theory the files of a change set might have the same comment)
• possible bug ids (7 digit numbers) in the comments (in theory the files of a change set might have the same bug id)
• user id's (in theory a change set would have the same user id)
• times with no time zone :^( (maybe in theory, the files of a change set would have been changed within 24 hours of each other?)

Why not just do a blind 'one file revision' to 'one change set'? I guess it's part of being an engineer, we can do better than that. :^) And it looks like a fun graph project too, humm, maybe I can get some use out of that Mathematics degree I have?

Of course, if we figure this out, and I don't get fired for doing a horrible job of it (just kidding), this SCCS revision conversion work could map to any of SCM's with a change set model, but we will be focusing on Mercurial for the time being.

The Open Solaris team did a fairly detailed study on SCMs which is available at http://opensolaris.org/os/community/tools/scm/.

Mercurial is an open source project and details about Mercurial can be found at http://www.selenic.com/mercurial/wiki/index.cgi. Martin Englund has also been nice enough to provide a Teamware to Mercurial mapping for the existing Teamware users at http://blogs.sun.com/martin/entry/mercurial_for_teamware_users.

No transition will be easy, and we know that some of you might be upset that the choice wasn't SubVersion, but for what we need to do in the JDK, with our JDK developers all over the world, our answer wasn't SubVersion.