David Walend's Blog

Better JavaDoc on http://java.net

The Problems with using CVS for JavaDoc

I have two problems with using CVS to store JavaDoc. The first is philosophical -- CVS is a source code control system, and JavaDoc .html files are a derived product, not source code. If you can check out the source code, you can build the JavaDoc yourself. If you change the JavaDoc, it doesn't matter; next time you rebuild it, those changes are lost. The second problem is logistical -- I either have to remove all the JavaDoc from the last release and add all the new JavaDoc from the latest release. Or I have to put each release under a new directory; that saves me removing it, but means that everyone's links point to an old copy and grows the www subdirectory over time. Either way, I have to add several layers deep of JavaDoc directories. And I get a 15 MB cvs change log in email every time.

Sourceforge's Approach

Sourceforge gives each user a unix account and each project a public web directory. Back when I kept projects on Sourceforge, I ftped a tar ball of .html files for the web pages, including the JavaDoc. It worked, but was fairly primitive. Plus it relied on letting the users log in remotely to sourceforge's servers. That's probably more than Sun, O'reilly and Collabnet are willing to pay for.

I've thought of three alternatives, but don't know which would be the best.

Subversion

Collabnet, one of java.net's sponsors, paid to create Subversion, a replacement for CVS. I've used it at home for a while, and on a few projects at work. I'd rather use Subversion than CVS in any case; Subversion treats each commit as a transaction, linking all the changes together. Using Subversion to store source code and web pages would still not address my philosophical concerns. However, Subversion's ability to easily remove and add whole directory structures would be a huge improvement over CVS. I could even get rid of the top level v1, v2, v3, etc. directories.

.war Files

In the previous blog, straun suggested a separate repository of .war files for web sites. Using .war files would let us maintain project web sites the way Java developers are used to. The approach addresses both of my concerns. When I'm ready to release a new version of the web site, I build a .war file and post it in the appropriate place, instead of checking in the files as source code. And I handle the site as a single .war file instead of single files in directories. How close is http://java.net from being able to support that? What other pitfalls are involved? Would the administrators have to do something special to keep developers from putting executable code in the .war files?

Shared JavaDoc Database

A more compelling solution: Join with JDocs, a shared cross-linked JavaDoc database, or to set up something similar on java.net hardware. The JavaDocs would be completely separate from source code. And over time, we would grow a large, cross-linked set of API docs that would help us use each others' work. The JDoc site hasn't added new material since September 7th. I don't know what would be involved for the java.net administrators to set up such a system, or the java.net sponsors to link up with JDoc.

Those are the options I've thought of. Which of these should we ask for? Is there some other option we should consider?

Thanks,

Dave

Comments

• I'm keeping the javadoc for my java.net projects offsite, uploading them by FTP and pointing to them from the java.net site.
  I find FTP a very easy way to do it.
  
  Using Internet Explorer (groan all you like - it works for me), I browse to the ftp directory where I keep the javadocs, delete the 'api' directory, then drag and drop the same directory from the latest build.
  30 seconds later and I'm done.
  
  I'd love to be able to FTP them up to java.net.
  I think it's sad that, at the moment, it's impractical for the whole project to be hosted here.

  Posted by: grlea on March 08, 2005 at 02:59 PM

• You should be able to request JDocs.org to add your API, and to update it when it changes. There are links on their site to do that.

  JDocs uses the Ashkelon FOSS system, which stores APIs in a relational database. It should be easy to attach to a website (that was the original intent for the project), or for java.net to coordinate with JDocs to provide for easy updating, etc.

  Personally, if I'm going to use the API much, I prefer to have the APIs locally, and am migrating to using Ashkelon locally as well.

  Patrick

  Posted by: pdoubleya on March 09, 2005 at 05:26 AM

• david,

  i don't know whether you know this or not but jdocs is based on my open-source project 'ashkelon.' they basically use ashkelon to populate their db. when they got started they contacted me and i helped them get set up. i even went as far as adding features to ashkelon to accomodate their needs.

  where it differs from ashkelon is the jsps's: they wanted their ui to look just like javadocs did (monochromatic, html 3.2 stuff). with the ashkelon base, it's _very_ easy to customize the ui for javadocs because they can all be templatized.

  anyhow, i'd be more than happy to work with the powers that be at java.net to make that happen.

  the ashkelon project is well-documented. furthermore, i'm giving talks on ashkelon all across the country with the no fluff just stuff conferences.

  - eitan

  Posted by: eitan on March 09, 2005 at 07:30 AM

• grlea,

  Maybe if they opened up the "documents and files" segment to be a better match for javadoc, that would open up something that works more like ftp. I don't really want to have to hold the javadocs on a separate site. Worth considering, though.

  Thanks,

  Dave

  Posted by: dwalend on March 09, 2005 at 07:08 PM

• eitan, Patrick,

  Do either of you know if the JDoc site is being kept up? The front page says they haven't added an API since September 7th, 2004.

  What's the overhead of running ashkelon? Does it jjust plug into a servlet engine? Any special security concerns? Anything that will make the java.net crew grind their teeth?

  Thanks,

  Dave

  Posted by: dwalend on March 09, 2005 at 07:18 PM

• You should be able to browse the Javadoc, but the developer shouldn't have to muck around with uploading them and keeping them up to date - that should be automated.
  
  Here's how it should work:
  Initial State = no javadoc stored
  Someone clicks a "look at the javadoc" link, and as appropriate (depending on the link) the servlet responsible for handling that request checks to see if the Javadoc exists. Also rather than compile the whole shebang, it should be smart enough to only make the relevant bits. Eg if you're looking for the index, it should compile only what it needs to for that (perhaps just using the file names and packages of the .java files). Or if I'm looking at a particular class it only needs to compile the Javadoc for that one class.
  
  The second thing it should do is in case the existing javadoc is older than the class you're looking at it should compile a new one.
  
  The third thing is that if noone's looked at a javadoc for a while, there should be a maintenance facility to maybe get rid of it (move it to cheaper storage maybe?)
  
  
  The point of all this is a not too smart automated facility could be quite quick, and not suck storage the way cvs would, while taking a lot of hassle off the coders, and also maximising the convenience for the viewers.

  Posted by: rickcarson on March 09, 2005 at 08:55 PM

• Rick: automation is an absolute must for people. I'm happy to do a few manual things as long as they're easy (which checking javadoc into CVS is not). I have a colleague who will almost refuse to do anything unless it's automatable. I wish he'd write my Ant files for me. : )
  
  David: I'd be pretty happy with any solution for putting javadocs onto java.net as long as:
  - it's strongly associated with the project (i.e. in the same subdomain)
  - it's not in CVS
  - it's easy to upload manually - akin to my single d'n'd with IE.
  Automation would be good, but isn't a requirement for me. It is for some people, though, which probably means it SHOULD be a requirement (in case anyone with control over the site is reading this).

  Posted by: grlea on March 09, 2005 at 09:15 PM

• Another thought on using Ashkelon. We could write a doclet for Ashkelon that outputs the docs in a standard XML format, which includes a version attribute. On request for a certain class, if not already in the central Ashkelon database, they are pulled in--and, if the version is different from that in the database, the newer version is pulled in. The load could be "chunked" using XPath to load by package, class, etc. Links would always go through the main servlet to allow for lazy loading.

  The generated XML can be uploaded to the project www directory, and overwrite whatever is currently in there, given a version change.

  Also, if the servlet can return either HTML or XML, using the XML one can write a rich client browser (one of my pet projects) or submit different XSL for different styled layouts.

  I think Ashkelon or something like it is the best solution for managing multiple JavaDoc sets. It is stable code and works well.

  Patrick

  Posted by: pdoubleya on March 10, 2005 at 02:05 AM

• Surely the sensible answer is for java.net to use your project's build script to create the javadocs form the checked-in version of the code. I'm absolutely with you being against checking in derived artifacts. We use cruisecontrol to run regular builds from our repository and make the generated reports, including javadoc, available. Jakarta also provide regular builds via gump.

  Posted by: mgladdish on March 10, 2005 at 07:47 AM

• rickcarson,

  Great suggestions on how the world should be. That might be putting too much burden on the java.net service. I'd be surprised if they had the resources to build out a new piece of software. And it'd be hard to justify in light of Ashkelon. (You could sign on with the Ashkelon project and pull the world toward your vision...)

  Dave

  Posted by: dwalend on March 11, 2005 at 05:20 AM

• grlea, Patrick, mgladdish,

  Great ideas on automating this stuff.

  The project automation tools mgladdish suggests aren't widely accepted yet. (I've found them a bit restrictive.) Ant seems to be universal enough, so that got me thinking...

  Perhaps what we need is a standard ant Task to hand off javadoc to a remote instance of Ashkelon. I'd slip it into my big release target, just after the tag task. Others could link it up with cruisecontrol or maven or what-comes-next.

  We'd need to convince the java.net crew to host an Ashkelon instance, and open the firewall enough to let us talk to it. Eitan said he could help.

  How's that sound?

  Dave

  Posted by: dwalend on March 11, 2005 at 05:28 AM

• sorry it took me a few days to get back to this thread. yall have some terrific ideas. i think we could do our community a great service by setting this up for all java.net projects. i have some initial ideas which i'll outline here for starters.

  the ashkelon repository manager stuffs apis into a db. ashkelon supports either mysql or postgres. every project maps nicely to a single api. we'll need to figure out how long it will take ashkelon to populate all java.net projects and how the viewer performs on that level/scale. my idea is to start simple: have the repository manager rebuild the database nightly so that every morning all projects have fresh javadocs, and this would all be automated. there is one piece that project developers would have to supply: the api.xml file which lists the packages to be documented (and the api name, and a few other tidbits). we can get the source code from cvs. we can also get much of the information that is traditionally supplied in api.xml directly from the java.net project information, so that the requirement from project leaders/developers would be minimal (possibly nil). once we get this working, we could then start refactoring: making ashkelon a little smarter so it doesn't have to rebuild the docs for an api if the source hasn't changed. also ashkelon supports a per-api add/remove feature so that updating an api can be performed independently of the others.

  it looks like a number of you are ready and willing to help out with setting this up, which would be terrific. we could surely use your help. a community effort!

  Posted by: eitan on March 15, 2005 at 03:22 PM

• Dave wrote:

  > Do either of you know if the JDoc site is being kept up? The front page
  > says they haven't added an API since September 7th, 2004.

  I don't really keep up with what goes on at javalobby these days. But you must realize that they're a small team. Also once the top 100+ APIs are documented, you pretty much have things covered for the community...with the exception of updating the APIs with newer version releases.

  > What's the overhead of running ashkelon? Does it just plug into a
  > servlet engine? Any special security concerns? Anything that will
  > make the java.net crew grind their teeth?

  the viewer is a war file you drop into a servlet/jsp container. security concerns should be minimal as the viewer is essentially read-only. though i really think we should think through various scenarios, possible vulnerabilities.

  Posted by: eitan on March 16, 2005 at 08:16 PM

This work is licensed under a Creative Commons License.