Tom Ball's Blog

Tools Archives

Testing Your Documentation

One responsibility of the JavaFX Script compiler project is to deliver a programming guide for the language. One mark of a good programming guide is that it has lots of short code examples which demonstrate the language, and as the language evolves, it's really important that the guide be updated to match the definition. Review should find most cases where the documentation no longer describes the language correctly, but it's often easy to overlook subtle changes in grammar which cause the example code to break. So there is a real tension where the more examples you have, the more work it is to verify they still build and run correctly.

Donald Knuth recognized this problem many years ago, and he responded by developing what he called Literate Programming. Literate programming's premise is that both the source code and its documentation should be co-mingled in common source files, which are processed to create the program's source files and printed documentation. The idea is that keeping a program's source code and documentation as close to each other as possible reduces divergence between them. Sound familiar? Right, javadoc and Doxygen are examples of embedding API documentation in source files; not as powerful as true literate programming (which documents the actual source code, not just its API), but a big reason why the large API sets developed over the past decade haven't collapsed from their own weight.

When we were setting up the JavaFX Script compiler project, we believed it was important that its programming guide be generated as part of our build, rather than delivered independently by a separate team. One reason was based on our experience with javadoc, and another is that our two doc-writers are on loan, and so could disappear at any time leaving the development engineers responsible for the manual's maintenance. So we chose to write it using DocBook, which is an authoring system that allows documentation sources to be written in XML, stored in our version control system, generated by Ant or make, and can generate many output formats including as HTML and Adobe's PDF.

Although we have a book which can be edited by anyone with access to our source repository, we still ran into major headaches keeping the code examples compilable. So I just committed a simple command-line tool and Ant task (available here) which extract the text from DocBook <programlisting> elements and generate source files which the build can verify are compilable. The tool is called "DocBookTangle", since Dr. Knuth named his source extractor "tangle". This idea of testing programming documentation by compiling it appears to have originated with Python's doctext module, and it's a good one Agile teams should consider adopting.

Unfortunately, my implementation is a little clunky because DocBook has a strict DTD which we want our documents to adhere to (another automated test possibility!). The <programlisting> element has only a few attributes defined, and the most applicable ones are enumerations. So to start a new source file, we declare <programlisting> with a continuation="restarts" attribute; this creates a file named <chapterName>-<count>.fx, where count is incremented with each new program listing in that chapter. If an example is broken into pieces with explanations in between, subsequent <programlisting> elements use the continuation="continues" attribute, so their text is appended to the source file being currently extracted. "In theory", Java examples can be extracted by setting a language="java" attribute. However, Java programs require that the file name match the file's main class name, so either the tool needs to scan the extracted source for the name, or the build script needs to rename the extracted file. Happily, JavaFX Script doesn't care what the source files are called, as long as they end in ".fx". (Yet another reason to try JavaFX Script. ;-)

How well does it work? After converting the introduction chapter and running this test, thirty compilation errors were found. This shows that the problem was worse than we thought, and shows the value of automating documentation verification. We're now scrambling to convert the other chapters and fix any remaining errors, then run this test with every build to make sure they don't return.

Divide and Conquer

As anyone following the JavaFX Script compiler project knows, we have a schedule gun to our heads to finish a useful tool quickly, so the interpreter-based prototype shown at last year's JavaOne can be retired. It's a big job since the language doesn't have a specification yet, but our team of four engineers plus some really good volunteers are set to deliver what looks to be a great product. Of course I'm biased in this appraisal since I'm one of its contributors, but feel free to browse the project's source or subscribe to its dev and/or commits email aliases and make your own assessment.

Unfortunately, it has become more obvious as the project progresses that when our clients say "compiler" they really mean a full SDK, similar to what is delivered by the JDK. In particular, our management just assumed that a version of the javadoc tool which understands JavaFX Script would magically appear without any stated requirements or staffing. So we had a sub-project with a short deadline, no engineer available, and frustrated customers.

To make matters worse, javadoc engineers are hard to find since the job requires two unique skill sets: compiler engineering and web designing. Why does the JDK's javadoc output look dated? Because compiler engineers have owned it for the past few years. Why was it lagging behind the language a few years ago? You guessed it, our doc-writers were supporting it. Now, our team understands compilers, obviously, but none of us have any design skills worth noting. However, we have some engineers with great design skills on other teams. The "correct" solution would be to pull one engineer from each team and have them work together, but there wasn't time nor anyone available. Sound familiar?

What we did was follow the Unix rules for tool development: break the task apart, define a tool for each part, and use ASCII text to communicate between them. I wrote a simple doclet that dumped all information available from the Doclet API to an XML file (other people have done this over the years), then copied the JDK's javadoc source and enhanced it to provide JavaFX Script-specific information for that doclet (the doclet also works with Java sources when used with the JDK's javadoc). Joshua Marinacci then wrote an XSLT translator to generate XHTML pages similar to what javadoc's standard doclet creates. We use this tool to generate the project's UI Runtime documentation, and soon, all JavaFX Script-based API. The full tool source is here.

This divide-and-conquer strategy really pays off. Neither Josh nor I put much time into this project (as shown by how short both the doclet and XSLT script are), so our main jobs weren't impacted much. This approach also makes it much easier for our community to contribute: don't like the (currently primitive) output? An XSLT file is much less intimidating than javadoc's source, so jump in and make it better. But why limit ourselves to one output format? Just use the tool's -xsltfile option to use your own translator. And why limit ourselves to documentation generation? That XML file defines a model of any JavaFX Script and/or Java-based program, which can be used by other tools. As the Unix tool masters learned early on, breaking a big tool up into smaller, specialized tools opens the door to all kinds of unanticipated uses.

My JavaOne Talk on Advanced Refactoring

• The acceptance notifications have been slowly trickling in (and as far as I know are not finished yet). Although I heard rumors that some speakers were notified in early February, I was notified less than a week ago.
  
  
• Last year there were many blog comments from people who felt such announcements were either boastful (if accepted) or whining (if not). "Once bitten, twice shy" so I expect some accepted speakers are keeping quiet.
  
  
• The talks most developers want to hear are from other developers, who tend to be introverted and so are lousy at promoting themselves. After all, anyone who prefers spending half their waking hours staring at a screen can't claim to be outgoing! In my case, it's easy to talk publicly about the technologies that excite me, but it's much harder to talk about personal news.

The Power of Community

    Although, what would really be neat is

    { $p$; $v = $e; return $v; } => { $p$; return $e; } :: declaredIn($v, $p$);
 
    which seems possible in principle, but not with the current rules language.

• I got a simple, but very useful, enhancement suggestion. Especially with tools, a developer really isn't the best person to know what parts of a project are useful and what it is missing -- potential users of it are.
• Even better is that I got a free code-review from some smart engineers who had never seen the language before. This is proof that good programmers can write in any language!

Jackpot Early Access Available

This work is licensed under a Creative Commons License.