Skip to main content

Testing Your Documentation

Posted by tball on April 21, 2008 at 2:14 PM PDT

One responsibility of the JavaFX Script compiler project is to deliver a programming guide for the language. One mark of a good programming guide is that it has lots of short code examples which demonstrate the language, and as the language evolves, it's really important that the guide be updated to match the definition. Review should find most cases where the documentation no longer describes the language correctly, but it's often easy to overlook subtle changes in grammar which cause the example code to break. So there is a real tension where the more examples you have, the more work it is to verify they still build and run correctly.

Donald Knuth recognized this problem many years ago, and he responded by developing what he called Literate Programming. Literate programming's premise is that both the source code and its documentation should be co-mingled in common source files, which are processed to create the program's source files and printed documentation. The idea is that keeping a program's source code and documentation as close to each other as possible reduces divergence between them.
Sound familiar? Right, javadoc and Doxygen are examples of embedding API documentation in source files; not as powerful as true literate programming (which documents the actual source code, not just its API), but a big reason why the large API sets developed over the past decade haven't collapsed from their own weight.

When we were setting up the JavaFX Script compiler project, we believed it was important that its programming guide be generated as part of our build, rather than delivered independently by a separate team. One reason was based on our experience with javadoc, and another is that our two doc-writers are on loan, and so could disappear at any time leaving the development engineers responsible for the manual's maintenance. So we chose to write it using DocBook, which is an authoring system that allows documentation sources to be written in XML, stored in our version control system, generated by Ant or make, and can generate many output formats including as HTML and Adobe's PDF.

Although we have a book which can be edited by anyone with access to our source repository, we still ran into major headaches keeping the code examples compilable. So I just committed a simple command-line tool and Ant task (available here) which extract the text from DocBook <programlisting> elements and generate source files which the build can verify are compilable. The tool is called "DocBookTangle", since Dr. Knuth named his source extractor "tangle". This idea of testing programming documentation by compiling it appears to have originated with Python's doctext module, and it's a good one Agile teams should consider adopting.

Unfortunately, my implementation is a little clunky because DocBook has a strict DTD which we want our documents to adhere to (another automated test possibility!). The <programlisting> element has only a few attributes defined, and the most applicable ones are enumerations. So to start a new source file, we declare <programlisting> with a continuation="restarts" attribute; this creates a file named <chapterName>-<count>.fx, where count is incremented with each new program listing in that chapter. If an example is broken into pieces with explanations in between, subsequent <programlisting> elements use the continuation="continues" attribute, so their text is appended to the source file being currently extracted. "In theory", Java examples can be extracted by setting a language="java" attribute. However, Java programs require that the file name match the file's main class name, so either the tool needs to scan the extracted source for the name, or the build script needs to rename the extracted file. Happily, JavaFX Script doesn't care what the source files are called, as long as they end in ".fx". (Yet another reason to try JavaFX Script. ;-)

How well does it work? After converting the introduction chapter and running this test, thirty compilation errors were found. This shows that the problem was worse than we thought, and shows the value of automating documentation verification. We're now scrambling to convert the other chapters and fix any remaining errors, then run this test with every build to make sure they don't return.

Related Topics >>


That's a neat approach. I had the same problem some time ago but attacked it from a different angle: I kept the source code in the source repo and wrote an ant task (snip) to extract snippets from that code and place them in ant variables, which can then easily be used to replace text in other files.

Along similar lines I wrote another task (rundoc) to execute console commands and convert the output to docbook for inclusion as examples. Perhaps you would find these helpful.

This manual page was generated as docbook using both of the above (snip on top, rundoc below).

- Marty Lamb