Tim Boudreau's Blog

What if we built Java code with...Java?

So he created the Gosling project. It's pretty embryonic - and I think some of the file and resource classes could be replaced by straight usage of things like javax.tools.FileObject, but it has a nice simplicity. Here's the constructor for Gosling's own build to build itself. It has a similar feel to what Jon did in designing Wicket's APIs:

    public GoslingProject() {
        final Folder root = new Folder("/Projects/gosling/workspace/gosling");
        final Folder source = root.folder("src");
        final Folder lib = root.folder("lib");
        final Folder target = root.folder("target");
        add(new JavaApplicationBuilder() {
            @Override
            protected Set getDependencies() {
                final ResourceDescriptorSet dependencies = new ResourceDescriptorSet();
                dependencies.add(Apache.apache.wicket.core.development.resources());
                dependencies.add(Apache.apache.wicket.extensions.development.resources());
                return dependencies;
            }

            @Override
            protected Set getJars() {
                return lib.nestedJars();
            }

            @Override
            protected Folder getSourceFolder() {
                return source;
            }

            @Override
            protected Folder getTargetFolder() {
                return target;
            }
        });
}

I mean, the appeal of things like Ant is that many things are built in a pretty similar way; Maven is even more "my way or the highway" in that regard. I use (and sometimes like) both of them. And in theory an Ant XML script (xml script - now there's an oxymoron) is human readable - although I challenge anyone to make heads or tails of this in ten minutes. It certainly beats Make and tab vs. space madness. There's an argument that it's toolable - that a tool can analyze an Ant script. This seems to me to be a red-herring if you do a design like Wicket or Gosling use - where you know that the entire state you need to analyze is going to be set up in the constructor of a known class of a known type. The Javac Tree API may not be for the faint of heart, but analyzing the closure of a constructor is perfectly doable.

My point is that Ant doesn't really deliver the clarity it promised except in the most trivial of cases. You could have at least as much clarity with plain-old Java code - you just need to start from a good design so the code can speak for itself. And a design where all targets will be added to the build in the constructor is pretty darned clear. Isn't one of the things the agile crowd talks about a lot letting code speak for itself?

Not to mention that running javac in-process ought to be blazingly fast. And that such a project could import and call existing Ant tasks to do things with a thin adapter layer - so anybody's custom tasks or missing functionality could be handled leveraging stuff that's already out there.

It seems like a pretty nice idea to me.

Comments

• The principal problem I see is a chicken and egg problem, how do you compile the Java class describing the build without something to build it? Java is always compiled and Ant addresses that by being purely interpretive. Tools like Gant and Rake solve that problem by having the language runtime accept scripts in addition to compiling them.

  As for running JavaC in-process: it ought to be blazingly fast, and for small cases of N (where N is the size of the project) it is. But as N grows you get issue relating to classpath and garbage collected memory that the early gains turn into a net loss. For those large cases of N the time issue is not an artifact of the process overhead or JVM footprint, but it is an artifact of the number of java files being compiled and the space and time needed to generate and walk the ASTs for each. But hey, it's still faster than linking large blocks of C code.

  Posted by: shemnon on January 30, 2008 at 01:51 PM

• Well, you only have the chicken and egg problem the first time you build it (yes I used Ant to build Gosling).

  Actually the memory requirements issue is something that the folks using Javac internally in NetBeans to build data structures have had to deal with as well. There's a simple solution that handles most cases: When memory grows to a certain size, stop the compilation process and then restart it against the build products of the previous run. With the exception of pathological cases (i.e. a bunch of classes whose closure is...everything), it solves the problem - building against .class files is much cheaper than building against source, and you can discard the class data and reread it if you need to.

  Posted by: timboudreau on January 30, 2008 at 02:30 PM

• What about BeanShell? www.beanshell.org it's java (or javaish) and you do not have the chicken and the egg problem. Or Javascript and use the --runscript jvm parameter. But if you are looking for something thats here now, you can checkout gant (GroovyAnt) http://gant.codehaus.org/ you even get to use ant tasks, just no XML.
  

  Posted by: aberrant on January 30, 2008 at 02:48 PM

• Very much the philosophy of the more productive languages, i.e. Ruby/RoR has Rake (Ruby make) files. I think we need Java to become just a little less verbose in order to allow such DSL-ness.

  Posted by: mrmorris on January 30, 2008 at 02:50 PM

• Well, I don't know that verbosity is such an issue; if the API you're calling solves the scope of the problem well and cleanly, you shouldn't need that much code (but when you do need to do something complex, you can).

  Posted by: timboudreau on January 30, 2008 at 03:44 PM

• what about Raven? Rake and Gems on JRuby. Allows dependencies to be autoloaded from repository just like Maven.

  Posted by: xagile on January 30, 2008 at 03:47 PM

• Well, as someone with extensive experience with VoiceXML, I can say I completely agree with XML being a bad fit for scripting. If you want an idea of what a java api might look like, just try implementing ant tasks using the programmatic ant api. You can do things like instantiate a javac task and add filesets to it, etc. Obviously, there would be changes you'd make if you wanted to make that the primary api, but it's a place to start.

  Posted by: richunger on January 30, 2008 at 04:10 PM

• Tim,
  instead of just limiting it to Java, you could consider exposing Gosling object into a scripting engine via JSR223. Then there you would be able to express your build in any scripting language.

  Posted by: chukmunnlee on January 30, 2008 at 05:45 PM

• Maven relies on XML for project description and plugin configuration. The actual plugins (that do the work in the build) are written in Java. In Maven 2.x at least. Maven 1.x used xml (Apache Commons' Jelly) to write plugins and it was indeed "sub-optimal".

  Posted by: mkleint on January 30, 2008 at 09:00 PM

• "XML is good for describing data and terrible for describing behavior"

  I've heard this argument / complaint about Ant a hundred times, and I still don't get it. You don't "program in XML" with ant. Ant is (almost) completely declarative, and it's dead simple to use:

  1. Copy these files here
  2. Compile these files into this directory
  3. jar up these files.

  What exactly are you trying to program? Just declare what you want done in the xml and be done with it. Write a custom task if you need to do something esoteric. IMHO, if you're "programming in xml", you're doing something wrong with the tool.

  Posted by: jadonohu on January 30, 2008 at 09:43 PM

• You might want to look at
  gant.

  That project integrates ant into groovy.

  Posted by: berni on January 30, 2008 at 11:54 PM

• My personal gripe with ANT is the way you have to think backwards to do anything. Its not "do-this-then-this-then-this" but more like :
  4. Jar up these files once you've
  3. Run and passed all these unit test, not not until you've
  2. Compiled these files into this directory once you've.
  1. Copied these files here
  I agree with the idea of using java to do this. All the stuff needed is in ANT already, surely we could just wrap some friendly API's around the existing code?

  Posted by: luggypm on January 31, 2008 at 12:03 AM

• jadonohu: "I've heard this argument / complaint about Ant a hundred times, and I still don't get it. You don't `program in XML' with ant. Ant is (almost) completely declarative, and it's dead simple to use. ...Write a custom task if you need to do something esoteric. IMHO"

  I hate XML. I find it unnecessarily difficult to read, and writing a custom task is more difficult than it should be. If Java is more power than necessary, than how about implementing the build process using a declarative language such as LISP, Prolog, ML or Haskell?

  Posted by: fsilber on January 31, 2008 at 07:11 AM

• I agree: a Java-based build tool sounds like it would be nicer than mucking with XML if it was done right.

  XML is useful for cross-language communication (machines talking to machines) not for humans to play around with. XML should not be used for configuration!

  Posted by: cowwoc on January 31, 2008 at 07:21 AM

• My experiences with build systems so far is: if it is necessary to program within build files, something went wrong. Programming in build files is to complicated. A build file must be configured. If you feel the necessity to program, the build system is to weak!

  Posted by: fwilhelm on January 31, 2008 at 07:40 AM

• With the number of people that know how Ant works, vs the number of people that know how that custom build Java application will work, I bet you'd have fewer headaches fixing a broken build if it was ANT. Yes you can move away from Java and toward a Java scripting language, but then yes, you lose tooling. I don't think Ant and Maven tooling is a red-herring, it allows all of those custom widgets in your IDE know what to do when you click on things. Maven tooling is very excellent in Netbeans because it uses a standard Maven pom, whereas Ant support in Netbeans is limited to build scripts generated and maintained by Netbeans. The maven integration is way smarter. You couldn't get any of that using a custom program. Even if you decided to create a framework to limit the amount of code you write for each project, you'd essentially be re-creating a new Ant or Maven and may even find yourself start to create XML configuration files to run it. Then you'd be back at square one.

  Posted by: coding on January 31, 2008 at 09:00 AM

• Check out Savant. It is an advanced build tool that uses Groovy for building build plugins and targets. It also has a robust dependency management tool and lots of other cool features. Right now it doesn't have any releases, but you can look at the source code and some of the plugins. Hopefully Savant will start picking up steam this year!

  Posted by: voidmain on January 31, 2008 at 09:40 AM

• Great idea! ...wish I'd thought of it. I also get "stuck" with ant as soon as things get non-trivial, and just resort to a simple shell. I think this Gosling project is the way to go; I'd be happy to start off with a bit of boilerplate code in Java that I wouldn't need with ant, so that later on, I won't get stuck and have to start mixing ant-xml with scripts.

  
  Here's a classic article about XML readability: http://www.ibm.com/developerworks/xml/library/x-sbxml.html

  Posted by: atripp on January 31, 2008 at 03:37 PM

• I agree with fwilhelm. For me, the key point with automated build is that it should pretty much be the same for most projects. If it isn't then you need to look at why and try to fix that. Once you have a uniform structure you need only little information to build it.
  

  Posted by: asjf on February 01, 2008 at 03:53 AM

• A current project I've joined is using Maven and I'm also find myself thinking "this is so not the way to build software!".

  Posted by: gustavosantucho on February 01, 2008 at 07:37 AM

• Why don't you just use Ant tasks in Java programs? It's easy! Ever wanted to copy a complete directory structure? Don't reinvent the wheel: Simply use the ant "Copy" task in you Java program! Since it's all plain Java code all you have to do is to create a project object & set the tasks properties. This way you can easily manage complex tasks that would be difficult to define in ant's XML.

  Posted by: michael_schnell on February 01, 2008 at 11:48 AM

• I would be more interested in exploring a hyperdata replacement for ant. That would possibly solve a much larger problem, and one that is becoming more and more pressing as open source projects more and more often use components from other projects.

  I explore the idea further in my recent post "replacing ant with rdf"
  

  Posted by: bblfish on February 06, 2008 at 03:00 AM

• Just to throw another scripting language in this discussion. What about JavaFX? It has this declarative quality that i even tried to use it as a replacement for xml since its more human readable. And its a very powerful scripting laguage as well, i think not only for building ui stuff.
  JavaApplicationBuilder {
  path: "{project.path}/libs"
  dependencies: ["org.apache.wiket.extensions","org.apache.wiket.development"]
  jars: getNestedJars()
  ...}
  would look quite nice? Just a thought....

  Posted by: vorgi on March 19, 2008 at 03:11 PM

This work is licensed under a Creative Commons License.