Skip to main content

What if we built Java code with...Java?

Posted by timboudreau on January 30, 2008 at 1:36 PM PST

My friend Jon had an interesting insight: Both Ant and Maven rely on lots of XML. XML is good for describing data and terrible for describing behavior. A build is mostly behavior. What if, instead of tormenting Ant into iterating on a bunch of subprojects of subprojects, if we just used an actual programming language to write build scripts. Like, oh, say...Java, for instance?

So he created the Gosling project. It's pretty embryonic - and I think some of the file and resource classes could be replaced by straight usage of things like javax.tools.FileObject, but it has a nice simplicity. Here's the constructor for Gosling's own build to build itself. It has a similar feel to what Jon did in designing Wicket's APIs:

    public GoslingProject() {
        final Folder root = new Folder("/Projects/gosling/workspace/gosling");
        final Folder source = root.folder("src");
        final Folder lib = root.folder("lib");
        final Folder target = root.folder("target");
        add(new JavaApplicationBuilder() {
            @Override
            protected Set getDependencies() {
                final ResourceDescriptorSet dependencies = new ResourceDescriptorSet();
                dependencies.add(Apache.apache.wicket.core.development.resources());
                dependencies.add(Apache.apache.wicket.extensions.development.resources());
                return dependencies;
            }

            @Override
            protected Set getJars() {
                return lib.nestedJars();
            }

            @Override
            protected Folder getSourceFolder() {
                return source;
            }

            @Override
            protected Folder getTargetFolder() {
                return target;
            }
        });
}

Brazilian Salt Shakers

I spent last night wrestling with writing an Ant script to build, test and build Javadoc for an ad-hoc collection of projects, where a bunch of custom information needs to be gathered from the projects and embedded elsewhere, and I just found myself thinking this is so not the way to build software!.

I mean, the appeal of things like Ant is that many things are built in a pretty similar way; Maven is even more "my way or the highway" in that regard. I use (and sometimes like) both of them. And in theory an Ant XML script (xml script - now there's an oxymoron) is human readable - although I challenge anyone to make heads or tails of this in ten minutes. It certainly beats Make and tab vs. space madness. There's an argument that it's toolable - that a tool can analyze an Ant script. This seems to me to be a red-herring if you do a design like Wicket or Gosling use - where you know that the entire state you need to analyze is going to be set up in the constructor of a known class of a known type. The Javac Tree API may not be for the faint of heart, but analyzing the closure of a constructor is perfectly doable.

My point is that Ant doesn't really deliver the clarity it promised except in the most trivial of cases. You could have at least as much clarity with plain-old Java code - you just need to start from a good design so the code can speak for itself. And a design where all targets will be added to the build in the constructor is pretty darned clear. Isn't one of the things the agile crowd talks about a lot letting code speak for itself?

Not to mention that running javac in-process ought to be blazingly fast. And that such a project could import and call existing Ant tasks to do things with a thin adapter layer - so anybody's custom tasks or missing functionality could be handled leveraging stuff that's already out there.

It seems like a pretty nice idea to me.

Related Topics >>

Comments

Just to throw another scripting language in this discussion. What about JavaFX? It has this declarative quality that i even tried to use it as a replacement for xml since its more human readable. And its a very powerful scripting laguage as well, i think not only for building ui stuff.
    JavaApplicationBuilder {
      path: "{project.path}/libs"
      dependencies: ["org.apache.wiket.extensions","org.apache.wiket.development"]
      jars: getNestedJars()
        ...}
    would look quite nice? Just a thought....

I would be more interested in exploring a hyperdata replacement for ant. That would possibly solve a much larger problem, and one that is becoming more and more pressing as open source projects more and more often use components from other projects. I explore the idea further in my recent post "replacing ant with rdf"

A current project I've joined is using Maven and I'm also find myself thinking "this is so not the way to build software!".

I agree with fwilhelm. For me, the key point with automated build is that it should pretty much be the same for most projects. If it isn't then you need to look at why and try to fix that. Once you have a uniform structure you need only little information to build it.

Great idea! ...wish I'd thought of it. I also get "stuck" with ant as soon as things get non-trivial, and just resort to a simple shell. I think this Gosling project is the way to go; I'd be happy to start off with a bit of boilerplate code in Java that I wouldn't need with ant, so that later on, I won't get stuck and have to start mixing ant-xml with scripts.
Here's a classic article about XML readability: http://www.ibm.com/developerworks/xml/library/x-sbxml.html

Check out Savant. It is an advanced build tool that uses Groovy for building build plugins and targets. It also has a robust dependency management tool and lots of other cool features. Right now it doesn't have any releases, but you can look at the source code and some of the plugins. Hopefully Savant will start picking up steam this year!

With the number of people that know how Ant works, vs the number of people that know how that custom build Java application will work, I bet you'd have fewer headaches fixing a broken build if it was ANT. Yes you can move away from Java and toward a Java scripting language, but then yes, you lose tooling. I don't think Ant and Maven tooling is a red-herring, it allows all of those custom widgets in your IDE know what to do when you click on things. Maven tooling is very excellent in Netbeans because it uses a standard Maven pom, whereas Ant support in Netbeans is limited to build scripts generated and maintained by Netbeans. The maven integration is way smarter. You couldn't get any of that using a custom program. Even if you decided to create a framework to limit the amount of code you write for each project, you'd essentially be re-creating a new Ant or Maven and may even find yourself start to create XML configuration files to run it. Then you'd be back at square one.

My experiences with build systems so far is: if it is necessary to program within build files, something went wrong. Programming in build files is to complicated. A build file must be configured. If you feel the necessity to program, the build system is to weak!

I agree: a Java-based build tool sounds like it would be nicer than mucking with XML if it was done right.

XML is useful for cross-language communication (machines talking to machines) not for humans to play around with. XML should not be used for configuration!

jadonohu: "I've heard this argument / complaint about Ant a hundred times, and I still don't get it. You don't `program in XML' with ant. Ant is (almost) completely declarative, and it's dead simple to use. ...Write a custom task if you need to do something esoteric. IMHO" I hate XML. I find it unnecessarily difficult to read, and writing a custom task is more difficult than it should be. If Java is more power than necessary, than how about implementing the build process using a declarative language such as LISP, Prolog, ML or Haskell?

My personal gripe with ANT is the way you have to think backwards to do anything. Its not "do-this-then-this-then-this" but more like :

  • 4. Jar up these files once you've
  • 3. Run and passed all these unit test, not not until you've
  • 2. Compiled these files into this directory once you've.
  • 1. Copied these files here

    I agree with the idea of using java to do this. All the stuff needed is in ANT already, surely we could just wrap some friendly API's around the existing code?

You might want to look at gant. That project integrates ant into groovy.

"XML is good for describing data and terrible for describing behavior" I've heard this argument / complaint about Ant a hundred times, and I still don't get it. You don't "program in XML" with ant. Ant is (almost) completely declarative, and it's dead simple to use: 1. Copy these files here 2. Compile these files into this directory 3. jar up these files. What exactly are you trying to program? Just declare what you want done in the xml and be done with it. Write a custom task if you need to do something esoteric. IMHO, if you're "programming in xml", you're doing something wrong with the tool.

Maven relies on XML for project description and plugin configuration. The actual plugins (that do the work in the build) are written in Java. In Maven 2.x at least. Maven 1.x used xml (Apache Commons' Jelly) to write plugins and it was indeed "sub-optimal".

Tim, instead of just limiting it to Java, you could consider exposing Gosling object into a scripting engine via JSR223. Then there you would be able to express your build in any scripting language.

Well, as someone with extensive experience with VoiceXML, I can say I completely agree with XML being a bad fit for scripting. If you want an idea of what a java api might look like, just try implementing ant tasks using the programmatic ant api. You can do things like instantiate a javac task and add filesets to it, etc. Obviously, there would be changes you'd make if you wanted to make that the primary api, but it's a place to start.

what about Raven? Rake and Gems on JRuby. Allows dependencies to be autoloaded from repository just like Maven.

Well, I don't know that verbosity is such an issue; if the API you're calling solves the scope of the problem well and cleanly, you shouldn't need that much code (but when you do need to do something complex, you can).

Very much the philosophy of the more productive languages, i.e. Ruby/RoR has Rake (Ruby make) files. I think we need Java to become just a little less verbose in order to allow such DSL-ness.

What about BeanShell? www.beanshell.org it's java (or javaish) and you do not have the chicken and the egg problem. Or Javascript and use the --runscript jvm parameter. But if you are looking for something thats here now, you can checkout gant (GroovyAnt) http://gant.codehaus.org/ you even get to use ant tasks, just no XML.

Well, you only have the chicken and egg problem the first time you build it (yes I used Ant to build Gosling).

Actually the memory requirements issue is something that the folks using Javac internally in NetBeans to build data structures have had to deal with as well. There's a simple solution that handles most cases: When memory grows to a certain size, stop the compilation process and then restart it against the build products of the previous run. With the exception of pathological cases (i.e. a bunch of classes whose closure is...everything), it solves the problem - building against .class files is much cheaper than building against source, and you can discard the class data and reread it if you need to.

The principal problem I see is a chicken and egg problem, how do you compile the Java class describing the build without something to build it? Java is always compiled and Ant addresses that by being purely interpretive. Tools like Gant and Rake solve that problem by having the language runtime accept scripts in addition to compiling them. As for running JavaC in-process: it ought to be blazingly fast, and for small cases of N (where N is the size of the project) it is. But as N grows you get issue relating to classpath and garbage collected memory that the early gains turn into a net loss. For those large cases of N the time issue is not an artifact of the process overhead or JVM footprint, but it is an artifact of the number of java files being compiled and the space and time needed to generate and walk the ASTs for each. But hey, it's still faster than linking large blocks of C code.

Why don't you just use Ant tasks in Java programs? It's easy! Ever wanted to copy a complete directory structure? Don't reinvent the wheel: Simply use the ant "Copy" task in you Java program! Since it's all plain Java code all you have to do is to create a project object & set the tasks properties. This way you can easily manage complex tasks that would be difficult to define in ant's XML.