Skip to main content

My Ant Adventure (Updated 1/23/2007)

Posted by kellyohair on January 3, 2007 at 1:46 PM PST

Update for 1/23/2007, just a very short note on windows.


    The findbugs target needs to add vmlauncher="false", so the line:

    changes to


    It's not exactly clear why this is necessary, but this allows
    the findbugs target to work on windows, and also works everywhere else. The findbugs -version exec was also removed because this
    will trigger the GUI to start up if it runs findbugs.bat,
    and findbugs.bat doesn't seem to have a -version option.

    In addition, the ant 1.7.0 install bits seem to have a windows
    bin/ant.cmd file that does not have execute permissions,
    by adding execute permissions to this file, ant can be run from a
    Makefile. Again, it's not exactly clear why this is needed, but
    it makes sense for it to have execute permissions, so this seems
    harmless.

Updated 1/22/2007, tried to mark
changes in bold underline.
Also added the actual

build.xml

to download.

Now I have never liked ants (see my
ants
blog), but
this story is about my adventure with the
Apache Ant
build system.
I can safely say that I still hate ants, all ants, but even ants
have a place in the world, ant build scripts included.

So I was crawling along with my little
NetBeans
Java project just happy as a clam using NetBeans 5.5 with the
findbugs
plugin,
and the
JUnit
tests that NetBeans help me to create and run so easily.
I never had to write an Ant script before, I just pointed NetBeans
at my sources, set a few properties and
let NetBeans handle it. Overall it was pretty easy, and simple.

But when it came time for a batch product build, I had used a Makefile and a separate build process. So the romantic picnic at the park was over, it was time
for the ants to take over. :^)
The Makefile worked, but
wasn't ideal, and it was always a temporary thing, but
it did create the bin scripts and run some bin script tests
that I hadn't managed to get the NetBeans building to do.

First off, it's never good to have two different ways to build a product, you never know for sure if the two build mechanisms really built the same thing. So in general, when it comes to a build process,
everyone should follow the
Highlander
rule of "There can be only one".
Granted, a "development build" will always be slightly different,
probably a subset, but it should be using the same basic mechanisms
as a more formal and complete product build.

Second, I was missing some of the things I wanted built into the batch
build process, namely running findbugs and running all my JUnit tests.
So I wasn't happy with my existing batch build process.

Ideally I want a batch build process to:

  1. compile with
    full javac error checking and treat all errors as fatal
    (javac -Xlint:all -Werror)
  2. construct the
    'dist' area (the jar file and including any necessary bin scripts)
  3. run all the JUnit tests
  4. run the bin script tests
  5. run javadoc and treat all errors as fatal
  6. run findbugs with full checking and treat all errors as fatal

Lucky for me I've got a very clean set of source files, and I
want to keep them that way, otherwise I'd have to back off on some
of my "treat all errors as fatal" requirements. ;^)
The first two items above represent the typical development
build or "full build"
while inside NetBeans, but it was important to me that all the
steps should be manually runnable inside NetBeans too.

So I started my adventure to make this happen.

My first thought was about having the Makefile
run findbugs and JUnit directly,
but that seemed wrong and didn't solve
my first problem above. So the answer was fairly obvious, if I
needed a Makefile, it would be trivial to have it run
the ant script, right? So I just needed to create an ant script.
Of course it's never that easy. I did manage to get this to work,
and I did learn enough about ant to create this standalone ant script,
one that did everything I wanted, but it was NOT simple and easy,
and the end result
was not very satisfactory.

It turns out that when you create a NetBeans project with
an existing ant script, some of the NetBeans features
(like debugging a JUnit test) will not work. :^(
So some NetBeans features (or what I perceive to be features) are tied to the specific ant build scripts that
NetBeans creates.

Then there were the ant problems, which involved having a
version of ant that would work with the latest
findbugs
ant task and the
junit
ant task. I finally settled on using ant 1.7 but
since I allow for my project to build anywhere, I needed ant 1.7
everywhere, and that meant I had to manage my own version of ant
(I could not trust all systems to have an ant that would work for me).
I think the ant inside NetBeans 5.5 was 1.6 something, and it
didn't work with the findbugs ant task for some reason, but that was ok because while
inside NetBeans you really want to use the findbugs plugin for the
best interaction with your sources.
So I finally gave up on the findbugs ant task and just used:

    [prettify]
       <condition property="findbugs.home" value="/Applications/findbugs">
            <os family="mac"/>
        </condition>
        
        <target name="findbugs-batch" depends="init"
                description="Run findbugs in batch mode">
            <exec 
                executable="${findbugs.home}/bin/findbugs"
                failonerror="true">
                <arg value="-textui"/>
                <arg value="-effort:max"/>
                <arg value="-low"/>
                <arg path="${dist.jar}"/>
            </exec>
        </target>
    
    [/prettify]


An UPDATE on this.
Turns out that Windows is giving me
no end of grief regarding pathnames,
so this was changed to assume findbugs was in
the PATH setting, overall this seems to make life easier.
In general avoiding having ant deal with full paths is ideal,
that also means avoiding the ant property ${basedir}
which WILL be a full path.
The Makefile changed too, see below.
I want this ant script
to work on MacOSX, Solaris, Linux, and Windows, others
may not have this requirement.
I also wanted it to work no matter where the source
was moved to, so again, avoid fullpaths.
I also added the findbugs
-exitcode option so that errors trigger a
non-zero process exit code and also the
-maxHeap 512 option to give findbugs lots of heap space
(it seems to need it, and runs a bit faster this way).
So now I use something more like:

    [prettify]
        <target name="myfindbugs" depends="init"
                description="Run findbugs in batch mode">
            <exec 
                executable="findbugs"
                failonerror="true">
                <arg value="-maxHeap"/>
                <arg value="512"/>
                <arg value="-textui"/>
                <arg value="-effort:max"/>
                <arg value="-low"/>
                <arg value="-exitcode"/>
                <arg path="${dist.jar}"/>
            </exec>
        </target>
    
    [/prettify]

As it turns out, because using the
findbugs ant task
causes the ant VM to run out of memory, so you had to restart ant
with more memory, what a pain.
Using my above batch target the findbugs process
uses it's own VM. This really doesn't change the performance much since findbugs takes a few minutes to run anyway, and just using the bin findbugs script must set the max memory up higher or something.

So now I had to deal with this loss of NetBeans Junit debug
capability.
I did like how NetBeans automatically created the menu for the
targets in my build.xml file, but I could not live without the
ability to quickly debug any JUnit test. So I needed to either
configure my build.xml file better, or try another approach.

So I went back to letting NetBeans create it's own ant scripts
and when I found where NetBeans placed these files (build.xml plus the nbproject directory),
I just copied over all the nbproject directory and the build.xml
file to my project's
Mercurial
repository and reopened the repository as a
pre-configured NetBeans project.
That way I didn't have to worry about matching the NetBeans ant
conventions to get the JUnit debug feature to work.


An update on this, first, do NOT include the nbproject/private directory
in your repository.
And second,
if you tell NetBeans to use anything other than the default
Java platform, things don't work very well.
This was easy for me to avoid, but the way the Java platforms
are defined in the ant scripts didn't make the files transportable.
Maybe this was a Mac specific thing?

Then I edited the NetBeans generated build.xml file
(which is now my primary product build.xml file) and
added to it.


  • Some properties and hooks into the NetBeans ant scripts I needed
    to set for some reason.
    Some of these properties
    should not need to be set since they are set for you
    already in the nbproject files, so I'm puzzled why you have to set
    property values sometimes and not others.
    You will also notice the echo targets I have created, which is
    just my style, I like to know when targets are being run, and the
    ant verbose option is too verbose, so I added appropriate echo
    commands.



    [prettify]
        <!-- project name (determines jar and script name) -->
        <property name="project.name"           value="jprt"/>
        
        <!-- top level package name -->
        <property name="package.name"           value="jprt"/>
        
        <!-- top level paths -->
        <property name="src.dir"                value="src"/>
        <property name="test.src.dir"           value="test"/>
        <property name="lib.dir"                value="lib"/>
        
        <!-- source paths -->
        <property name="bin.src"                value="${src.dir}/bin"/>
        <property name="sbin.src"               value="${src.dir}/sbin"/>
        <property name="doc.files.src"          value="${src.dir}/${package.name}/doc-files"/>
        
        <!-- build paths -->
        <property name="build.dir"              value="build"/>
        <property name="manifest.file"          value="${build.dir}/mainfest.mf"/>
        <property name="build.classes.dir"      value="${build.dir}/classes"/>
        <property name="build.test.classes.dir" value="${build.dir}/test/classes"/>
        
        <!-- dist paths -->
        <property name="dist.dir"               value="dist"/>
        <property name="bin.dir"                value="${dist.dir}/bin"/>
        <property name="sbin.dir"               value="${dist.dir}/sbin"/>
        <property name="dist.jar"               value="${dist.dir}/${project.name}.jar"/>
        
        <!-- path to test scripts  -->
        <property name="test.script"            value="${test.src.dir}/test.sh"/>
        <property name="test.system.script"     value="${test.src.dir}/test_system.sh"/>
        
        <!-- javadoc paths -->
        <property name="dist.javadoc.dir"       value="${dist.dir}/javadoc"/>
        <property name="doc.files"              value="${dist.javadoc.dir}/${package.name}/doc-files"/>
        
        <!-- classpath settings -->
        <property name="javac.classpath"        value=""/>
        <property name="run.classpath"          value="${build.classes.dir}"/>
        <property name="javac.test.classpath"   value="${build.classes.dir}:${lib.dir}/junit.jar"/>
        <property name="run.test.classpath"     value="${build.test.classes.dir}:${javac.test.classpath}"/>
        
        <!-- default system options -->
        <condition property="system.instance" value="testsystem-ant">
            <not> <isset property="system.instance"/> </not>
        </condition>
        
        <!-- always provide debugging information -->
        <property name="javac.debug"            value="true"/>
        
        <!-- the main class for the project -->
        <property name="main.class"             value="${package.name}.tools.Main"/>
        
        <!-- make sure netbeans knows we have a manifest file -->
        <property name="manifest.available"     value="true"/>
    
        <!-- hooks into netbeans ant files -->
        <target name="-pre-compile"             depends="mycompilestart"/>
        <target name="-post-compile"            depends="mycompileend"/>
        <target name="-do-jar-with-manifest"    depends="mymanifest"/>
        <target name="-pre-jar"                 depends="myjarstart,mymanifest"/>
        <target name="-post-jar"                depends="myjarend,mybinfiles,mysbinfiles,mypropfiles,myjartests"/>
        <target name="javadoc"                  depends="myjavadoc"/>
    
    
  • The special targets of my own:
    
        <!-- just used to add messages at certain points -->
        <target name="mycompilestart"> <echo message="COMPILING"/>           </target>
        <target name="mycompileend">   <echo message="COMPILING COMPLETED"/> </target>
        <target name="myjarstart">     <echo message="BUILDING JAR"/>        </target>
        <target name="myjarend">       <echo message="JAR COMPLETED"/>       </target>
        
        <!-- create my own manifest file -->
        <target name="mymanifest" description="Create manifest file">
            <echo message="Creating manifest file: ${manifest.file}"/>
            <manifest file="${manifest.file}">
                <attribute name="Built-By"   value="${user.name}"/>
                <attribute name="Main-Class" value="${main.class}"/>
            </manifest>
        </target>
        
        <!-- copy the bin files -->
        <target name="mybinfiles" description="Create bin files">
            <echo message="Populating bin files"/>
            <mkdir dir="${bin.dir}"/>
            <copy todir="${bin.dir}">
                <fileset dir="${bin.src}" casesensitive="yes">
                    <include name="*"/>
                </fileset>
            </copy>
            <chmod dir="${bin.dir}" perm="ugo+rx" includes="*"/>
        </target>
        
        <!-- copy the sbin files -->
        <target name="mysbinfiles" description="Create sbin files">
            <echo message="Populating sbin files"/>
            <mkdir dir="${sbin.dir}"/>
            <copy todir="${sbin.dir}">
                <fileset dir="${sbin.src}" casesensitive="yes">
                    <include name="*"/>
                </fileset>
            </copy>
            <chmod dir="${sbin.dir}" perm="ugo+rx" includes="*"/>
        </target>
        
        <!-- copy the property files into the sbin directory -->
        <target name="mypropfiles" description="Create sbin property files">
            <echo message="Populating ${dist.dir} with prop files"/>
            <mkdir dir="${dist.dir}"/>
            <copy todir="${dist.dir}">
                <fileset dir="${src.dir}" casesensitive="yes">
                    <include name="*.properties"/>
                </fileset>
            </copy>
            <echo message="System instance: ${system.instance}"/>
            <!-- Ant echo is Broken: <echo file="${dist.dir}/config-default.properties"
                  message="jprt.system.instance=${system.instance}"/> -->
            <exec executable="echo" failonerror="true"
                  output="${dist.dir}/config-default.properties">
                <arg value="jprt.system.instance=${system.instance}"/>
            </exec>
        </target>
        
        <!-- test the jar file -->
        <target name="myjartests" description="Test jar.">
            <echo message="Testing: java -jar ${dist.jar} "/>
            <java jar="${dist.jar}" fork="true" failonerror="true">
                <assertions> <enable/> </assertions>
            </java>
            <echo message="Testing: java -jar ${dist.jar} help"/>
            <java jar="${dist.jar}" fork="true" failonerror="true">
                <arg value="help"/>
                <assertions> <enable/> </assertions>
            </java>
            <echo message="Testing: java -jar ${dist.jar} usage"/>
            <java jar="${dist.jar}" fork="true" failonerror="true">
                <arg value="usage"/>
                <assertions> <enable/> </assertions>
            </java>
            <echo message="Testing: java -jar ${dist.jar} version"/>
            <java jar="${dist.jar}" fork="true" failonerror="true">
                <arg value="version"/>
                <assertions> <enable/> </assertions>
            </java>
            <echo message="TESTING JAR COMPLETED."/>
        </target>
        
        <!-- test the bin script -->
        <target name="mybintests" description="Test bin.">
            <chmod file="${test.script}" perm="ugo+rx"/>
            <echo message="Testing: sh -c ${test.script} ${dist.dir}"/>
            <exec executable="sh" failonerror="true">
                <arg value="-c"/>
                <arg value="${test.script} ${dist.dir}"/>
            </exec>
            <echo message="TESTING BIN SCRIPT COMPLETED."/>
        </target>
        
        <!-- test the system script -->
        <target name="mysystemtest" description="Test system.">
            <chmod file="${test.system.script}" perm="ugo+rx"/>
            <echo message="Testing: sh -c ${test.system.script} ${dist.dir} ."/>
            <exec executable="sh" failonerror="true">
                <arg value="-c"/>
                <arg value="${test.system.script} ${dist.dir} ."/>
            </exec>
            <echo message="TESTING SYSTEM SCRIPT COMPLETED."/>
        </target>
        
        <!-- findbugs target -->
        <target name="findbugs" depends="myfindbugs"
                description="Run findbugs in batch mode"/>
        
        <!-- my target that runs findbugs directly (in separate VM with 512M heap) -->
        <target name="myfindbugs" description="Run findbugs in batch mode">
            <echo message="findbugs -maxHeap 512 -textui -effort:max -low -exitcode ${dist.jar}"/>
            <exec executable="findbugs" failonerror="true" vmlauncher="false">
                <arg value="-maxHeap"/>
                <arg value="512"/>
                <arg value="-textui"/>
                <arg value="-effort:max"/>
                <arg value="-low"/>
                <arg value="-exitcode"/>
                <arg path="${dist.jar}"/>
            </exec>
            <echo message="FINDBUGS COMPLETED."/>
        </target>
        
        <!-- my own javadoc target because doc-files don't seem to get copied over -->
        <target name="myjavadoc" depends="init,-javadoc-build,-javadoc-browse" 
                description="Build Javadoc.">
            <mkdir dir="${doc.files}"/>
            <echo message="Populating doc-files directory: ${doc.files}"/>
            <copy todir="${doc.files}">
                <fileset dir="${doc.files.src}" casesensitive="yes">
                    <include name="*"/>
                </fileset>
            </copy>
            <echo message="JAVADOC COMPLETED."/>
        </target>
        
        <!-- all target -->
        <target name="all" depends="jar,test,mybintests,javadoc,findbugs"
                description="Do everything"/>
        
    
    [/prettify]
    


So all the above is new and updated.
I could not get the echo to add a newline to the file, even following
all the documentation on echo, so I just used the exec target and the
echo command of the system.
I also had problems with javadoc populating the doc-files, so I
had to add my own doc-files copy. I'm not sure what I did
to break this. The javadoc command seemed to be sensitive to
relative vs. full paths, or the current directory setting.

My recommendation again is to avoid the use of full paths
everywhere you can, it just makes life easier.
Also, any need to do simple shell commands like:



domainname | cut -d'.' -f2 | tr '[:upper:]' '[:lower:]'



had better be done in a shell script or in the Makefile.
In my real Makefile I do some system specific shell commands
to determine the value of an ant property that I pass into ant.
This works well for me because when running ant directly I don't
need this setting.
The other way to do this would be to exec a shell script and
capture it's output.
Running shell scripts in ant seems to work best when you
run them with sh command.
Windows often will not recognize a shell script.

The findbugs task was not connected into the NetBeans
build/test system, and is just used by the Makefile or directly
from the ant script.

Well, it turns out that this works very nicely.


The following Makefile was updated to avoid the use
of full paths and simplify how the ant targets are used.

The Makefile was trivial, and looks like:

    # Makefile to simulate a NetBeans build

    # This Makefile is located one directory below the ant basedir
    TOPDIR  = ..

    # Value of ant property that needed to be created OS specific
    system_instance=testsystem

    # How to run ant
    ANT_OPTIONS += -Djprt.system.instance=${system_instance}
    ANT = ant $(ANT_OPTIONS)

    # All ant targets of interest
    ANT_TARGETS = all jar test javadoc findbugs clean init compile

    # Create a make target for each
    $(ANT_TARGETS):
            ( cd $(TOPDIR) && $(ANT) $@ )

    # Declare these phony (not filenames)
    .PHONY: $(ANT_TARGETS)

Update, now the path to ant and findbugs just need to be put in your PATH.

I'm sure there are better ways to do some of this, but I did get the above working, and I am just an ant beginner. I still don't see any easy way to loop over a set of names with ant, and the 'condition' task is really confusing. (No wonder there are 600+ page books on how to write ant scripts.) I suppose people say the same thing about Makefiles. ;^)

Hope someone gets something out of this, and yes I've learned to live with ants. ;^)

-kto

Related Topics >>