Hope these tidbits are helpful to somone.

-kto

JVM TI Agents Article

Just a plug (and additional reference) on my December 2006 article on JVM TI at http://java.sun.com/developer/technicalArticles/J2SE/jvm_ti.

I really had not expected this article to be very popular, but I was assuming that only people writing JVM TI agents would be interested. It appears there is quite a bit of general curiosity on VM agents. Of course the Java Lobby and Sun System News links helped too. :^)

Big thanks to Janice for all her help with this article! It would never have read as well without her help.

Just for reference, there is also a JVMPI to JVM TI conversion article at: http://java.sun.com/developer/technicalArticles/Programming/jvmpitransition/.

-kto

My Ant Adventure (Updated 1/23/2007)

Update for 1/23/2007, just a very short note on windows.

The findbugs target needs to add vmlauncher="false", so the line:

<exec executable="findbugs" failonerror="true">

changes to

<exec executable="findbugs" failonerror="true" vmlauncher="false">

It's not exactly clear why this is necessary, but this allows the findbugs target to work on windows, and also works everywhere else. The findbugs -version exec was also removed because this will trigger the GUI to start up if it runs findbugs.bat, and findbugs.bat doesn't seem to have a -version option.

In addition, the ant 1.7.0 install bits seem to have a windows bin/ant.cmd file that does not have execute permissions, by adding execute permissions to this file, ant can be run from a Makefile. Again, it's not exactly clear why this is needed, but it makes sense for it to have execute permissions, so this seems harmless.

Updated 1/22/2007, tried to mark changes in bold underline. Also added the actual build.xml to download.

Now I have never liked ants (see my ants blog), but this story is about my adventure with the Apache Ant build system. I can safely say that I still hate ants, all ants, but even ants have a place in the world, ant build scripts included.

So I was crawling along with my little NetBeans Java project just happy as a clam using NetBeans 5.5 with the findbugs plugin, and the JUnit tests that NetBeans help me to create and run so easily. I never had to write an Ant script before, I just pointed NetBeans at my sources, set a few properties and let NetBeans handle it. Overall it was pretty easy, and simple.

But when it came time for a batch product build, I had used a Makefile and a separate build process. So the romantic picnic at the park was over, it was time for the ants to take over. :^) The Makefile worked, but wasn't ideal, and it was always a temporary thing, but it did create the bin scripts and run some bin script tests that I hadn't managed to get the NetBeans building to do.

First off, it's never good to have two different ways to build a product, you never know for sure if the two build mechanisms really built the same thing. So in general, when it comes to a build process, everyone should follow the Highlander rule of "There can be only one". Granted, a "development build" will always be slightly different, probably a subset, but it should be using the same basic mechanisms as a more formal and complete product build.

Second, I was missing some of the things I wanted built into the batch build process, namely running findbugs and running all my JUnit tests. So I wasn't happy with my existing batch build process.

Ideally I want a batch build process to:

1. compile with full javac error checking and treat all errors as fatal (javac -Xlint:all -Werror)
2. construct the 'dist' area (the jar file and including any necessary bin scripts)
3. run all the JUnit tests
4. run the bin script tests
5. run javadoc and treat all errors as fatal
6. run findbugs with full checking and treat all errors as fatal

Lucky for me I've got a very clean set of source files, and I want to keep them that way, otherwise I'd have to back off on some of my "treat all errors as fatal" requirements. ;^) The first two items above represent the typical development build or "full build" while inside NetBeans, but it was important to me that all the steps should be manually runnable inside NetBeans too.

So I started my adventure to make this happen.

My first thought was about having the Makefile run findbugs and JUnit directly, but that seemed wrong and didn't solve my first problem above. So the answer was fairly obvious, if I needed a Makefile, it would be trivial to have it run the ant script, right? So I just needed to create an ant script. Of course it's never that easy. I did manage to get this to work, and I did learn enough about ant to create this standalone ant script, one that did everything I wanted, but it was NOT simple and easy, and the end result was not very satisfactory.

It turns out that when you create a NetBeans project with an existing ant script, some of the NetBeans features (like debugging a JUnit test) will not work. :^( So some NetBeans features (or what I perceive to be features) are tied to the specific ant build scripts that NetBeans creates.

Then there were the ant problems, which involved having a version of ant that would work with the latest findbugs ant task and the junit ant task. I finally settled on using ant 1.7 but since I allow for my project to build anywhere, I needed ant 1.7 everywhere, and that meant I had to manage my own version of ant (I could not trust all systems to have an ant that would work for me). I think the ant inside NetBeans 5.5 was 1.6 something, and it didn't work with the findbugs ant task for some reason, but that was ok because while inside NetBeans you really want to use the findbugs plugin for the best interaction with your sources. So I finally gave up on the findbugs ant task and just used:

   <condition property="findbugs.home" value="/Applications/findbugs">
        <os family="mac"/>
    </condition>
    
    <target name="findbugs-batch" depends="init"
            description="Run findbugs in batch mode">
        <exec 
            executable="${findbugs.home}/bin/findbugs"
            failonerror="true">
            <arg value="-textui"/>
            <arg value="-effort:max"/>
            <arg value="-low"/>
            <arg path="${dist.jar}"/>
        </exec>
    </target>

An UPDATE on this. Turns out that Windows is giving me no end of grief regarding pathnames, so this was changed to assume findbugs was in the PATH setting, overall this seems to make life easier. In general avoiding having ant deal with full paths is ideal, that also means avoiding the ant property ${basedir} which WILL be a full path. The Makefile changed too, see below. I want this ant script to work on MacOSX, Solaris, Linux, and Windows, others may not have this requirement. I also wanted it to work no matter where the source was moved to, so again, avoid fullpaths. I also added the findbugs -exitcode option so that errors trigger a non-zero process exit code and also the -maxHeap 512 option to give findbugs lots of heap space (it seems to need it, and runs a bit faster this way). So now I use something more like:

    <target name="myfindbugs" depends="init"
            description="Run findbugs in batch mode">
        <exec 
            executable="findbugs"
            failonerror="true">
            <arg value="-maxHeap"/>
            <arg value="512"/>
            <arg value="-textui"/>
            <arg value="-effort:max"/>
            <arg value="-low"/>
            <arg value="-exitcode"/>
            <arg path="${dist.jar}"/>
        </exec>
    </target>

As it turns out, because using the findbugs ant task causes the ant VM to run out of memory, so you had to restart ant with more memory, what a pain. Using my above batch target the findbugs process uses it's own VM. This really doesn't change the performance much since findbugs takes a few minutes to run anyway, and just using the bin findbugs script must set the max memory up higher or something.

So now I had to deal with this loss of NetBeans Junit debug capability. I did like how NetBeans automatically created the menu for the targets in my build.xml file, but I could not live without the ability to quickly debug any JUnit test. So I needed to either configure my build.xml file better, or try another approach.

So I went back to letting NetBeans create it's own ant scripts and when I found where NetBeans placed these files (build.xml plus the nbproject directory), I just copied over all the nbproject directory and the build.xml file to my project's Mercurial repository and reopened the repository as a pre-configured NetBeans project. That way I didn't have to worry about matching the NetBeans ant conventions to get the JUnit debug feature to work.

An update on this, first, do NOT include the nbproject/private directory in your repository. And second, if you tell NetBeans to use anything other than the default Java platform, things don't work very well. This was easy for me to avoid, but the way the Java platforms are defined in the ant scripts didn't make the files transportable. Maybe this was a Mac specific thing?

Then I edited the NetBeans generated build.xml file (which is now my primary product build.xml file) and added to it.

• Some properties and hooks into the NetBeans ant scripts I needed to set for some reason. Some of these properties should not need to be set since they are set for you already in the nbproject files, so I'm puzzled why you have to set property values sometimes and not others. You will also notice the echo targets I have created, which is just my style, I like to know when targets are being run, and the ant verbose option is too verbose, so I added appropriate echo commands.
  

  
      <!-- project name (determines jar and script name) -->
      <property name="project.name"           value="jprt"/>
      
      <!-- top level package name -->
      <property name="package.name"           value="jprt"/>
      
      <!-- top level paths -->
      <property name="src.dir"                value="src"/>
      <property name="test.src.dir"           value="test"/>
      <property name="lib.dir"                value="lib"/>
      
      <!-- source paths -->
      <property name="bin.src"                value="${src.dir}/bin"/>
      <property name="sbin.src"               value="${src.dir}/sbin"/>
      <property name="doc.files.src"          value="${src.dir}/${package.name}/doc-files"/>
      
      <!-- build paths -->
      <property name="build.dir"              value="build"/>
      <property name="manifest.file"          value="${build.dir}/mainfest.mf"/>
      <property name="build.classes.dir"      value="${build.dir}/classes"/>
      <property name="build.test.classes.dir" value="${build.dir}/test/classes"/>
      
      <!-- dist paths -->
      <property name="dist.dir"               value="dist"/>
      <property name="bin.dir"                value="${dist.dir}/bin"/>
      <property name="sbin.dir"               value="${dist.dir}/sbin"/>
      <property name="dist.jar"               value="${dist.dir}/${project.name}.jar"/>
      
      <!-- path to test scripts  -->
      <property name="test.script"            value="${test.src.dir}/test.sh"/>
      <property name="test.system.script"     value="${test.src.dir}/test_system.sh"/>
      
      <!-- javadoc paths -->
      <property name="dist.javadoc.dir"       value="${dist.dir}/javadoc"/>
      <property name="doc.files"              value="${dist.javadoc.dir}/${package.name}/doc-files"/>
      
      <!-- classpath settings -->
      <property name="javac.classpath"        value=""/>
      <property name="run.classpath"          value="${build.classes.dir}"/>
      <property name="javac.test.classpath"   value="${build.classes.dir}:${lib.dir}/junit.jar"/>
      <property name="run.test.classpath"     value="${build.test.classes.dir}:${javac.test.classpath}"/>
      
      <!-- default system options -->
      <condition property="system.instance" value="testsystem-ant">
          <not> <isset property="system.instance"/> </not>
      </condition>
      
      <!-- always provide debugging information -->
      <property name="javac.debug"            value="true"/>
      
      <!-- the main class for the project -->
      <property name="main.class"             value="${package.name}.tools.Main"/>
      
      <!-- make sure netbeans knows we have a manifest file -->
      <property name="manifest.available"     value="true"/>
  
      <!-- hooks into netbeans ant files -->
      <target name="-pre-compile"             depends="mycompilestart"/>
      <target name="-post-compile"            depends="mycompileend"/>
      <target name="-do-jar-with-manifest"    depends="mymanifest"/>
      <target name="-pre-jar"                 depends="myjarstart,mymanifest"/>
      <target name="-post-jar"                depends="myjarend,mybinfiles,mysbinfiles,mypropfiles,myjartests"/>
      <target name="javadoc"                  depends="myjavadoc"/>
  
  

• The special targets of my own:
  

  
      <!-- just used to add messages at certain points -->
      <target name="mycompilestart"> <echo message="COMPILING"/>           </target>
      <target name="mycompileend">   <echo message="COMPILING COMPLETED"/> </target>
      <target name="myjarstart">     <echo message="BUILDING JAR"/>        </target>
      <target name="myjarend">       <echo message="JAR COMPLETED"/>       </target>
      
      <!-- create my own manifest file -->
      <target name="mymanifest" description="Create manifest file">
          <echo message="Creating manifest file: ${manifest.file}"/>
          <manifest file="${manifest.file}">
              <attribute name="Built-By"   value="${user.name}"/>
              <attribute name="Main-Class" value="${main.class}"/>
          </manifest>
      </target>
      
      <!-- copy the bin files -->
      <target name="mybinfiles" description="Create bin files">
          <echo message="Populating bin files"/>
          <mkdir dir="${bin.dir}"/>
          <copy todir="${bin.dir}">
              <fileset dir="${bin.src}" casesensitive="yes">
                  <include name="*"/>
              </fileset>
          </copy>
          <chmod dir="${bin.dir}" perm="ugo+rx" includes="*"/>
      </target>
      
      <!-- copy the sbin files -->
      <target name="mysbinfiles" description="Create sbin files">
          <echo message="Populating sbin files"/>
          <mkdir dir="${sbin.dir}"/>
          <copy todir="${sbin.dir}">
              <fileset dir="${sbin.src}" casesensitive="yes">
                  <include name="*"/>
              </fileset>
          </copy>
          <chmod dir="${sbin.dir}" perm="ugo+rx" includes="*"/>
      </target>
      
      <!-- copy the property files into the sbin directory -->
      <target name="mypropfiles" description="Create sbin property files">
          <echo message="Populating ${dist.dir} with prop files"/>
          <mkdir dir="${dist.dir}"/>
          <copy todir="${dist.dir}">
              <fileset dir="${src.dir}" casesensitive="yes">
                  <include name="*.properties"/>
              </fileset>
          </copy>
          <echo message="System instance: ${system.instance}"/>
          <!-- Ant echo is Broken: <echo file="${dist.dir}/config-default.properties"
                message="jprt.system.instance=${system.instance}"/> -->
          <exec executable="echo" failonerror="true"
                output="${dist.dir}/config-default.properties">
              <arg value="jprt.system.instance=${system.instance}"/>
          </exec>
      </target>
      
      <!-- test the jar file -->
      <target name="myjartests" description="Test jar.">
          <echo message="Testing: java -jar ${dist.jar} "/>
          <java jar="${dist.jar}" fork="true" failonerror="true">
              <assertions> <enable/> </assertions>
          </java>
          <echo message="Testing: java -jar ${dist.jar} help"/>
          <java jar="${dist.jar}" fork="true" failonerror="true">
              <arg value="help"/>
              <assertions> <enable/> </assertions>
          </java>
          <echo message="Testing: java -jar ${dist.jar} usage"/>
          <java jar="${dist.jar}" fork="true" failonerror="true">
              <arg value="usage"/>
              <assertions> <enable/> </assertions>
          </java>
          <echo message="Testing: java -jar ${dist.jar} version"/>
          <java jar="${dist.jar}" fork="true" failonerror="true">
              <arg value="version"/>
              <assertions> <enable/> </assertions>
          </java>
          <echo message="TESTING JAR COMPLETED."/>
      </target>
      
      <!-- test the bin script -->
      <target name="mybintests" description="Test bin.">
          <chmod file="${test.script}" perm="ugo+rx"/>
          <echo message="Testing: sh -c ${test.script} ${dist.dir}"/>
          <exec executable="sh" failonerror="true">
              <arg value="-c"/>
              <arg value="${test.script} ${dist.dir}"/>
          </exec>
          <echo message="TESTING BIN SCRIPT COMPLETED."/>
      </target>
      
      <!-- test the system script -->
      <target name="mysystemtest" description="Test system.">
          <chmod file="${test.system.script}" perm="ugo+rx"/>
          <echo message="Testing: sh -c ${test.system.script} ${dist.dir} ."/>
          <exec executable="sh" failonerror="true">
              <arg value="-c"/>
              <arg value="${test.system.script} ${dist.dir} ."/>
          </exec>
          <echo message="TESTING SYSTEM SCRIPT COMPLETED."/>
      </target>
      
      <!-- findbugs target -->
      <target name="findbugs" depends="myfindbugs"
              description="Run findbugs in batch mode"/>
      
      <!-- my target that runs findbugs directly (in separate VM with 512M heap) -->
      <target name="myfindbugs" description="Run findbugs in batch mode">
          <echo message="findbugs -maxHeap 512 -textui -effort:max -low -exitcode ${dist.jar}"/>
          <exec executable="findbugs" failonerror="true" vmlauncher="false">
              <arg value="-maxHeap"/>
              <arg value="512"/>
              <arg value="-textui"/>
              <arg value="-effort:max"/>
              <arg value="-low"/>
              <arg value="-exitcode"/>
              <arg path="${dist.jar}"/>
          </exec>
          <echo message="FINDBUGS COMPLETED."/>
      </target>
      
      <!-- my own javadoc target because doc-files don't seem to get copied over -->
      <target name="myjavadoc" depends="init,-javadoc-build,-javadoc-browse" 
              description="Build Javadoc.">
          <mkdir dir="${doc.files}"/>
          <echo message="Populating doc-files directory: ${doc.files}"/>
          <copy todir="${doc.files}">
              <fileset dir="${doc.files.src}" casesensitive="yes">
                  <include name="*"/>
              </fileset>
          </copy>
          <echo message="JAVADOC COMPLETED."/>
      </target>
      
      <!-- all target -->
      <target name="all" depends="jar,test,mybintests,javadoc,findbugs"
              description="Do everything"/>
      
  
  

So all the above is new and updated. I could not get the echo to add a newline to the file, even following all the documentation on echo, so I just used the exec target and the echo command of the system. I also had problems with javadoc populating the doc-files, so I had to add my own doc-files copy. I'm not sure what I did to break this. The javadoc command seemed to be sensitive to relative vs. full paths, or the current directory setting.

My recommendation again is to avoid the use of full paths everywhere you can, it just makes life easier. Also, any need to do simple shell commands like:
domainname | cut -d'.' -f2 | tr '[:upper:]' '[:lower:]'
had better be done in a shell script or in the Makefile. In my real Makefile I do some system specific shell commands to determine the value of an ant property that I pass into ant. This works well for me because when running ant directly I don't need this setting. The other way to do this would be to exec a shell script and capture it's output. Running shell scripts in ant seems to work best when you run them with sh command. Windows often will not recognize a shell script.

The findbugs task was not connected into the NetBeans build/test system, and is just used by the Makefile or directly from the ant script.

Well, it turns out that this works very nicely.

The following Makefile was updated to avoid the use of full paths and simplify how the ant targets are used.

The Makefile was trivial, and looks like:

# Makefile to simulate a NetBeans build

# This Makefile is located one directory below the ant basedir
TOPDIR  = ..

# Value of ant property that needed to be created OS specific
system_instance=testsystem

# How to run ant
ANT_OPTIONS += -Djprt.system.instance=${system_instance}
ANT = ant $(ANT_OPTIONS)

# All ant targets of interest
ANT_TARGETS = all jar test javadoc findbugs clean init compile

# Create a make target for each
$(ANT_TARGETS):
        ( cd $(TOPDIR) && $(ANT) $@ )

# Declare these phony (not filenames)
.PHONY: $(ANT_TARGETS)

Update, now the path to ant and findbugs just need to be put in your PATH.

I'm sure there are better ways to do some of this, but I did get the above working, and I am just an ant beginner. I still don't see any easy way to loop over a set of names with ant, and the 'condition' task is really confusing. (No wonder there are 600+ page books on how to write ant scripts.) I suppose people say the same thing about Makefiles. ;^)

Hope someone gets something out of this, and yes I've learned to live with ants. ;^)

-kto

This work is licensed under a Creative Commons License.