The Source for Java Technology Collaboration
User: Password:



Kirill Grouchnikov

Kirill Grouchnikov's Blog

Using Java compiler in your Web Start application

Posted by kirillcool on May 27, 2005 at 05:56 AM | Comments (6)

In JAXB Workshop project, I was faced with the following problem: after the XJC generator produces a set of Java source files, I needed to compile them and load them into the running JVM in order to collect cross-reference information:

The standard technique for compiling Java source files in regular standalone application is to use the tools.jar that resides under jdk/lib directory, and use com.sun.tools.javac.Main class and its compile function to do the job. Another, more sophisticated technique that doesn't require explicitly specifying the location of the tools.jar is to use java.home system property and deduce the location of tools.jar from it:
private static Method javac = null;

static {
   try {
      File jreHome = new File(System.getProperty("java.home"));
      System.out.println("java.home = " + jreHome.getAbsolutePath());
      File toolsJar = new File(jreHome.getParent(), "lib/tools.jar");
      System.out.println("tools.jar = " + toolsJar.getAbsolutePath());
      ClassLoader toolsJarLoader = new URLClassLoader(
            new URL[] { toolsJar.toURL() });
      javac = toolsJarLoader.loadClass("com.sun.tools.javac.Main")
            .getMethod("compile", new Class[] { String[].class, PrintWriter.class });
      hasToolsJar = true;
   } catch (Exception exc) {
      javac = null;
      hasToolsJar = false;
   }
}   
In the code above lies our first problem when the code will be executed in Web Start environment - no JDK installation is needed for such environment. Moreover, all default installations of JDK 6.0 early builds (that install JRE also), point the javaws.exe to work with the JRE and not the JDK. In this case, the value of java.home will point to the JRE installation directory, that doesn't have tools.jar under lib.

At this point it becomes obvious that the Java compiler must be bundled with your application. And now the bad news - the size of the tools.jar is staggering 6,590 KB! The reason - it contains a lot of tools, such as apt, javah and so on. A simple solution would be to leave only the classes necessary for javac, resulting in about 1.5 MB, but that would not sit good with the Java license, especially for commercial applications.

Currently, the only fully-TCK compliant alternative (for Java 5.0 features) is the JDT compiler that comes with Eclipse 3.1M7. The relevant jar is org.eclipse.jdt.core_3.1.0.jar and its size is 3,430 KB, almost half that of tools.jar. The list of all available compiler options is available here. At this point, it appeared that all was left to do - replace the package name and the function parameters. However, the problem that laid ahead was unexpected - missing classes in the classpath. The files generated by JAXB's XJC generator, reference classes in jsr173_api.jar and jaxb-api.jar (both parts of JAXB 2.0 Early Access Reference Implementation). These files are, of course, properly referenced in the JNLP descriptor (along with the JDT compiler jar):
<resources>
  <j2se version="1.5+" />
  ...
  <jar href="jaxb-api.jar"/>
  <jar href="jsr173_api.jar"/>
  <jar href="org.eclipse.jdt.core_3.1.0.jar"/>
  ...
</resources>
Nonetheless, the compiler was failing with error messages indicating that the corresponding Java classes couldn't be found in the classpath. A quick look at the compiler parameters show that we can specify -cp followed by the list of jar files. The first attempt would be to simply provide the following value:
-cp jaxb-api.jar;jar173_api.jar
This approach fails, because the Web Start cache (even if you manage to find its location on the remote computer) stores the file under very cryptic names. For example, on my machine the cache is under C:\Documents and Settings\Owner\Application Data\Sun\Java\Deployment\cache\6.0, and sample file name is 3595ab4e-24c448c4.

The second approach attempts to retrieve the actual URL of these two jars in the current JVM. Here we are once again faced with a problem specific to Web Start - the remote jar files do not appear in the java.class.path system property. The only way to retrieve them is via manifest files.

A manifest file is, in fact, the most prominent component that makes jar file different from zip file. It is located under META-INF directory, and its name is MANIFEST.MF. So, how can we obtain a list of all available jars? Very simple - ask the class loader to enumerate all META-INF/MANIFEST.MF resources:
try {
   for (Enumeration<?> e = org.jvnet.jaxbw.Workspace.class
         .getClassLoader().getResources("META-INF/MANIFEST.MF");
         e.hasMoreElements();) {
      URL url = (URL) e.nextElement();
      _logger.info("Scanning " + url.getFile());
      ...
} catch (IOException exc) {
   exc.printStackTrace();
}
Sample logger output from the above code in Web Start environment is
INFO: Scanning https://jaxb-workshop.dev.java.net/webstart/jaxb-api.jar!/META-INF/MANIFEST.MF
From the above URL, we can obtain the URL of the jar itself (marked in red), and then give it to the compiler in the classpath:
-cp https://jaxb-workshop.dev.java.net/webstart/jaxb-api.jar;https://jaxb-workshop.dev.java.net/webstart/jar173_api.jar
However, the compiler still complains about the missing files. Obviously, the classpath should reference local files only. It becomes clear now that we must have local copies of the remote jars and provide their (local) filenames to the compiler. One way to do this is: download the relevant jar files based on their URLs and save them as temporary files. The files should be marked to be deleted on application exit. The code now becomes:
   private static synchronized String getLocalClassPath(Logger _logger) {
      if (isJarsLoadedLocal) {
         return localJarsClassPath;
      }

      Set classpathJars = new HashSet();

      try {
         for (Enumeration<?> e = org.jvnet.jaxbw.Workspace.class
               .getClassLoader().getResources("META-INF/MANIFEST.MF"); e
               .hasMoreElements();) {
            URL url = (URL) e.nextElement();
            _logger.info("Scanning " + url.getFile());
            if (url.getFile().contains("jaxb-api.jar")) {
               String jaxbApiLocal = getLocalJarFilename(url, _logger);
               if (jaxbApiLocal != null) {
                  classpathJars.add(jaxbApiLocal);
               }
            }
            if (url.getFile().contains("jsr173_api.jar")) {
               String jsr173ApiLocal = getLocalJarFilename(url, _logger);
               if (jsr173ApiLocal != null) {
                  classpathJars.add(jsr173ApiLocal);
               }
            }
         }
      } catch (IOException exc) {
         exc.printStackTrace();
      }

      String classpath = "";
      if (classpathJars.size() > 0) {
         classpath = " -cp \"";
         String separator = "";
         for (String localJarName : classpathJars) {
            classpath += (separator + localJarName);
            separator = File.pathSeparator;
         }
         classpath += "\"";
      }

      isJarsLoadedLocal = true;
      localJarsClassPath = classpath;

      return localJarsClassPath;
   }
Here, we use a static synchronized function. This function checks whether the classpath was already computed, and if so - returns the classpah immediately. Otherwise - it enumerates all available jars, and calls getLocalJarFilename function that creates a local temporary copy of the parameter jar URL (function shown below). Then, after all the jars that we were interested in have been downloaded, we create the final classpath (using File.pathSeparator). The code for getLocalJarFilename is:
   private static String getLocalJarFilename(URL remoteManifestFileName,
         Logger _logger) {
      _logger.info("Trying to create local version of "
            + remoteManifestFileName);
      // remove trailing
      String urlStrManifest = remoteManifestFileName.getFile();
      String urlStrJar = urlStrManifest.substring(0, urlStrManifest.length()
            - MANIFEST.length() - 2);
      _logger.info("Remote : " + urlStrJar);
      InputStream inputStreamJar = null;
      File tempJar;
      FileOutputStream fosJar = null;
      try {
         URL urlJar = new URL(urlStrJar);
         if (urlJar == null) {
            _logger
                  .info("No remote version (probably not running in JNLP)");
            return null;
         }

         inputStreamJar = urlJar.openStream();
         String strippedName = urlStrJar;
         int dotIndex = strippedName.lastIndexOf('.');
         if (dotIndex >= 0) {
            strippedName = strippedName.substring(0, dotIndex);
            strippedName = strippedName.replace("/", File.separator);
            strippedName = strippedName.replace("\\", File.separator);
            int slashIndex = strippedName.lastIndexOf(File.separator);
            if (slashIndex >= 0) {
               strippedName = strippedName.substring(slashIndex + 1);
            }
         }
         tempJar = File.createTempFile(strippedName, ".jar");
         _logger.info("Created temporary file '" + tempJar.getAbsolutePath()
               + "'");
         tempJar.deleteOnExit();
         fosJar = new FileOutputStream(tempJar);
         byte[] ba = new byte[1024];
         int bytesWritten = 0;
         while (true) {
            int bytesRead = inputStreamJar.read(ba);
            if (bytesRead < 0) {
               break;
            }
            fosJar.write(ba, 0, bytesRead);
            bytesWritten += bytesRead;
         }
         _logger.info("Written " + bytesWritten + " bytes to local file");
         return tempJar.getAbsolutePath();
      } catch (Exception ioe) {
         System.out.println(ioe.getMessage());
         ioe.printStackTrace();
      } finally {
         try {
            if (inputStreamJar != null) {
               inputStreamJar.close();
            }
         } catch (IOException ioe) {
         }
         try {
            if (fosJar != null) {
               fosJar.close();
            }
         } catch (IOException ioe) {
         }
      }
      return null;
   }
Important points - we create a temporary file using File.createTempFile (no need to guess the remote computer OS etc.), mark it with deleteOnExit() (can then reuse it as long as the application is running) and finally return its absolute path.

The code for compiling a set of Java source files now becomes:
ByteArrayOutputStream baosMessages = new ByteArrayOutputStream();
ByteArrayOutputStream baosWarnings = new ByteArrayOutputStream();
long time0 = System.currentTimeMillis();
StringBuffer bigAllNames = new StringBuffer();
for (SingleClassInfo currClassInfo : allClasses) {
   String currFilename = currClassInfo.getRootDirName()
+ File.separator + currClassInfo.getRelJavaFilename();
   this._logger.info("Adding " + currFilename + " to compilation");
   bigAllNames.append("\"" + currFilename + "\" ");
}
String warningSetting = new String("allDeprecation,"
      + "allJavadoc," + "assertIdentifier," + "charConcat,"
      + "conditionAssign," + "constructorName," + "deprecation,"
      + "emptyBlock," + "fieldHiding," + "finalBound,"
      + "finally," + "indirectStatic," + "intfNonInherited,"
      + "javadoc," + "localHiding," + "maskedCatchBlocks,"
      + "noEffectAssign," + "pkgDefaultMethod," + "serial,"
      + "semicolon," + "specialParamHiding," + "staticReceiver,"
      + "syntheticAccess," + "unqualifiedField,"
      + "unnecessaryElse," + "uselessTypeCheck," + "unsafe,"
      + "unusedArgument," + "unusedImport," + "unusedLocal,"
      + "unusedPrivate," + "unusedThrown");

// Create classpath. For compiling XJC-generated files we need two
// jar files, jaxb-api.jar and jsr173_api.jar. When Milano is run
// from the local computer, these jars should be on classpath
// (explicitly or on PATH varibable). However, when our application
// runs from Java Web Start, these jars are not available by their
// names. Here, we will try to fetch them as resources and create
// temporary files on the local machine.
String classpath = Workspace.getLocalClassPath(this._logger);
this._logger.info("Classpath is " + classpath);

// Start compilation using jdtcore
boolean compilationStatus = org.eclipse.jdt.internal.compiler.batch.Main
      .compile("-1.5 -source 1.5 -warn:" + warningSetting + " "
   + classpath + " " + bigAllNames.toString(),
   new PrintWriter(baosMessages), new PrintWriter(
baosWarnings));

// Parse the messages and the warnings.
long time1 = System.currentTimeMillis();
BufferedReader br = new BufferedReader(new InputStreamReader(
      new ByteArrayInputStream(baosMessages.toByteArray())));
while (true) {
   String str = br.readLine();
   if (str == null) {
      break;
   }
   this._logger.info("/javac/ " + str);
   compilerMessages.addLast(str);
}
BufferedReader brWarnings = new BufferedReader(
      new InputStreamReader(new ByteArrayInputStream(baosWarnings
   .toByteArray())));
LinkedList<String> warnings = new LinkedList<String>();
while (true) {
   String str = brWarnings.readLine();
   if (str == null) {
      break;
   }
   this._logger.warning("/javac/ " + str);
   compilerMessages.addLast(str);
   warnings.addLast(str);
}
this._logger.info((time1 - time0) + " milliseconds, "
      + (compilationStatus ? "success" : "failure"));
return compilationStatus;

Bookmark blog post: del.icio.us del.icio.us Digg Digg DZone DZone Furl Furl Reddit Reddit
Comments
Comments are listed in date ascending order (oldest first) | Post Comment

  • The 'tools.jar' technique is actually completely broken in practice outside a few Sun-derived implementations, as one can see in the various complex 'where is the hidden compiler' startup scripts in different projects.

    Thank you for writing up a much better, portable way to get the job done.

    cheers,
    dalibor topic

    Posted by: robilad on May 29, 2005 at 01:50 PM

  • Depending on what you trying to accomplish, adding a language similar to Java (BeanShell, Janino, Dynamic JavaNice, GJ, Pizza) may be preferable. Of course there is no shortage of other languages for the JVM.

    Posted by: coxcu on May 31, 2005 at 05:52 AM

  • coxcu,
    I am not able to see how the list you gave is relevant to running JDK 5.0 compliant compiler in you application. Not to mention the fact that half of your list was last updated in 2002.

    Posted by: kirillcool on May 31, 2005 at 06:15 AM

  • If compiling the full language supported by the JDK 5.0 javac compiler is a requirement, then none of the things I mentioned will be solutions. That's why I started with the "Depending on what you trying to accomplish" clause. I think that a Java compiler is such a generally useful thing that the JRE should include one and a standard API for accessing it.

    There are many situations, however, where being able to compile a code snipet is all you need.

    Posted by: coxcu on May 31, 2005 at 07:44 AM

  • coxcu,
    Among the list of features scheduled for Mustang i saw the compiler API in the core JVM. This list, however, is not final.

    Posted by: kirillcool on May 31, 2005 at 08:48 AM

  • Instead of using org.eclipse.jdt.core_3.1.0.jar, in 3.2 or 3.3, a jar containing only the batch compiler is available directly from the Eclipse download page. It is called ecj.jar. Its size is around 1Mb (1.5M in 3.3).
    org.eclipse.jdt.core_3.1.0.jar contains much more than just the batch compiler.
    Hope this help,

    Olivier

    Posted by: olivier_thomann on May 04, 2007 at 07:02 AM





Powered by
Movable Type 3.01D
 Feed java.net RSS Feeds