Skip to main content

Peering into the Pit of Jar Hell: How Symlinks to Jars Can Make You Run Screaming

Posted by garysweaver on May 22, 2009 at 10:00 AM PDT

Today I glanced into /usr/share/java in a CentOS release 5.2 (Final) server with Tomcat and Java installed via RPM. It literally burned my eyes, and hopefully you can see why.

My eyes first caught this:

libgcj-4.1.2.jar -> libgcj-4.1.1.jar
libgcj-tools-4.1.2.jar -> libgcj-tools-4.1.1.jar

Owww!!! It burns! Regardless of how trivial a small change in a version of a jar might be in one case for one version of an application, since this is a shared area for jars, you don't know what some other application would expect out of that jar. And if the person trying to track down an issue thinks they are using one version of a jar, but it is really pointed at a different version... Aaaaarghh! Support issue that take 5 months to determine coming right up, says the evil one.

Next up:

xerces-j2.jar -> xerces-j2-2.7.1.jar

Jar names can just be all over the place. It isn't uncommon at all to see a jar without a version name associated with an app. But here in a shared area for jars, to have a symlink with a generic non-versioned name pointing at one version of a jar that could be changed to point somewhere else is nearly as evil. Once that swap is made- how do you know which version of the jar it was supposed to have been using? Reinstall/redownload it? Was that really the version? Who knows?

How about something that is just wrong (yes this was actually in there):

wsdl4j.jar -> qname-1.5.2.jar

Even if the two jars have equivalent classes in the same package, unless it was really just a jar rename, you really don't know what classes someone else using the jar expect to be in there. Just because it works in one application, doesn't mean it will work for all. I'd say this one gets the 5 pitchfork seal of DISapproval.


servletapi5.jar -> tomcat5-servlet-2.4-api-5.5.23.jar

Be very wary of symlinks to jars that have generic names but point to a non-generic implementation. This is very likely also a pitchfork rapping at the door. If you are going to have symlinks, they better darn well be specifically named for the name and version of the jar it is!

Ok. I'm sorry if I've ruffled any feathers on a Friday, and I didn't mean to scare the rest of you so much. But just know that there is a Jar Hell encrusted with symlinks out there. And it might just be right at your backdoor.

P.S.- For those that just can't get enough, I'll leave you with the full list. I'm sure you'll find others:

activation.jar -> classpathx-jaf-1.0.jar
ant.jar -> ant-1.6.5.jar
ant-launcher.jar -> ant-launcher-1.6.5.jar
antlr.jar -> antlr-2.7.6.jar
bcel.jar -> bcel-5.1.jar
catalina-ant5.jar -> catalina-ant-5.5.23.jar
classpathx-jaf.jar -> classpathx-jaf-1.0.jar
classpathx-mail-1.3.1-monolithic.jar -> classpathx-mail-1.3.1-monolithic-1.1.1.jar
commons-beanutils-1.7.0.jar -> jakarta-commons-beanutils-1.7.0.jar
commons-beanutils-bean-collections-1.7.0.jar -> jakarta-commons-beanutils-bean-collections-1.7.0.jar
commons-beanutils-bean-collections.jar -> commons-beanutils-bean-collections-1.7.0.jar
commons-beanutils-core-1.7.0.jar -> jakarta-commons-beanutils-core-1.7.0.jar
commons-beanutils-core.jar -> commons-beanutils-core-1.7.0.jar
commons-beanutils.jar -> commons-beanutils-1.7.0.jar
commons-collections-3.2.jar -> jakarta-commons-collections-3.2.jar
commons-collections.jar -> commons-collections-3.2.jar
commons-daemon-1.0.1.jar -> jakarta-commons-daemon-1.0.1.jar
commons-daemon.jar -> commons-daemon-1.0.1.jar
commons-dbcp-1.2.1.jar -> jakarta-commons-dbcp-1.2.1.jar
commons-dbcp.jar -> commons-dbcp-1.2.1.jar
commons-digester-1.7.jar -> jakarta-commons-digester-1.7.jar
commons-digester-1.7-rss.jar -> jakarta-commons-digester-1.7-rss.jar
commons-digester.jar -> commons-digester-1.7.jar
commons-digester-rss.jar -> commons-digester-1.7-rss.jar
commons-discovery-0.3.jar -> jakarta-commons-discovery-0.3.jar
commons-discovery.jar -> commons-discovery-0.3.jar
commons-el-1.0.jar -> jakarta-commons-el-1.0.jar
commons-el.jar -> commons-el-1.0.jar
commons-fileupload-1.0.jar -> jakarta-commons-fileupload-1.0.jar
commons-fileupload.jar -> commons-fileupload-1.0.jar
commons-httpclient-3.0.jar -> jakarta-commons-httpclient-3.0.jar
commons-httpclient3.jar -> commons-httpclient.jar
commons-httpclient.jar -> commons-httpclient-3.0.jar
commons-launcher-0.9.jar -> jakarta-commons-launcher-0.9.jar
commons-launcher.jar -> commons-launcher-0.9.jar
commons-logging-1.0.4.jar -> jakarta-commons-logging-1.0.4.jar
commons-logging-api-1.0.4.jar -> jakarta-commons-logging-api-1.0.4.jar
commons-logging-api.jar -> commons-logging-api-1.0.4.jar
commons-logging.jar -> commons-logging-1.0.4.jar
commons-modeler-1.1.jar -> jakarta-commons-modeler-1.1.jar
commons-modeler.jar -> commons-modeler-1.1.jar
commons-pool-1.3.jar -> jakarta-commons-pool-1.3.jar
commons-pool.jar -> commons-pool-1.3.jar
com-sun-javadoc.jar -> com-sun-javadoc-0.7.7.jar
com-sun-tools-doclets-Taglet.jar -> com-sun-tools-doclets-Taglet-0.7.7.jar
dom3-xerces-j2-2.7.1.jar -> xerces-j2-2.7.1.jar
dom3-xerces-j2.jar -> dom3-xerces-j2-2.7.1.jar
eclipse-ecj.jar -> /usr/share/eclipse/plugins/org.eclipse.jdt.core_3.2.1.v_677_R32x.jar
ejb.jar -> geronimo/spec-ejb-2.1.jar
gnu-classpath-tools-gjdoc.jar -> gnu-classpath-tools-gjdoc-0.7.7.jar
hibernate_jdbc_cache.jar -> /etc/alternatives/hibernate_jdbc_cache
j2ee-connector.jar -> geronimo/spec-j2ee-connector-1.5.jar
j2ee-deployment.jar -> geronimo/spec-j2ee-deployment-1.1.jar
j2ee-management.jar -> geronimo/spec-j2ee-management-1.0.jar
jacc.jar -> geronimo/spec-j2ee-jacc-1.0.jar
jaf-1.0.2.jar -> classpathx-jaf-1.0.jar
jaf.jar -> /etc/alternatives/jaf
jakarta-commons-beanutils-bean-collections.jar -> jakarta-commons-beanutils-bean-collections-1.7.0.jar
jakarta-commons-beanutils-core.jar -> jakarta-commons-beanutils-core-1.7.0.jar
jakarta-commons-beanutils.jar -> jakarta-commons-beanutils-1.7.0.jar
jakarta-commons-collections.jar -> jakarta-commons-collections-3.2.jar
jakarta-commons-daemon.jar -> jakarta-commons-daemon-1.0.1.jar
jakarta-commons-dbcp.jar -> jakarta-commons-dbcp-1.2.1.jar
jakarta-commons-digester.jar -> jakarta-commons-digester-1.7.jar
jakarta-commons-digester-rss.jar -> jakarta-commons-digester-1.7-rss.jar
jakarta-commons-discovery.jar -> jakarta-commons-discovery-0.3.jar
jakarta-commons-el.jar -> jakarta-commons-el-1.0.jar
jakarta-commons-fileupload.jar -> jakarta-commons-fileupload-1.0.jar
jakarta-commons-httpclient.jar -> jakarta-commons-httpclient-3.0.jar
jakarta-commons-launcher.jar -> jakarta-commons-launcher-0.9.jar
jakarta-commons-logging-api.jar -> jakarta-commons-logging-api-1.0.4.jar
jakarta-commons-logging.jar -> jakarta-commons-logging-1.0.4.jar
jakarta-commons-modeler.jar -> jakarta-commons-modeler-1.1.jar
jakarta-commons-pool.jar -> jakarta-commons-pool-1.3.jar
jasper5-compiler.jar -> jasper5-compiler-5.5.23.jar
jasper5-runtime.jar -> jasper5-runtime-5.5.23.jar
javamail.jar -> /etc/alternatives/javamail
jaxp_parser_impl.jar -> /etc/alternatives/jaxp_parser_impl
jaxp_transform_impl.jar -> /etc/alternatives/jaxp_transform_impl
jdtcore.jar -> /usr/share/java/eclipse-ecj.jar
jms.jar -> geronimo/spec-jms-1.1.jar
jmxri.jar -> /etc/alternatives/jmxri
jspapi.jar -> tomcat5-jsp-2.0-api-5.5.23.jar
jsp.jar -> /etc/alternatives/jsp
jta.jar -> geronimo/spec-jta-1.0.1B.jar
ldapbeans.jar -> ldapbeans-4.18.jar
ldapfilt.jar -> ldapfilt-4.18.jar
ldapjdk.jar -> ldapjdk-4.18.jar
ldapsp.jar -> ldapsp-4.18.jar
libgcj-4.1.2.jar -> libgcj-4.1.1.jar
libgcj-tools-4.1.2.jar -> libgcj-tools-4.1.1.jar
log4j.jar -> log4j-1.2.13.jar
regexp.jar -> regexp-1.4.jar
servletapi5.jar -> tomcat5-servlet-2.4-api-5.5.23.jar
servlet.jar -> /etc/alternatives/servlet
tomcat5-jsp-2.0-api.jar -> tomcat5-jsp-2.0-api-5.5.23.jar
tomcat5-servlet-2.4-api.jar -> tomcat5-servlet-2.4-api-5.5.23.jar
wsdl4j.jar -> qname-1.5.2.jar
xalan-j2.jar -> xalan-j2-2.7.0.jar
xalan-j2-serializer.jar -> xalan-j2-serializer-2.7.0.jar
xerces-j2.jar -> xerces-j2-2.7.1.jar
xml-commons-apis.jar -> xml-commons-apis-1.3.02.jar
xml-commons-resolver.jar -> xml-commons-resolver-1.1.jar

See the tomcat-users mailing list thread for more comments.

Related Topics >>


Can I add a plea for the creators of jar files to include the version within the jar file manifest --- a standard form for doing this has existed since Java 1.2 (if I recall correctly). This makes it easier to identify the real version being used at runtime regardless of any fiddling with file names. Unfortunately while Maven has an option to do this, I understand that it is off by default.

Comments from Christopher Schultz: "Oh, this is gonna be great ;) > xerces-j2.jar -> xerces-j2-2.7.1.jar - It's even more obnoxious to link the 4.1.2 version to the 4.1.1 JAR since you'd think you were getting 4.1.2 but you're really getting 4.1.1. Shared libraries do this all the time, but they do it in a sane way: -> -> (this is the real one) So, basically, clients can either load a specific version ( or the latest in a major version (1) and they always get what's appropriate. They can also say "latest" by loading directly. This is great for shared libraries and /might/ be great for JAR files, but the above does not look sane to me. You wouldn't link to :( servletapi5.jar -> tomcat5-servlet-2.4-api-5.5.23.jar - The API JAR should not be stuck to a specific version of Tomcat: this is a generic JAR. I suspect this packaging reflects an ignorance of the relationship between the Servlet API and Tomcat. The servlet 2.4 API is the servlet 2.4 API, no matter who is using it. The version used for 5.5.23 is (or at least should be!) the same as is used for 5.5.1 and 5.5.27. "

Not to leave out dissenting opinion, here is Martin Gainty: (please see tomcat-users list archive for full post " you can see wsdl4j functions are empty stubs that call to qname for real work (the justification for symlink alias of wsdl4j.jar -> qname-1.5.2.jar is validated)" and my rebuttal: "Thanks much for the time you spent on the explanation. However (and hopefully I'm being brief also)- one of issues in doing this is that wsdl4j.jar could (in-general) be any version of wsdl4j not necessarily something that just happens to be populated with one or more classes that do nothing more than have methods that just then call classes in (version specific, because method signatures/classes/packages could change in diff versions) qname-1.5.2.jar. (This is - if that is what you are saying- I do not know what version of wsdl4j it is here, nor have I looked at the source, since I don't know what version it is from the name). The impression I get from looking at the mess of symlinks is that people are assuming (like vendors of Windows products that contributed to Microsoft's DLL hell starting mostly in Win95) that playing around with filenames and versions is perfectly acceptable if it gets the job done (for reducing space they take up in an attempt to share files, or in this case possible reducing the stack level by bypassing methods in an interface jar completely). But in fact, when this is done the only substantial good it does that is not outweighed by negatives is that RedHat will end up selling more support licenses for people that get fed up with RPMs on CentOS/Fedora not working properly (after all, they make money off of support, right?). That maybe a fatalistic viewpoint on my part, and I don't mean to start a firestorm, but basically (in this case) unless you were to have a directory that contained a bunch of jars where each filename were to have a version that actually corresponds to the well-known version of that specific jar, then I think you are really asking for trouble."

Comments from Ken Bowen (on tomcat-users list): "Probably the most common question asked on this list is: 'Are you using one of those @#$%$#$ Tomcats from a third party distribution?' The follow-up is always: 'You'll have to get help from the people creating that distribution.' BTW: On my own CentOS box, I simply ignore the distribution, and use the download from Apache."

Some comments from Chuck Caldarale: "Yup - which is why many of us strongly recommend not to use the 3rd-party repackaged junk, but instead get a real Tomcat download. Unfortunately, the Tomcat committers have no control over what the repackagers do, and the repackagers seem oblivious to the damage done and trouble caused." "Completely concur - but it's not done by Apache Tomcat, but rather whoever decided to repackage it in an anti-social manner. You can lodge your complaints to CentOS, and then stick with running a real Tomcat (and JVM)."

The attempted justification for at least some of this, I'm told, is the backporting of patches to provide compatibility with applications that might not be aware of the backporting of patches. That doesn't answer the points above though. If the jar isn't what it says it is, and symlinks don't change that. Can anyone provide a convincing argument for this mess?