Osvaldo Pinali Doederlein's Blog

New Java 7 language features should be backwards compatible

Right NOW, I am just starting to use J2SE 5.0's features, like generic types or java.util.concurrent, in my bigger projects. The reason is that large corps are slow, and they use application servers from even slower vendors... As a project architect, I'd love to send an email to my clients today, informing that our projects are moving to Java SE 6 and Glassfish V2 (or other up-to-date server) next week - we'll even do that migration for free!! because my life will become much better afterwards... but no, this is obviously not an option.

So that's the real world: I'm just starting to use Java 5. Java 6 is a distant dream - IBM's JDK 6 just went gold after an year-long beta, but there is no WebSphere update yet to include the new JDK, I guess I'll have to wait another 1-2 years until this update is fully complete, supported, tested, endorsed, and available in my clients' production servers. So when it comes to Java 7, well, I think I'll be happy if I can use that in large-scale, mission-critical applications (with IBM gear anyway), before I retire...

The real problem is that Java 5's language features were implemented with too many ties to the J2SE 5.0 platform: -source 5 will only accept -target 5. There's an unsupported javac switch that will allow some generics to target 1.4. But javac could do much better, as proven by third-party tools like RetroWeaver and JBoss Retro. (Similar support from javac would be much simpler than in those tools; javac could generate alternative code for older JVMs straight away, instead of generating 5.0-only bytecode that needs to pass through complex bytecode transformations to be replaced by alterenative code.) But it's also hopeles to use these tools in the environments I deal with. These translators have two severe problems:

1) They are not official tools. If I propose to use these things, my client's first question will be "is this supported by [insert large app server vendor here]?".
2) They make the edit/compile/deploy/debug cycle even more complex. I have systems with several dozen EARs and over a hundred projects, and I depend on the productivity of IDEs that update deployment into servers, automatically and quickly, and these IDEs have no flexibility to add a bytecode post-processor anywhere (but correct me if I am wrong, it's been some time since I last checked these tools).

So, my #1 wish for Java SE 7, whatever is the set of new language features that we decide to implement, is: make a best effort to support these features with -target 5, or at the very least -target 6. All features proposed to date are in the category of syntax sugar that's easily compilable into bytecode that current JVMs can load. Of course there's always some corner casees, like J2SE 5.0's enums which required new serialization support in the guts of the Java core; or in the same release, generics requiring new reflection APIs. So even if the proposals for Java SE 7 require special runtime support, there are several workarounds:

- Generating different code for older JVMs, even if it's slower (good solution when the runtime support is only an optimization).
- Backporting the new runtime support to the older JVMs, but making it private - for example, repackaging new APIs into sun.* packages, and translating calls to these APIs.
- Proving compatibility jars that can be dropped in older JREs' lib/endorsed. BTW, this could also be a great solution for critical new core APIs. See the backport-util-concurrent project for a good example - but it shares the same problems of being a non-official, non-supported package (although it's easier to sneak into projects: just drop the jar in your EAR's /lib and don't tell anyone...).

Comments

• Hi Osvaldo, i guess we all suffer from this same problem. Here at Globo.com, most of the code i work with runs on JDK 1.4, and there's still a big application running on JDK 1.3 that we hope to migrate to JDK 1.5 this year.

  In our case, the problem is tougher because the application running on JDK 1.3 serves many client applications via EJB 2.1 over JBoss RMI, and this certainly made the migration much tougher due to the high coupling.

  Fortunately, it seems likely that we are going to migrate this application from JBoss 3.2.7 on JDK 1.3 to JBoss 4.2.x on JDK 1.5. It's a huge frustration to be working mostly on Java 1.4 when Java 6 is avaiable and Java 7 is coming.

  With the growth of new technologies and the need to be more agile and inter-operate more, perhaps the lower coupling and the use of service oriented architectures can give us more flexibility and allow us to have more freedom to choose technologies.

  Best Regards,

  Bruno Luiz Pereira da Silva
  http://brunopereira.com.br

  Posted by: blpsilva on January 14, 2008 at 05:14 AM

• 
  blpsilva: This remembers me that application servers produced by giant, three-letter, blueish companies are not the only ones to lag behind the specs of JavaSE or JavaEE. JBoss's evolution has been disappointing in 2007, although I can understand this as an effect of the RedHat merger. It's had compatibility issues with Java6's JMX, I hope these are settled down now - in projects where appserver choice is mine, I won't touch a product that doesn't comply with the latest JavaEE spec and JavaSE runtime.

  
  Coupling is the right word here. Up to this day, the whole Java SE platform is tightly coupled: language syntax, APIs and runtime features all evolve in tandem. There are exceptions, like runtime enhancements (like new HotSpot optimizations) which are often back-ported to older JREs. My wish is for Sun to decouple the language syntax too. In this new world of a more open Java SE platform, embracing scripting and dynamic languages (even officially - Groovy is a JCP standard, Rhino is bundled, JRuby is sponsored by Sun...), it's weird that the Java language is the only one that is in disadvantage. You can use most alternative languages on several releases of the JVM, but the latest Java language can only be used in the latest JVM, which tends to make its adoption much more sluggish than necessary. If this doesn't change, perhaps we should just desist of improving the Java syntax because in a few years everybody will be using Groovy, Scala or Ruby...
  

  Posted by: opinali on January 14, 2008 at 06:58 AM

• I disagree. If you are still using Java 1.4 VM's then there is no way you are going to upgrade to Java VM 5 or 6 anytime soon. You're going to be stuck on it for sometime yet.

  However, the rest of us who can upgrade our applications to use newer JVM's can take advantage of faster, more secure, and more advanced JVM's. If your application is too fragile to upgrade it to Java 5 or 6, then leave it alone.

  But don't hinder the rest of us. Java 5's Generic became hindered because some people whined that they couldn't possible have the Java binaries incompatible with the old JVM. If you want the new features, use the new JVM. Don't hold the rest of us back in order to save yourself some work in testing and upgrading.

  Posted by: tegbains on January 14, 2008 at 11:02 AM

• What you're asking for isn't backward compatibility, it is forward compatibility. See http://en.wikipedia.org/wiki/Forward_compatibility
  

  Posted by: gafter on January 14, 2008 at 11:46 AM

• Neal, thanks for the input - but I'm not sure which is the better term to use here. As defined by wikipedia's article, "forward compatibility" would be a property of the older runtime - e.g., a Java 5 VM being able to run programs in the Java 7 syntax. I'm not asking Sun to do exactly that, e.g. issuing patches of old VMs to handle newer programs, or rearchitecting the VM to handle classes from future releases with arbitrary new features (the article mentions a web browser that ignores unknown tags - in a JVM it's much harder to imagine something similar: dropping new bytecodes or references to inexisting APIs won't work). What I want is that the newer compiler produces code that can run on older runtimes... it's the new javac that must be made [something]wards-compatible, not the older JVM. You should notice that the tools that compile Java 5 syntax into older runtimes are called Retroweaver and JBoss Retro, and not Forwardweaver and JBoss Forward... so my interpretation of this terminology may be wrong, but it's very popular. ;-)

  Posted by: opinali on January 14, 2008 at 12:32 PM

• 
  tegbains: It seems you didn't understand what I said. The situation is that I am stuck on old VMs, but I want to use new language features - only new syntax, not new APIs or anything that would demand a new JVM. There is NO reason why most new syntax enhancements couldn't be made available for older JVMs; the existence of retro-compiling tools is evidence of this -- if third-party hackers can provide this support, then certainly Sun could do it too, and certainly with much less work and more complete support.

  
  "But don't hinder the rest of us." - You are confused here... I didn't suggest anything that would hinder developers who can move to a newer JVM. And your comment about generics is wrong, please check Puzzling Through Erasure: answer section for details about the tradeoffs in Java 5's generics, remarkably the section "Migration Compatibility". To fix your comment, you'd say that "...some people whined that they couldn't possibly have the Java binaries [for legacy apps] incompatible with the NEW JVM" - this is what would happen if we'd chosen to go without erasure.

  
  "If your application is too fragile to upgrade it to Java 5 or 6 (...) in order to save yourself some work in testing" - now these parts make clear that you replied without even reading the full text. Now you're welcome to disagree with me, but please take the time to go beyond the title and summary.
  

  Posted by: opinali on January 14, 2008 at 12:58 PM

• "forward compatibility" isn't necessarily a feature of the older runtime - it can be accomplished by selecting implementation techniques for a new feature such that the older runtime doesn't need to change.

  Posted by: gafter on January 14, 2008 at 10:55 PM

• 
  Neal: I was just being stubborn ;-) I'll happily call it smorgasbord-compatible, or whatever... since you commented only about the terminology, I wonder what is your opinion about the actual subject of my blog. Do you think I'm just being a whiner, a litte-kid-in-candy-store who wants everything? IMHO, I present a very important problem and a reasonable RFE.

  
  The evolution of the Java language is too slow and conservative for many people. Of course, many others think the opposite, and the balance is difficult. But the decoupling of language X runtime would also help each developer, team, and company, to move in their own pace. In particular, solution providers like me who are on top of the latest technology could benefit from (at least some of) the latest Java improvements, even when clients have restrictions to deploy the latest runtimes. This is one aspect of the "migration compatibility" issue. And if my product/library/component has several clients, I could build separate binary distributions from the same source... the forward-compatible distro would maybe be bigger and slower (not using new bytecodes / classfile improvements etc), could depend on an endorsed jar, but it would work on older runtimes. This is also a real-world example, today I have some projects in this category and I have to resort to branching in the version control system, keeping separate sources for new and old runtimes. In many cases the branching would be required for other reasons - to allow use of new APIs - but with a forward-compatible syntax, at least the diff and the merge tracking would become MUCH simpler... and I could keep complaining, for example about important projects like Hibernate that would benefit massively from generics, but was not updated because of these issues (not even after it adopted Java5 for new functionality - annotations).

  
  When it comes to evolution, Java does one thing right - preserving backwards compatibility - but it fails everything else:

  
  - We can't link runtime stuff into each application - which avoids the whole issue in most languages. (There are static compilers for Java, but they are not very popular, and the dynamic nature of Java and the Java EE deployment model locks their use to niches.)
  - We don't have a good versioning / packaging system that allows side-by-side execution of apps or components that depend on different versions of the same dependencies. JSR-277 will provide some of this capability, but not everything, I think.
  - New APIs are still being defined in java.* packages that cannot be offered as extension jars for older runtimes. Look at the backport-util-concurrent, it's a great forward-compatibility project but it still forces users to change at least the imports, because it must live in a different package.
  

  Posted by: opinali on January 15, 2008 at 06:54 AM

• 
  Osvaldo
  
  > Java does one thing right - preserving backwards compatibility"
  
  and that's a big deal.

  Anyway, new syntax implies new runtime support,
  by example assert (1.4) needs java.lang.AssertionError,
  enums needs java.lang.Enum, foreach requires
  java.lang.Iterable, etc.

  You mention retrowear, if you take a look, it comes with a runtime libs.

  Thus using a new syntax requires to use a new runtime.

  Rémi
  

  Posted by: forax on January 15, 2008 at 11:44 AM

• 
  Rémi: On backwards compatibility being a big deal, granted, I've used many older languages (from C++ all way back to Z80) in a previous life, and I know the pain. But in the evolution of development environments, yesterday's marvel is today's commodity. Right now I'm not incredibly enthusiastic about Java's compatibility model, for same reasons that I'm not enthusiastic for its garbage-collected heap, or portable bytecode, or typesystem... these features are so 90's :) and they are present in other modern languages/runtimes - sometimes in an arguably better form, e.g. static typesystems with powerful and convenient type inference. Java definitely needs to move forward (if you excuse the pun).

  
  And I am fully informed about the runtime dependencies of all Java language features, and the existence of runtime jars in retro-tools. See the last item in the end of the blog, and also the last item of my last answer to Neal. I'm fine with a "javac -source 7 -target 5" capability that demands a small endorsed jar for old JVMs. It's much, much better than the alternatives.

  
  It's also true that many dependencies are actually optional. For example, in enhanced for, Iterable is most often used just a marker interface, unless the argument's static type is Iterable. If you have, say, a List, javac can issue a dynamicinvoke on List.iterator(), and so on. Some cases are harder, but the retro-tools can manage almost everything, even without the benefit of being implemented by Sun. (Sun could add special magic here and there in the JVMs - even in micro-versions of older releases, in sun.* packages or activated by JVM switches - to benefit compatibility libs). I will make this proposal more concrete by further specifying its behavior: If you use -source X -target Y, where Y<X, then

  
  1) If you don't actually use any new feature of X, nothing different happens, javac just creates classfiles compatible with Y.
  2) If you use new features of X that can be fully translated at compile time, not requiring any runtime dependency, it's just s good.
  3) If you use new features that depend on a runtime dependency, javac (with a proper -X flag) issues a warning so you know that.
  4) If you use new features that can be supported on older runtimes, but not perfectly (e.g. enums with incompatible on-the-wire serialization), issue a stronger warning (active by default). And when the old runtime loads the class, it will also issue a warning stating that the application may have some problem because it is only 100% compliant on runtimes X or better.
  5) Finally, if you use some feature of X that cannot reasonably be supported on Y runtimes, javac issues a compilation error.

  
  With this behavior, I think it's very easy for the developer to keep track of forward compatibility. Conservative, risk-averse developers will avoid any features that fit cases 4 and 5. Those who can't impose an endorsed jar dependency on their clients will, additionally, avoid code that fits case 3.

  Posted by: opinali on January 15, 2008 at 02:36 PM

• 
  Osvaldo, yes backward compat. is yesterday's marvel,
  but we continue to pay a tax now.
  Don't misunderstand my comment, i'am happy to pay that tax,
  i just want to say that backward compat. doesn't come for free.
  It's clearly possible to create the kind of forward
  compat you describe, but it will come
  with a higher cost at least in term of man hours.

  I agree with you that instead of having retrowear, backports
  of lot of API with different package names, etc, WE (and not SUN)
  can create a special OpenJDK group, a forward compat. group,
  that coordinates all forward compat. innovatives and
  provides a seamless integration with the OpenJDK.

  Rémi
  

  Posted by: forax on January 16, 2008 at 02:44 AM

• Rémi, you have a good point about the OpenJDK community embracing the task of forward compatibility. I suppose the JBoss Retro project could already contribute some of their tech to OpenJDK, since it's all GPL. Unfortunately RetroWeaver is BSD, but maybe they could move to GPL. So we could start with a -source 5 -target 1.4 capabiity, and latter, in Java 7 era, add support for (e.g.) -source 7 -target 5.

  Posted by: opinali on January 16, 2008 at 01:08 PM

This work is licensed under a Creative Commons License.