The Rumor

Hey, I like a rumor as much as anybody (Hey, did you hear that Larry Ellison is acquiring Europe?). But when the rumor is clearly wrong and it involves a project that I'm actually working on, I feel obliged to correct the misinformation (as did Java 2D's Dmitri Trembovetski, who has responded to the relevant threads in the next paragraph.  Go Dmitri!).

There was a post on Microsoft Watch, Windows Vista: Aero Glass and Java Don't Mix, last week that claimed that Java does not work on Windows Vista.  OSNews also picked it up. Then Javalobby picked it up on a forum entitled "Running Java on Vista Disabled Aero/Glass UI Effects". Imagine my surprise when I received the Javalobby newsletter with the subject "Java and Vista Not Playing Well Together".

I especially love this quote on Microsoft Watch:

Sun Microsystems would do well to give a ring to one of the interop contacts at Microsoft that came out of the firms' historic make-nice agreement back in 2004, and figure out how to make Java apps first-class Vista citizens.

Thanks for the tip, Jason; I'll pick up that phone right now...

So: Is the rumor true? Did Sun actually forget about Vista ("Whoops!") completely?

The Truth

Nope. The rumor is not true; we actually work great on Vista. (The rumor about Larry, on the other hand...).  In fact, we have been tuned into this release and making Java work on it since it was named after a breed of cattle.  Between regular calls with Microsoft, interaction with their engineers when problems or questions arose, and regular testing and engineering during Vista's development, we have been building a rock-solid release of Java for Vista.

So how did this bizarre rumor originate? Well, older versions of Java do have problems on Vista, and that's what the original report was about; someone tried running some older version of Java on Vista and noted some problems.  But that's like saying that your favorite XBox game, Bloody Mess X, doesn't work on XBox360.  Of course it doesn't; the original game was written for a completely different system.  Why should you expect it to work out of the [x]box on this new platform?  Presumably, given the popularity of Bloody Mess X, the developers are working on a port of the game to the new console ("Even more blood!"), so you will still be able to run your favorite game on this new system.

The port is the key; the new system is different enough in its fundamentals that software for the older system would not just work, but had to be ported.  It would be great if your game just worked.  Just like it would be great if Java just happened to work on Vista with no changes. But it just didn't happen that way; the platforms are different enough that changes are necessary.

It's the same thing here; Vista is not just XP++; there are fundamentally new things about the system that makes older software break. Is all software broken?  Probably not.  But the more of the system an application uses, the more likely it is to run into issues where the system has changed, and need to react to those issues.  In our case, Java is not just a simple win32 GUI application; it is a runtime platform with deep rooted needs in the operating system, the networking stack, the security model, the graphics system, ... if any of these change significantly, then we need to change our software in reaction. And in the case of Vista, it has been an ongoing process of learning, testing, debugging, submitting bugs against Microsoft, fixing our bugs, re-testing, ....  And since Vista has been a moving platform during the Java SE 6 development process, we've been in this development cycle continually with every new drop of Vista (they are still releasing weekly builds for us to test; we just found a bug in RC1 that has since been fixed in the latest release we got yesterday).

But the extra hassles of debugging a moving platform does not mean that we are unaware of the issues and haven't actually been fixing them; Vista has been one of the highest priorities for our Java SE 6 work.  In fact, Java SE 6 has been working quite well on Vista for months.  Some of the more obvious bugs (like Java 2D's disabling the Aero Glass whizzy desktop management system) have been fixed and available in snapshot releases of Java SE 6 for many months. And as we fixed problems in Java SE 6 (our primary platform for Vista support), we have also been backporting the more important fixes to older releases so that we can provide updates to these releases for customers that require the older releases to work on Vista as well.

As pictures are worth between 999 and 1,001 words, here are a few to help you see where we are at.  These are screenshots of Java GUI applications running on the latest versions of Vista (two of the SwingSet2 demo, one of Azureus, and one of Swing-based NetBeans 6):

The Problems We Fixed

Since this is a tech blog (okay, apart from the occasional geeky humor tangents, like this, this, this, and this) and not a whitepaper, I get to dive into a little more detail than I would otherwise.  In a whitepaper, we might say something like "We work well on Vista" and then defer to a PR representative whom you can call to have them say the same thing (with a lot more words).  But I'm thinking that Java developers might actually be interested in some of the technical details that we had to deal with to make our stuff work on Vista.  Those looking for things to occupy their time can chase down the real details in our bug database, of course, but I'll hit some of the highlights here:

Aero Glass

The "Aero Glass" desktop is enabled by the Desktop Window Manager (DWM) of Vista.  This system runs the desktop graphics on top of Direct3D.  All "windows" on the desktop are actually texture maps which are composited together onto the screen.  This system enables various effects that you can see on the screen, including: translucent title bars, application previews in the task bar, and the new Alt-Tab and Cmd-Tab functionality for switching between applications.

Problem 1: DWM disabling

• Symptom: On older releases of Java (1.4.2, 1.5 prior to update 8, SE 6 snapshots before about April of 2006), running a Java application would "punt the DWM", or would effectively disable the Aero Glass effect.  When you launch Java, you will see a change in the desktop appearance, the window borders will turn opaque, you will lose some of the whizzy effects, and you will essentially get more of a XP look & feel for your application windows.
• Problem: Java 2D uses DirectX for creating the Swing back buffer (and other offscreen images) and for copying these images to the screen.  Part of our sanity-check that the DirectX system is functional is to verify whether we can "lock" the screen, which gives us the capability to write or read pixels directly from the screen (we actually no longer use this facility for onscreen pixels; locking the screen to do this ends up causing artifacts with GDI graphics like software cursors and fading menu effects).  Locking the screen is also the easiest way (unfortunately) to flush the current queue of graphics commands (something that is necessary when someone calls Toolkit.sync(), for example). Locking is also a good way to just sanity-check that things are functional with DirectX before we start depending on it.  But locking is also, unfortunately, the quickest way to make the DWM punt on Vista; because of the new composited-desktop approach, applications in DWM no longer have direct access to the screen itself.  If an application demands access (such as through this older DirectDraw interface), Vista will oblige the request, but only by first disabling DWM, and thus disabling Aero Glass.
• Solution:  We found less obvious ways of doing what we needed (sanity-checking DirectDraw as well as flushing the graphics queue) and no longer lock the screen.  This avoids the DWM punt and makes Java run much more seamlessly on Vista.

Problem 2: Graphics artifacts (black rectangles)

• Symptom: Once we had fixed the DWM-punting bug above, we unmasked the problem that Swing windows would sometimes be littered with black boxes or draw the content in the wrong place.  The problems were intermittent and depended somewhat on the particular graphics operations that were taking place (scrolling or moving JInternalFrames around seemed particularly problematic).
• Problem : Desktop Java ends up using both DirectX and GDI to draw content into a Java window.  We use GDI for basic windowing functions such as creating the window and other heavyweight components that Swing uses underneath.  We use DirectX to get some of the more advanced 2D capabilities that Swing benefits from, such as caching the back buffer in video memory for accelerated copies to the screen. This worked fine on Windows pre-Vista ; GDI would draw directly to the screen, Direct X would copy directly to the screen, and life was good.  But in the new composited-desktop world of Vista, this system breaks down.  GDI is now rendered in software into a GDI texture map, DirectX is rendered into a different texture map, and, like small kids crammed together into the back seat on a long car trip, they simply don't play well together. Our use of both rendering systems for the same windows simply causes mayhem in the Vista rendering system and artifacts on the screen. The response we got from Microsoft when we pointed out the issue was akin to "Ohhhhhh - you're doing that?  That'll never work". See Greg Schechter's blog on the topic for more details.
• Solution: The quickest solution was to simply disable our use of DirectX on Vista.  We are basically forced to use one or the other: GDI or DirectX.  We cannot avoid GDI because we need access to basic window system capabilities, so DirectX had to go. There is a more complete solution in the works (for Java SE 7) that will involve a much more complete implementation on top of Direct3D with a more involved and capable buffering system that will allow us to use DirectX in a much more Vista-friendly manner.  But until we have finished that work (how is it going, Dmitri?) we will be using a software-rendering approach on Vista.  The graphics performance geek in me is not overjoyed about the solution; of course I want to see as much hardware acceleration used as possible. All of the time. For everything. But in reality, typical Vista machines (which will be pretty beefy) running typical Swing applications (with basic, static GUIs) will run absolutely fine.  In fact, given our basic level of acceleration so far, many current systems with decent CPUs (not just the high-end ones) run typical Swing applications faster than they would with this level of acceleration; it all depends on the combination of system resources and application requirements.

Swing Native Look & Feel

Vista has a completely new look & feel.  Beyond the Aero Glass effects discussed above, there are simply new themes and new looks to the buttons, the windows, and all of the other normal desktop widgets.  But Vista did not simply tweak the XP look & feel resources; they implemented these new looks through a completely different mechanism.  And they added animation to some of these widgets.  Swing's old approach of using the XP look & feel resources had to be re-thought in the context of this new world.

Problem 1: New native look & feel

• Symptom: A Swing application using the native look & feel on older Java releases actually does look native on Vista; it just happens to look like a native Windows 2000 application.
• Problem: In Windows XP, Swing derived its native look & feel from resources on the system which XP itself used.  So, essentially, we were using the same core resources as XP itself to make our components look native.  However, there were some corner cases in which this did not work on XP.  Even more problematic, Vista uses a completely new system with new resource locations and formats.  So not only did we not look Vista-native on Vista; we didn't even look XP-native (because the XP resources no longer exist).
• Solution: In Java SE 6, the Swing native look & feel for both Windows and GTK was re-written to use the native platform rendering engine to draw Swing widgets.  Instead of our old approach of using the native resources and doing our own rendering, we actually call the native rendering system to draw the widgets for us (on Windows, this is done through the UXTheme API). This means that we are finally able to achieve true native fidelity for these components because, by definition, our components are drawn in the same way that the native components are.  Not only did this fix corner cases for our XP look & feel, we also made our native look & feel work successfully on Vista.  More information about this work can be found on Bino George's and my blogs. The fix has been available in Java SE 6 snapshots since as early as build 14 (for reference, we're now past build 100).  The work has also been backported to J2SE 1.5, and is available in the currently available 1.5 update.  There have been ongoing fixes to other minor issues in native look & feel since then (Vista draws some components in new ways, which we needed to incorporate into our implementation).

Problem 2: Native look & feel animation

• Symptom: Some Vista components animate between different states. For example, moving the mouse over a button will cause a cross fade between the default state and the mouseover state.  Meanwhile, Swing components (like pre-Vista Windows native desktop components) have always simply switched between these states.  So where you might see an animation on a component in a native Vista application, you would see a simple (non-animated) state change in Swing.
• Problem: There was no animation system on older operating systems; components simply changed state and would be drawn in the appropriate way for that new state. In order to handle this change in behavior, we had to implement new logic to change the appearance over time.
• Solution: Igor Kushnirsky implemented Swing animation support in build 97 of Java SE 6 (the latest snapshot release on the download site has these fixes).  There are still some cases not completely handled (for example: progress bars have an animating "sparkle" every second or so that we do not currently render), but things look good in general.  Further fixes are planned for Java SE 6 update releases as well as Java SE 7.

Deployment

One of the more major areas of change in Vista affected the deployment team.  Vista has made some significant changes in its security system, in an attempt to make it more difficult for malicious software to gain control on the machine's resources.  For example, the user is now prompted when an application needs access to a system resource such as the registry; they will get a dialog asking whether they approve this access, and asking them for the Administrator password if they are not logged in as the administrator already.  Internet Explorer 7 (IE7) takes this a step further and protects that entire process from accessing the raw system, so that even if an application inside the browser gains access to the system, it can only perform operations inside the very restricted sandbox that the browser offers.

These changes had wide-reaching implications for our various deployment pieces, including the Installer, Java Update, Java Plug-in, and Java Web Start.  Many of the issues were overlapping, so I'll just cover the general areas of problems that we had to overcome.

Problem 1: Administrative Privileges Required

There were many cases in which we need administrator privileges in order to perform a necessary deployment task.  This includes storing things to registry keys, writing to locations on disk, and executing processes.

• Symptom: Some installer/uninstaller situations would not work correctly, some things that were formerly unknown to the user would now be more visible (for example, Java Update would ask at every login whether you wanted to let it run), and some things would need to ask your permission to write to a location that would not normally be available to a process.
• Problem: Vista tries to lock down system access as much as possible.  Some processes (such as installers) need "elevated privileges" that require user permissions to run.  In some situations, we would run an installer as an exec'd process, which would fail because there was no opportunity to ask the user for appropriate permission.  Similarly, Java Update would need elevated privileges in order to check for an update; since update runs at every login, this would happen whenever the user logged in (which, frankly, gets pretty annoying).
• Solution: We figured out ways to run the installer without resorting to exec'ing processes, so those problems were fixed.  There is no easy answer to the Java Update problem, so we simply reduced the frequency at which Update runs.

Problem 2: IE7 Sandbox

IE7 on Vista is really clamping down on security holes, but restricting access to the filesystem from the browser process and everything associated with it.  This means that Java Plug-in, which runs as an ActiveX control within IE7, is wholly contained within that IE7 sandbox.  This has various implications, from the ability to save into arbitrary locations to the ability to read from and write to the normal deployment installation directory (which Java Plug-in shares with Java Web Start).

• Symptom: Java Plug-in was unable to cache jarfiles and share other data with Java Web Start and the Java Control Panel. 
• Problem : Java Web Start and Java Control Panel run as standalone processes, and therefore have no trouble reading and writing to necessary places on disk (such as the usual deployment directory).  Java Plug-in, however, could not access these directories, and would thus end up writing to a new virtualized location on disk which could not be seen by these other applications.  We could no longer share the information between these very related applications. It was also not clear whether the virtualized directory used by Plug-in would actually be persistent; there was no guarantee that these virtualized locations were permanent.
• Solution: Later releases of Vista provide a persistent directory that all processes can reach.  The directory is within the sandboxed area that the IE7 browser can write to, so the overall filesystem is still protected, but as long as other non-IE7 processes (such as Java Web Start and the Java Control Panel) can reach this location, we have the sharing we need.  We chose this directory to house our deployment directory so that all of our Java deployment components could use and share it effectively.

Problem 2a: Vista and IE7 Sandboxes: The Unfixable

There is an issue worth noting here than cannot be fixed: IE7 on Vista will not let us access the file system outside the IE7 sandbox for applets.  This means that even "signed" applets have no permission to write to arbitrary locations on disk.  This behavior differs from the old behavior of applets on Windows, but given the restrictions of IE7, there is really no way around the problem.  Applications will need to adjust and write to locations that are reachable from their applications.  Note that if an applet simply needs to save information and access it later, this is completely possible; the information will reside in the sandboxed area of the browser.  But if they want to save information in a place that is accessible by other applications, that is trickier since applets cannot save outside of this area and the area may not be obvious to applications outside the browser.

There is another related issue also worth noting: Vista itself has proclaimed various directories off-limits for writing.  For example, it is no longer possible (or at least no longer trivial) to save files to the root directory (C:\) or to other system-level folders; non-administrator users are typically restricted to writing only within their home directories.  This change will affect not only applets, like the IE7 sandbox constraints noted above, but also standalone and Java Web Start applications.  This is obviously not a Java-specific problem, but is one that our users must deal with just like they will have to deal with it in other native applications; files must be saved in Vista-friendly locations.

Problem Summary

That's it for the techy details.  Hopefully you found it interesting; my purpose in detailing this stuff was not to bore or frighten you with the details, but rather to interest you in the relatively huge changes that Vista required from us, and to show why simply running out of the box on Vista was not really feasible for such a large and diverse platform as Java.  But, again, we did the work, we work well, and we encourage you take the time to run Java SE 6 on Vista to see how well it all works together for your applications.

The Plan

Many people reading this might be wondering about things beyond the bugs we fixed, for example: "When can I get a release of Java that works well on Vista?".  Well, here is the plan.

Java SE 6 is the Best Solution for Vista

First of all, you should note that the primary delivery of Java for Vista is Java SE 6; that release has received most of our focus during the Vista beta release timeframe, and it is where most of the fixes to the known problems currently reside.  We are just finishing up that release and it should be done and shipping by sometime next month.*

In the meantime, we encourage you to go to the Java SE 6 download site and get the latest snapshot for testing; the release is pretty close to final, so it is working very well at this point. In particular, all of the serious Vista problems have been fixed in this release for months, so it is a particularly good test vehicle for Java on Vista.

It is also worth mentioning that we are still aggressively pursuing OEM deals. We have distribution agreements with over 20 PC manufacturers, including all the top 10. They have all been helping us test Java SE 6 as they prepare their new lines of Vista-based systems for shipment, so that Java SE 6 will just be there on any of their new systems running Vista.  By the way; if you have a local PC maker who you prefer over the big guys (or if that is your business), let them know they can sign up to bundle Sun's Java Runtime on their systems, just like Dell, HP, Sony, and all the others do just by going to http://java.com/pcoem.

J2SE 1.5 Will Also Work

Many of the Vista fixes have already been, or will soon be, back-ported to J2SE 1.5.  However, don't look for everything to work exactly the same; our primary focus was to make Java SE 6 the main release vehicle for Vista. In an ideal world with infinite engineers working on everything, perhaps we would make sure everything worked the same everywhere.  But here in the Real World, we have to pick our battles, and the battle we picked this time was to make the new release the primary one for Vista.  Besides the effort of simply moving code back and forth between releases, there are architectural differences between some areas of the current and older releases that makes porting efforts more difficult and time-intensive, so we carefully choose what to back-port, based on a cost-benefit tradeoff. 

J2SE 1.5 should work fine, but there may be some nuances that may not be as perfect.  For example, the basic Swing native look & feel work has been back-ported, so that Swing native look & feel applications look like native applications on Vista; however, some additional Vista-specific fixes in this area (such as component animation) may not be back-ported, so the fidelity may not be as close as that in Java SE 6.

Some of the Vista fixes are already in the current release of J2SE 1.5 (as of this writing, update 9 is available on the java.sun.com download site ).  For example, the "Aero Glass" issues described above were addressed way back in update 8.  But the full gamut of Vista work that we feel is necessary for J2SE 1.5 should be available in update 11, which we hope to release around January of 2007.*

J2SE 1.4.2 Will Basically Work...

We plan to also backport necessary fixes to 1.4.2.  However, the same caveat goes for 1.4.2 that I detailed for 1.5, but even more so; we plan to fix only the necessary items to make 1.4.2 work on Vista, not to spend much time improving upon the basics.  A good example is native look & feel.  The rearchitecture of Swing's native look & feel engine involves a lot of code; we do not feel that it is worth the time and risk porting this code to this older platform with Java SE 6 being the primary release vehicle.  We see 1.4.2 as being functional, usable, and perfect for situations where a customer is absolutely locked into that particular release for now.  But we encourage developers and customers to migrate to a more full-feature Vista release soon.

We plan to ship this Vista-enabled update of 1.4.2 sometime following the Vista release of J2SE 1.5. I hope this will be in the first quarter of 2007*, but details are still being nailed down.  Our primary focus now is on finishing Java SE 6 and making it a stellar release, then jumping immediately onto the J2SE 1.5 update to make sure that it has the Vista support it needs.  Then when we are satisfied that these vehicles are robust, we can address Vista on J2SE 1.4.2.

* Don't quote me too exactly on dates.  First of all, I'm an engineer.  You should really talk to a business person that does this kind of stuff for a living if you want official dates and commitments.  These dates are pretty solid based on our schedules and projections, but you're a software person (aren't you?  how'd you get to this blog otherwise?) and understand that software release dates, like flies on your lunch, can move quickly as you get close to them.