Chet Haase's Blog

May 2007 Archives

Existence Proof

I may joke about never finishing the book, but as Romain shows on his blog, we're now seen what some of the book will probably look like in final form. Now, to finish editing the proofs (should be done by mid-June) and get the darn thing printed (supposedly happening in July)...

And yes, we will be publishing the demo code from the book soon. We just have to get the website written and the projects uploaded and organized. We'll be sure to announce it when it's all ready.

Media Frenzy

Java Media Components

or

Making Java a Player

The question comes up regularly, and on a more frequent and random basis lately, "What is Sun doing about media support in Java?" And sometimes it is not phrased quite so politely.

Video used to be a feature that we lacked:

Now, it seems sometimes that it is the feature that we lack:

In the current world of YouTube, Flash, and iTunes, video has gone from a niche capability to a checkbox item that all platforms must support.

So what are we doing about it, besides pointing out the obvious issue that it isn't there now?

If you attended the Java Technology Desktop overview session at JavaOne this year, you learned, along 500+ of your closest friends in the room, that we're working on a new project that we call Java Media Components(JMC). This new feature, hopefully shipping in Java SE 7, is intended to support basic playback support for Java applications. JMC is also, eventually, intended to address capture and streaming capabilities.

Playing Fair

There are actually two levels of playback that we would like to support:

• Native players
• Java player

Going Native

The native players on most platforms have substantial support for multiple video formats. It would not make sense (read: it would cost serious $$$) for Java to implement players for some of these existing formats. So why not leverage the native platform capabilities instead?

The idea is to expose a simple API for playback control (e.g., play, stop, and rewind), and to allow a Java application to thus control playback of specified content through whatever was available on the native system. For example, playback of an AVI file on Windows might launch Windows Media Player. Moreover, the native player would be exposed through a GUI component that you could integrate into the rest of your application's interface. There would also, depending on the underlying native support, be options to either expose the native controls for the player (e.g., the Play button), or to allow the developer to skin their own controls around the player component.

This native-wrapper functionality is the easiest part of the whole JMC project. The main work is in the simple API design and the glue code to the various players on different platforms. In fact, we already have prototypes of this working for several players, including Windows Media Player and Flash.

Curious readers familiar with lightweight/heavyweight mixing issues in Swing applications might wonder at this point how we will deal with a native player component in a Swing GUI. Good question, thanks for asking. This issue is one of the reasons that we also plan to support mixed heavyweight/lightweight components in Java SE 7. In case I don't write a blog on this feature soon, here's the quick scoop: We can enable this through clipping the heavyweight components against the shapes of the overlaying lightweight components. There are some cases that do not yet work with this mechanism, such as shaped and translucent lightweights, but the system works pretty well in general for other cases.

Playing Nice with Java

Supporting common formats through the native player mechanism will solve a key part of the problem. The ability for a Java application to easily play video that resides on a user's local system through a standard local player is an important case that will be handled through this simple mechanism.

But native player support is not a complete solution. For one thing, if a developer wants a specific video to play through the application, they will have to ensure that the video is available in whatever formats will be supported on all of their users' native platforms. For example, a Quicktime movie would not be playable on Windows unless Quicktime happened to be installed on that system. Similarly, an AVI file may not have much luck playing on Linux.

To solve this problem, we need one common format, and an accompanying codec (coder-decoder), so that any Java application can count on being able to play back any content in that format, regardless of the runtime platform.

This cross-platform format support is a much more involved issue, since it immediately gets into harder issues of:

• Licensing technology from existing player developers, and/or
• Creating or integrating new technology into the platform

We plan to tackle this part of the JMC project after the native players are well in hand. Meanwhile, we are trying to figure out the right solution (which formats, which technologies, etc.).

Playback versus Pixel Access

There is an important consideration in both the native and Java player solutions above: How much do players need to simply display their pixels on the screen, and how much do they need to actually expose the raw pixel data? To a great extent, simply playing on the screen is probably good enough. If an application just wants to play a video, then blasting it to the user's screen is fine. But there are subtle cases here that make the issue a bit more complicated.

For example, what if you wanted to do something more interesting with the video content, like play it on a lightweight Swing widget? Remember that I said that we plan to allow mixing of heavyweight and lightweight components, which should allow a JMC player to at least cooperate with Swing widgets. But that's not the same as a JMC component actually being a Swing component. For example, what if you want to use the video as the icon in a JButton? Or what if you want the video playing in a translucent component? Or what if you want to get really crazy and display video in a 3D object, via JOGL?

For any of these cases where you want to do anything with the pixels besides simply show them on the screen, you will need access to the pixel data from the video stream. Many of the native players do not offer this capability, but merely allow the controls of playing, rewinding, and stopping the video. Some native systems, such as DirectShow, do allow more flexibility, so we plan to use these approaches when we can. And the Java Player will certainly allow this capability. Since we will provide the decoder for the video through the Java player, we can have those pixels appear on the screen, or in an image, or wherever the application specifies.

What about Capture and Streaming?

Capturing video from an input device and streaming between Java clients are two areas that we would also like to support eventually. Having the Java player is actually a good initial step toward these capabilities, since we will be able to encode and decode into our cross-platform format. But we are taking baby steps here, and feel that the top priority is solid playback support. We would like to get rolling on that playback project first before we take on even more, and harder, issues.

What about Java Media Framework (JMF)?

JMF is an existing media library, provided as an optional package on top of Java SE. JMF is a very large and powerful media development platform that is capable of doing much more than I've discussed here. In particular, while it's possible to write a complete video editing solution in JMF, we are clearly going nowhere near that level of functionality with the JMC project, but rather focusing squarely on playback at first. Again, we are concentrating on attacking the pain points in our existing video support (or lack thereof), and we feel that playback is simply much more in demand than full-on video editing APIs.

One solution here might be to simply revamp our investment in and support of JMF itself, and make it the platform for future video endeavours. However, the level of effort that we would need to invest to bring JMF up to the current level of requirements, including providing codecs for modern formats, would be huge. Also, at about a 2 Megabyte download size now, we're not sure that simply adding in the current JMF to the core platform is the right approach. It seems better to solve the top priority of simple playback by a targeted solution that is possible with a much smaller footprint impact.

We may actually implement parts of the JMC stack on portions of JMF, which would give us the advantage of re-use of some of JMFs capabilities without taking on the entire library. Also, work on codecs for JMC could end up also benefitting the JMF library eventually in any case.

That's the Plan

Plans change, of course, but this writeup represents what we would like to see happen in Java SE 7. Does it sound reasonable to you? Would having simple playback functionality, through both native players and a new Java player, satisfy your craving for media in Java? And if we spin up a new project on SwingLabs soon, would you be interested in helping out in the design and implementation of these pieces?

Consumer JRE: Leaner, Meaner Java

Coming soon to a Java SE 6 update release near you

Recently, Thorsten Laux and I gave a talk at JavaOne 2007 about where we are and where we're going in Desktop Java. We covered the download numbers (awesome!), statistics on how many PCs have Java installed (excellent!), and highlighted various desktop applications that use our Desktop Java technologies (fantastic!). We then quickly covered some of the larger features that were implemented in the recent Java SE 6 release (which I won't go into here; check out the various blogs and articles on the topic for more on those features). We concluded with the following technical diagram:

Clearly, the combination of growing ubiquity, power of the platform, and features in the latest release means that we are done, right?

Well, perhaps not. There are still some outstanding issues that need to be addressed, as we can see in the following diagram:

The reality is, we have several outstanding issues with Java as a consumer desktop platform, which need to be fixed soon in order to make us competitive now and in the future.

The good news is that we are, in fact, aware of these issues. The better news is that we're working on the problems. The best news is that we are close to solutions and intend to delilver them as an update to the SE 6 release, in a release that we call the Consumer JRE.

What's In It?

The Consumer JRE consists of several important pieces of functionality, some of which are depicted in detail in the above diagram. One of the keys to getting this release out quickly as an update to an existing release is to only make changes that do not affect API. So, for example, we can add functionality to make startup faster without affecting the APIs or functionality that an application is using. But we cannot add a new animation API in an update release. Fortunately, this constraint is not too cumbersome since the major problems we are trying to solve now are below the level of API changes, and are thus completely suitable for this release.

Here are the main items that we are shooting for in this release:

Quickstarter: Radically reduce the startup time for Java applications and applets.

Java Kernel: Reduce the time-to-install-and-launch when the user needs to install the JRE in order to run an application.

Deployment Toolkit: Enable easy detection and installation of the JRE.

Installer Improvements: Improve the user experience of installation.

Windows Graphics Performance: Enable default graphics acceleration for simple and advanced 2D rendering.

Nimbus Look & Feel : Release a new cross-platform look & feel based on Synth.

Quickstarter

This is probably going to be the most popular item in the mix, making the launch of any Java application or applet much faster. This is one of the most serious holdups to further applet development and deployment today, as the launch of the first applet in a browser can take several seconds. Quickstarter will cut down the launch time significantly, vastly improving the first-launch experience for consumer Java applications.

There are actually 2 issues with Java application startup: warm start and cold start. We define warm start as the time that it takes for a Java application to start when you have recently run Java recently on your system. Cold start, on the other hand, refers to the time that it takes to launch the first Java application after a fresh reboot.

Warm start times are reasonable these days, due to plenty of ongoing work on improving performance, in addition to machines simply getting faster over Java's lifetime. A simple application or applet will take 1-2 seconds to start up, which is in the same ballpark as standard web pages and within an acceptable range for the user.

Cold start, on the other hand, continues to take an unreasonably large amount of time. It is not unusual to see an applet take 5-10 seconds, or even longer, to start. While such a delay might be acceptable for a large desktop application, such as an IDE that's going to run all day after cranking it up in the morning, startup times like this for applets are unacceptable and limit applet viability for lightweight consumer content.

The problem turns out to be at the operating system (OS) level. Don't misunderstand me: I'm not saying, "It's not our fault!, " as tempting as that route always is for any hard problem. Instead, I'm saying that the Java platform is running into some basic physical constraints at the OS and hardware level that we must work within. In particular, the files that make up the complete Java platform are, well, large. For example, a recent version of Java SE 6 that I have sports an rt.jar file of over 40 MB alone. If you add in the various other jarfiles, native libraries, and resource files that get touched at startup, regardless of any application-specific code, that's a lot of data that has to be read in.

At the OS level, this means that all of these megabytes (or, rather, tens of megabytes) have to be read from disk, which is a very (to use a technical hardware term) slow operation. Actually, it's the seek time of the disk that's the killer; reading large files sequentially is relatively fast, but seeking the bits that we actually need is not.  So even though we only need a small fraction of the data in these large files for any particular application, the fact that we're seeking all over within the files means that there is plenty of disk activity. No wonder it takes so long for Java to start up, since we are dependent upon the speed of such an inherently slow operation to begin with.

The reason that warm start is so much faster is that once some data has been read off of disk, the OS places it in the disk cache. The next time that memory is needed, the OS can retrieve it from the disk cache, which is a significantly (again, a technical term here) faster operation.

The fix, then, is for us to take advantage of the disk cache to make sure that the memory pages on disk that we must read at startup have already been loaded before we need them. How do we do this? We cannot magically pre-load the pages just prioir to launching; unfortunately, the VM currently lacks the ability to see into the future to detect when the user will be launching Java (we would love to have this feature in the future, but it is not yet present). But we can pre-load at some earlier time, such as Windows boot or login time. And we can keep the pages warm in the disk cache as machine and memory conditions allow.

Note that this approach is not the same as running a full Java VM. That approach would solve some of the same problems, but in a less efficient manner, locking up the memory in a less OS-friendly way. Our approach will work just with the disk cache itself, allowing the operating system to use the system memory and disk cache as it sees fit.

Java Kernel

The Java Kernel project addresses the time-to-launch problem for users that do not have the proper JRE installed. For example, if your application requires the user to hava Java SE 6 and they do not currently have it, then you require that they install that full release prior to launching the application. Given the size of the JRE, and depending on the amount of bandwidth available to the user, the download and installation time can take anywhere from tens of seconds to tens of minutes. The Java Kernel project will cut this time down dramatically, allowing the application to install just what it needs to launch itself.

Just like QuickStarter, the main issue behind the time that download and installation takes is size: Java is large, even when zipped and Pack200 compressed, and there are physical realities to bandwidth and bit copying that we cannot overcome. Of course, one solution would be to simply shrink the platform, but in a world where Java is backward compatible and any Java application compiled to a particular release can expect to run on any release of that version or later, the idea of breaking up the Java runtime into subsets simply doesn't work.

The big idea behind Java Kernel is to take the sub-setting approach, breaking up the monolithic Java platform into discrete chunks of functionality that get downloaded and installed according to what any specific application needs, but to then stream down the rest of the platform in the background. Taking this approach ensures that an application that installs the Java Kernel will have the functionality that it needs with a much faster install and startup, but that any future application can still take advantage of the entire Java platform for that release, since the Java Kernel installation process will ensure that all of the proper bits get installed eventually.

The basic workings of Java Kernel are as follows:

• Download base functionality that every application needs (VM, garbage collector, security, classloader, and enough basic networking functionality to be able to download the rest of the bits)
• Download additional dependencies that this application specifies
• Download any "Class not found" exception culprits as needed
• Download the rest of the JRE in parallel until the entire release exists on the user's system

Work is still ongoing on Java Kernel (Ethan Nicholas is madly cranking away at it), but initial results show that it is possible to cut the download size by over 60% for mid-sized Swing applications. For example, here are [very] preliminary numbers for some sample applications to show how applications-specific bundle sizes compare with the complete size of the JRE:

Comparison of various Swing application download sizes with the full JRE

Deployment Toolkit

Wouldn't it be nice to be able to detect, from the browser, whether the user has Java installed, and what version they have? The Deployment Toolkit feature will make this process much easier, allowing detection of Java from either JavaScript or browser plugins.

To date, the main mechanism for automatic detection of Java on the user's system was the ActiveX control called Auto-Install that we released in J2SE 5.0. This mechanism was necessarily limited to Internet Explorer, and only if the user allowed that ActiveX control to run. Other than that, the process was quite manual and usually involved sending the user off to java.com to optionally install Java, from which adventure they might never return. For an example of how the current system works, check out the article Auto-Install: Easier Launching of Java Web Start Applications.

The Deployment Toolkit project is about enabling a much more powerful and ubiquitous system to run across multiple browsers and platforms, allowing developers to more automatically detect what the user has, what to do about it, and how to launch the application when Java is then installed.

There is still a browser plugin that provides a high level of detection, installation, and launching support, but now that plugin has been ported to also work in Firefox on Windows. But if the plugins can not be used, there is also a JavaScript solution, hosted primarly on Sun's site with a small piece of code that runs on the deployer's site, that can do far more than the current manual approach. The plugins can detect Java versions down to the update release, and can automatically trigger an installation of Java, launching the application when installation completes. The JavaScript solution is not as powerful, but still enables detecting Java versions down to the family level (i.e., J2SE 5.0, Java SE 6, etc.). The JavaScript version also cannot start the Java installer directly, but it can redirect the user to the appropriate download page and then poll in the background, waiting to return to the original page and launch the application when installation is complete.

Installer Improvements

The installation process for a new JRE should be simpler and more user-friendly. Besides the size+time issue that we plan to address with Java Kernel, there needs to be a better overall experience that users have during the installation process. Here are example before and after views of the first installation dialog:

Graphics Performance on Windows

We are re-writing the default graphics pipeline on Windows to take advantage of Direct3D for performing everything from simple rectangular fills and copies, which is what you get now by default, to translucency, gradients, arbitrary transformations, and other more advanced 2D operations. Swing applications simple and complex should benefit from much better runtime performance on Windows as a result.

Swing performs its rendering through Java 2D, and is therefore dependent upon the graphics rendering speed of Java 2D for fast Swing performance. In J2SE 1.4, we started accessing native hardware acceleration through DirectX on Windows, but only for the basic operations of filling and copying rectangular areas and horizontal and vertical lines. These simple primitives end up being quite important for Swing rendering, since much of the UI is comprised of these primitives, and the ability to cache the Swing back buffer as a VRAM image enables very fast double-buffering. But UIs are getting more complicated and the ability to accelerate more advanced operations, such as translucency, anti-aliased text, gradients, and scaling operations, is becoming increasingly important.

We have continued to work on the DirectX rendering pipeline in Java 2D, and have the ability to accelerate a wide variety of operations through the native Direct3D library on Windows. However, we have not been able to enable these performance improvements by default due to a combination of speed and robustness issues. Meanwhile, we have also implemented a parallel OpenGL rendering pipeline for Java 2D with even more advanced capabilities, but once again we cannot enable it due to robustness issues on various platforms.

Now we are finally rewriting the DirectX pipeline to mirror the capabilities that we have with our OpenGL pipeline, with fixes for robustness that make it a more viable default rendering pipeline. We should finally be able to enable this feature by default, exposing Swing to extremely fast hardware acceleration through the graphics processor. This should enable faster performance for current Swing applications, but will also enable much more powerful Swing applications to run fast as well, even those incorporating much richer and more dynamic, animated effects in their GUIs.

Nimbus Look & Feel

The Metal look & feel was good in its day. And Ocean was a decent theme for Metal, especially given the backward-compatible constraint of maintaining the metrics used by the UI components. But these cross-platform look & feels are, frankly, dated, and we need a more modern look for Swing applications. Of course there are other look & feels out there, some quite good, and we encourage developers to look to those products and projects as well for possible usage in their applications. But in the meantime, we also feel that we should provide a decent default for Swing developers in the core platform. Nimbus will be that new look & feel.

Here some samples of the the Nimbus look & feel so far:

Sample Nimbus components

SwingSet2, using the Nimbus look & feel

For more information on the Nimbus project, check out Jasper Pott's blog.

When Will It Ship?

As I mentioned earlier, the Consumer JRE will be an update for Java SE 6, which means that we will be able to deliver it much more quickly than we could if we waited for another major release like Java SE 7. But it's still a lot of work, and we're not done yet. Our best estimates now put the release date in early 2008, although we are trying to pull in the date if at all possible. Given the amount and type of changes in this release (especially Java Kernel), we need to send this release through extensive testing to make sure that it's as solid as you should expect it to be.

We will roll out some features as they are available, so that you don't have to wait for early 2008 to get everything here. For example, the improved installation experience should be out in update 2 of Java SE 6, which is currently set for late June.

The Fine Print

Most of these features are still very much in-development, and are thus apt to change scope or be removed from the release if our goals for them are not reachable. One of the main priorities for the release is getting it out to you as soon as we possibly can, with as much of these features as are doable in that timeframe. So if it looks like any particular feature would stall the release too long, we will have to weigh cutting or changing it in order to still keep our tight timeline. One of the things that is very clear to us is that the features we are providing are not things that would be nice to have in the future; they are necessary pieces of functionality that our developers and users want now. So we hope you'll agree that keeping strict control over the timeline is as important as the functionality of any particular feature we've discussed.

And who knows? Maybe we will also get the chance to add other cool and powerful features along the way, as long as they do not mess with our all-important timeline.

Stay Tuned

I will try to post updates to my blog as we know them, so stay tuned. And look forward, as I will, to a new era of consumer-enabled Desktop Java. In the meantime, check out the slides for the relevant talks at JavaOne when they become available. Some of these items were discussed at our Desktop Java overview, Danny Coward's Java SE: Present and Future, and Ethan's Easy Deployment sessions.

Life's Rough

I thought it would be there a month ago, and then I thought that it wouldn't make it at all, but it turns out that the "Rough Cut" of Romain's and my book is available on Safari Books Online.

So it made JavaOne after all. Which is nice, since we have 3 sessions on it this year (we just finished our JavaU session, then we have a session on it Wednesday and Friday).

I believe (having not seen it yet) that the online version is quite close to what we submitted as our "final" manuscript. This means that all of the content is solid, but there are probably issues with typos, some wording, and formatting that will be cleaned up before the final printed book version.

As far as the printed book is concerned, we are in the midst of the "copy editing" phase, just finishing up chapter 14 (of 19) today. Then it goes through formatting/layout (can't they just call revalidate() on it and then repaint()?) and we get to do one final (I hope) round of edits on that. Then it prints in July and floods the market in August.

But if you just can't wait, check out the Rough Cut on Safari. And look for our demos to be posted online little by little until then.

This work is licensed under a Creative Commons License.