Chet Haase's Blog

J2SE Archives

Tiger on the Desktop

I know Tiger's getting a lot of press lately (after all, we don't ship a new major Java release that often) and that there's a lot more about the new release than the graphics end of things. In particular, with a platform whose language changes so little and so infrequently, the types of changes introduced in Tiger are pretty substantial and attention-grabbing.

But still, there were some features that we worked on in the desktop client area of the platform for this release and if I can push! just squeeze shove! my way through the teeming throng of Tiger admirers, I'd like to mention a few of the highlights.

First, I should point out that on nearly every release, most of the work we do on the desktop client goes into things that don't percolate up into "features". This basically means fixing bugs, speeding things up, supporting any new platforms, and just making the existing platform run better every release. We've heard, loud and clear, that stability is one of the top requirements for our users, so the number one priority around here is quality (and everything that goes along with it, including making performance better and better, but not at the cost of core functionality).

Nonetheless, we do eventually get to work on new, cool stuff to enhance the platform. It's that stuff that I'll be discussing today.

The main things on Swing's feature parade for Tiger were some new Look & Feel features as well as table printing. I also have to add here my favorite feature: JFrame.add().

• Look & Feels

  There were two significant improvements in the Look & Feel area: Ocean and Synth.

  • Ocean

    The first was an update to the cross-platform look & feel. This one hasn't changed since the introduction of the Metal l&f. It had been a few years, that l&f was beginning to look a bit dated, so we worked on a more modern theme called "Ocean".

    One of the primary goals with Ocean was to Not Break Stuff. So now adjustments were made to the l&f which would have broken existing applications; the sizing of widgets is the same and everything should just work out of the box; it's just the skin of that l&f that's different (and, I think you'll agree, better).

    Users will get the Ocean l&f for free; an app that is run against 5.0 will pick up the Ocean l&f. Note that an app that subclasses Metal, or has other dependencies upon that look & feel, will still get the old Metal look & feel; again, backwards compatibility is the Prime Directive here.

  • Synth

    In addition to simply updating the cross-platform Java look & feel, the Swing team wanted to make it easier for developers to create newer, cooler look & feels for their applications. So they came up with synth, which is an architecture that makes it much easier to create new skins for applications, either through static XML definitions or programmatically through Java. I will leave the details of this approach to the experts; for more information on Synth, check out The Synth Look and Feel by Scott Violet on javadesktop.org.

• JTable Printing

  Don't you hate it when you've got a great GUI with some awesome data in it and you can't output that data in any other place that the screen? The Swing team thought so too, so they made printing work with JTable. Enough said.

• JFrame.add()

  The Swing team didn't call out this item when I asked them for their Tiger highlights, but it's probably my favorite Tiger feature in Swing and maybe the entire Java platform. Okay, so perhaps that's overstating it a bit. But I like it a lot in any case.

  Previously, you had to do the following to add a component to JFrame:

  	JFrame f = new JFrame();
  	f.getContentPane().add(someComponent);
  

  This is because of the Swing containment hierarchy; you cannot add a component to the JFrame itself, but only to the pane inside the JFrame which is tasked with being a container for sub-components. However, it was always awkward to have to remember that nuance and have to deal with Swing's containment hierarchy at this level. Well, they've fixed it. Now you can simply do this:

  	JFrame f = new JFrame();
  	f.add(someComponent);
  

  Internally, it's doing the same thing as before, only it's now handling the pesky containment details for you instead of your having to deal with it in your code.

Here are a few of the more significant features that you might notice about the graphics engine in Tiger. For more information on 2D Tiger features, check out our Release Notes.

• OpenGL

  This is an item that's not going to be apparent to most users right now, but should be an exciting thing for some developers and a taste of the great things to come, in terms of Java2D hardware accelerated graphics.

  We worked hard to enable nearly all Java2D functionality through hardware-accelerated OpenGL graphics on all Sun Java platforms (Solaris, Linux and Windows). This means we get accelerated graphics performance for such things as simple GUI operations (filling rects, drawing lines, copying buffers, caching and copying text glyphs as images, and so on), but we also get acceleration for way more interesting (and currently time-consuming) operations such as gradients, translucency, anti-aliased text, and transforms. This means that typical Swing applications work well, but more advanced 2D graphics apps just fly!

  There's a pretty big BUT here that I have to stick in at this point: this stuff all works extremely well ... except for the fact that OpenGL support on various platforms, graphics cards, and driver versions can be, er, less than optimal. Some of the problem lies with the way that we use the API and the types of features and performance we need; for example, we require different capabilities for Swing's back buffer than a typical GL app requires for its double-buffering operations. Some of the problem lies with the cards or drivers themselves; they simply have bugs that we tend to trigger (because we use the GL API in different ways than, say, Doom3 or other typical OpenGL applications).

  The net result is that we have an excellent implementation whose vast powers ... lie untapped by default. You can play with the OpenGL implementation by passing in a command-line argument to tell us to enable it:

  	java -Dsun.java2d.opengl=true 
  

  or you can set the same system property in your main() method:

  	System.setProperty("sun.java2d.opengl", "true");
  

  But otherwise, our OpenGL acceleration is disabled by default.

  But don't fear; I wouldn't have bothered mentioning it if it weren't way cool. Not only does it provide excellent benefits in the right situations now, but it also provides the necessary building blocks for us to have this kind of acceleration enabled by default on future releases. So it only gets better from here...

  For more information on this OpenGL renderer, check out the information in The OpenGL section of our Release Notes.

• Managed Images Everywhere!

  We introduced the concept of Managed Images in jdk 1.4. Basically, if you happened to create your images in certain specific ways (including Toolkit.getImage(), Component.createImage(w, h), and GraphicsConfiguration.createCompatibleImage(w, h)) then we would try to create accelerated, cached versions of the images underneath from which we could then do hardware-accelerated image copies.

  This system worked great for users of those particular APIs; developers didn't have to hassle with the management aspects of VolatileImage in order to enjoy the benefits of hardware accelerated image copies.

  But why the restrictions on which method calls you had to use? Why couldn't developers get the same benefit regardless of how the image was created?

  It was all a matter of just getting around to it. So in Tiger, we finally did.

  Now, you can create an image any way you like (including new BufferedImage(...)); if we can accelerate it, we will.

  Note that there are other things that can complicate the life and performance of a managed image; if this is a topic that interests you might check out these blogs on Images and this article on Intermediate Images for more details about image performance.

• Font Rearchitecture

  This is another "internal implementation" detail, but one which some users will definitely notice.

  The main work here was to take crufty old native code and move it up into elegant new Java code. This doesn't benefit the developers directly except that it makes the code much cleaner and easier to maintain (read: you'll get your bugs fixed faster and better).

  The real noticeable benefit of this change was that we removed threading constraints in the old code at the same time. Previously, much of the text processing was done in a synchronized fashion so that only one thread could access that key code at a time. This meant, for example, that an application rendering text to many different images at the same time would bottleneck in this code waiting for access.

  In Tiger, this constraint is removed and the font code is now Multithread Happy. Imagine the super-scalar servers that can now take advantage of all of those processors cranking away on text rendering simultaneously...

• CUPS Printing Support

  For the Unix printing fans out there, we now have CUPS support!

• ImageIO: plugins and performance

  We added a couple of key image format plugins in Tiger: BMP and WBMP (apparently popular image formats on some niche platforms; who knew?). We also worked on some major performance bottlenecks in image loading so the library is faster in some key areas.

• Startup performance

  The platform overall enjoys some serious startup improvements in Tiger. Small parts of those improvements came in the graphics arena, in the way that we were loading and initializing things that could have been done faster (and now it is).

  There's still a ways to go toward Startup Performance Nirvana, but we're making progress.

• Miscellaneous Performance Tweaks

  There were fixes in some rendering niches that are worth mentioning:

  • Primitive batching on Windows: we are smarter now about how we batch up line rendering calls when using our Direct3D renderer on Windows. This reduced the (significant) overhead per line and got us up to a 10x improvement in micro benchmarks (your mileage will vary; the biggest improvement will be seen when you issue all line calls together, not interspersed with other types of rendering operations).

  • Complex clip rendering speed was significantly improved in Tiger.

  • Footprint reduction for Toolkit images: We used to create intermediate images in our Toolkit image code which tended to bulk out the Java memory heap footprint. We've reduced much of that unnecessary garbage and got some pretty big footprint reductions as a result. Images tend to take up a pretty huge amount of memory (imagine: a buffer the size and depth of an average screen today takes about 5 megs of memory!), depending on the size and depth of the image, so saving the number of copies sitting around on the heap can help things out quite a bit.

There were lots of bug fixes in the AWT arena for 5.0. There were also a couple of areas of major functionality work. For more information on all of the AWT-related fixes and features in Tiger, check out the 1.5 AWT Release Notes. Here's a quick take on some of the larger items implemented:

• XAWT

  AWT was reimplemented on both Solaris and Linux to avoid being tied to the old Motif widget library system. The new XAWT toolkit was implemented using only Xlib, with a minimum of native code (and thus a minimum of native platform dependencies). XAWT is now the default on Linux, but Solaris still defaults to the Motif implementation for now. The default for either platform can be altered by use of the AWT_TOOLKIT environment variable or the -Dawt.toolkit command-line parameter; values for both can be set to either sun.awt.X11.XToolkit (for XAWT) or sun.awt.motif.MToolkit (for Motif). You can read more about XAWT at http://java.sun.com/j2se/1.5.0/docs/guide/awt/1.5/xawt.html

• X Drag and Drop

  Prior to Tiger, only Motif Drag and Drop (Motif DnD) protocol was supported on X11 platforms; now we also support the XDnD protocol. XDnD is a common feature on Linux and Java Desktop System platforms because of its support by GTK and Qt.

There were a few interesting (and huge) features added to our internationalization offerings in Tiger. For more information on these features than the blurbs below, check out http://java.sun.com/j2se/1.5.0/docs/guide/intl/enhancements.html. You can also get information on internationalization support in general at http://java.sun.com/j2se/corejava/intl/index.jsp.

• Multilingual Fonts

  In the past, logical fonts were monolingual or maybe bilingual. This means, for example, if you were running in a Japanese environment, you could display Japanese and English text, but not Arabic or Chinese. To use a different language, you'd have to quit the application, change a few environment settings (the LANG environment variable on Unix or the language selection in the Regional Options control panel on Windows), then restart the application. This made life difficult, for example, if the application you were using was an IDE, and you were using it to develop software that was localized into multiple languages.

  With the multilingual font configurations in Tiger, all you have to do is install fonts for all the languages you care about on your system and the JRE will use them all together when rendering with logical fonts. Note that today's operating systems usually come with support for many languages on their installation CDs - all you have to do is ask for that support to be installed. The list of writing systems that the JRE will support when provided with the necessary fonts and input methods is at http://java.sun.com/j2se/1.5.0/docs/guide/intl/locale.doc.html.

• Unicode 4.0 Support

  Character handling is now based on Unicode version 4.0 and there is also support for supplementary characters. For more information on supplementary character support, check out the article at http://java.sun.com/developer/technicalArticles/Intl/Supplementary/.

• Big Things

  The DecimalFormat class now supports BigDecimal and BigInteger.

• Vietnamese Support

  After a successful grass-roots campaign on the bug database (2517 votes on one of the bugs!), we have finally added support for Vietnamese in the java.util and java.text packages.

• Native Unicode Usage on Windows

  We now use the Unicode features of the native platform on Windows, which means we can now handle text in some components without restrictions on Windows locale settings.

That's about it, or at least that's the most visible part of our work on Tiger. We hope you enjoy it, and continue to send us feedback on what else you'd like to see in the future.

Happy Hacking!

Intermediate Images

"Freebird!"

I'm a little afraid of posting this blog, thinking that it could resulting in either

• So many and varied requests that it will be difficult to address them all

• No responses at all. (If a blog falls in the forest and noone reads it, does it still make a sound?)

Nevertheless, I'll forge ahead. After all, the whole point in my blogs/articles is to talk about stuff that developers want or need to know more about.

So, in the words of the unforgettable sourthern rock band and Java desktop client developers Lynyrd Skynyrd:

What is it you want to hear?

In other words, what would you like to see articles on? I have a few ideas kicking around in my head that I'd like to cover, and people have suggested a few more. But if you have other ideas that you would like to have considered, please tell me in the feedback section below.

Here are a few topics that I would like to cover sometime soon, to give you a feel for where things are headed. If you have opinions on these, feel free to post those below as well:

• Managed Images: I've written about Managed Images in various forums so far (such as the BufferedImage (Part 1 and Part 2), VolatileImage (Part 1 and Part 2), and Image Strategies articles). But maybe managed images deserve their own dedicated article.
• Old vs New: I believe that some of the confusion and performance problems in Java graphics usage comes from the difference between the old Java graphics APIs (those written in the jdk 1.x days, e.g. ImageProducer, ImageConsumer, PixelGrabber) and the new ones (starting from jdk 1.2, e.g. BufferedImage). I would like to write an article (or maybe a series of articles) on the differences between these APIs and their implementations. Specifically, I would like to examine how developers would do things (such as image pixel manipulation) on the old graphics APIs and how they would do things using the newer APIs (hopefully better, faster, and more efficiently).
• Server-based graphics: This is also probably a series of articles. One thread is a discussion of how to do graphics operations (such as image creation and manipulation) in server apps (when you are not displaying to screen, but rather to/from images only). Another thread involves rich clients for server-based applications (such as Web Services); how can we make it easier to write clients that interact on the network and with servers? These are all pretty complex topics, so understanding and explaining the issues may take some time. But in this increasingly connected world, this seems like a useful topic in graphics development.
• ImageIO Utilities: While ImageIO simplified many things in image reading/writing and manipulation, it complicated some operations that are simpler in older APIs. For example, displaying an animated GIF is simple when loaded through some of the old 1.x APIs such as Toolkit.getImage(). But when using ImageIO, the developer needs to do much more work to display the images correctly. It's that whole "with great power comes great responsibility" thing; the power of the ImageIO APIs and the information that it exposes to the developer also (for now) necessitates more work to make some operations succeed. I would like to explore some of these issues to see if we can derive some simple frameworks or approaches that developers can reuse in their work to simplify their applications.
• Image Scaling: This is related to the "Old vs. New" topic above; some operations in our APIs are not obvious. Also, even when there are obvious methods in the API for accomplishing tasks, sometimes it is not clear how these methods work, or that the best ways of accomplishing the tasks are perhaps not related to the most obvious means. For example, Image.getScaledInstance() is not, in general, the best way to scale an image, either performance- or quality-wise. The Graphics.drawImage() methods are much better in general, but they are not necessarily obvious to developers new to the Graphics class. I would like to cover some of these areas and discuss the merits of the different approaches.
• Rendering Hints: Wouldn't it be nice to have the hints discussed in a single area where it is more clear (than in the current javadocs) what happens when various hints are enabled?
• Transform Tips & Tricks: There are always questions about how to go about transforming things correctly or most efficiently. We could explore some of these topics in an article or series of articles.
• Animation: Some of the *Image* articles have touched on this topic, but perhaps an article dedicated to animation would be helpful. This article could cover everything from performance tips to timer usage.
  Done! I've just posted an article entitled Timing is Everything on java.net. It doesn't cover animation as a whole, but has some information about timers and utility code I developed to add higher-level functionality to the existing Java timers.
  
• Intermediate Images: This topic comes out of something we've discussed in our JavaOne talks before: using images to cache complex rendering. This can be a huge win for animation or performance-sensitive graphics apps. Some sample code and implementation details here might help people take advantage of this high-performance approach to Java graphics development.
  Done! An article on Intermediate Images is now up on java.sun.com.
  

That's all I can think of for now, although I tend to add to my internal list of blog TODOs frequently.

Threadaches

I find myself trying to multithread my life constantly. I've got so many things to do; surely there's a way I can multiplex the chores to get all of them done faster, right?

For example, I'll be brushing my teeth and realize I also need to comb my hair. I'm only using one hand to hold the toothbrush, so I reach for the comb with the other hand. Then I'll start combing my hair, at which point the other hand with the toothbrush stops moving, or (even worse) keeps moving, but in such as way that toothpaste starts running all over the place.

Now I've gone from two actions that would have been simple to perform serially to one interleaved complex action that's resulted in a toothpaste-and-spittle mess and an unruly mop on my head.

The problem here is that, despite the speed of our processors and the simplicity of our actions, some things in life simply cannot be multithreaded.

This is not to say that multithreading itself is unachievable or not worthwhile for some operations. As an example, consider breathing or blinking; if I had to stop any other action to make my eyes blink or to breathe, I'd never get anything done. Consider multiplexing actions while driving; if we couldn't do many things at once in this environment, we'd all be dead. (Of course, many people assume that this multiplexing-while- driving capability is universal and extends to complex conversations on phones while driving at 80 on the highway; I believe these folks will eventually be evolved out of our society, although they may take a few of us with them along the way).

So there are clearly some operations which are better done on separate threads; the trick is figuring out which ones are which if you don't want to be wiping up toothpaste off your clothes all the time. Or, in the case of your applications, if you don't want to be debugging thread deadlocks or performance problems due to thread abuse.

That reminds me; I was going to write about software in this article, not toothpaste and driving. I knew there was a reason I was posting this to a developer site...

Now, to get Technical

I've spent the better part of the last year thinking about multithreading problems, and I could probably spend the rest of my career doing the same (although it probably wouldn't be a long career since my head would pop off before too long). There are so many issues in this ugly space that to write about all of them would take too long. So for the purposes of a focused blog (although it's probably too late for that goal), let's just focus on one area of trouble for Java client developers: multi-threaded graphics.

When developers first discover the joys of multithreaded programming, it's like opening a new present on Christmas; the power of Java thread creation is that it is so easy to create and use new threads, that you can start taking advantage of multithreaded processing in a much easier fashion than you ever could before in previous programming languages. And the knowledge that your application can perform multiple tasks simultaneously, especially on systems with multiple processors or hyper-threaded cores, is huge. You no longer need to block on IO, waiting for the system to open a file. You no longer need to hang in your display code waiting for an image to downloaded over the network. You no longer need to hang the GUI while calculating a complex set of equations. In all of these cases, it is trivial to spawn a new thread to perform the bottlenecking operation, and use the result of that operation later as appropriate.

In fact, Java builds in much of this functionality for you, so you don't even need to do the work of multithreading some of your operations which may be blocking. For example, the old Toolkit Image loading methods automatically load images on a separate thread. The following code:

	Image img = Toolkit.getImage("duke.gif");

"Well heck," the Happy Developer says. "If two threads make my application that much faster, imagine how fast it'll be with ten!"

For example, suppose a developer finds that one of the more costly operations in their application is some rendering operation, like drawing some complex Shape. Every frame they have to draw numShapes of these shapes, like so:

	// Assume myShapes[] and numShapes are initialized appropriately elsewhere...
	public void paintComponent(Graphics g) {
	    for (int i = 0; i < numShapes; ++i) {
	    	g.draw(myShapes[i]);
	    }
	}

"Golly!", the Happy Developer says, "What if I use that cool Thread mechanism to speed this up?! Then it'll go way faster." They might write something like the following:

	class ShapeRenderer implements Runnable {
	    Graphics g;
	    Shape s;
	    ShapeRenderer(Graphics g, Shape s) {
	        this.graphics = g;
	        this.shape = s;
	    }
	    public void run() {
	        graphics.draw(s);
	    }
	}
	public void paintComponent(Graphics g) {
	    for (int i = 0; i < numShapes; ++i) {
	        Thread t = new Thread(new ShapeRenderer(g, myShapes[i]);
		t.start();
	    }
	}

They compile the code, run it hopefully ... and discover that it didn't fix their performance problem. In fact, they discover that the app is actually much slower than the original code.

What gives?

There are actually several different factors that can contribute to the performance of this particular approach, ranging from things that add no benefit to those factors that actually make multi-threading this application slower. Let's go through some of these factors, one by one:

1) Thread/object creation overhead
Okay, so we've all heard the mantra that object creation and destruction (and the associated garbage collection process) is actually pretty fast. Memory allocation and destruction on current garbage collection systems is almost free, in fact; it costs only the instructions involved in moving a pointer around (memory is assigned by reserving space on a pre-allocated heap, and is destroyed by another pointer move. Of course, there are more details here about how garbage collectors, but suffice it to say that simple creation/destruction of temporary objects is very, very cheap).

However, that doesn't mean to say that creating temporary objects is actually free. In particular, it doesn't mean that you want to create and initialize objects in your inner loop if you don't have to. In the above example, we are not only asking for temporary memory for the Thread and ShapeRenderer objects (a cheap operation), but we are also asking that those objects get created and initialized, which may not be so cheap, depending on the complexity of the objects involved and whatever initialization process they need to go through. In this case, the creation of a thread will probably involve a fair amount of processing at either/both the Java and native levels in order to create the underlying thread object.

The Happy Developer, realizing this, will of course take steps to minimize the temporary object creation. In this situation, they may realize that since the same thing happens every time through the loop, there is no reason that the applications needs to create the Thread and ShapeRendering objects every time through; they can just incur the overhead of creation one time and then reuse these objects whenever we need them.

I won't bother with an example here, just picture a variation of the above where the Thread and the ShapeRender objects are created only once. Then, inside the paintComponent() method, we need only update the Graphics object of each ShapeRenderer and then tell each Thread to do its thing.

Once again, the Happy Developer (this time with a slightly less huge smile of anticipation on their face) awaits the stunning results ... but discovers that this new variation is still worse than the original approach.

Things may be a bit better here than before; at least the application is not going through the contortions of creating and initializing the Thread and ShapeRenderer objects every time through the painting loop. But there's still something amiss in the messy threading details.

2) Thread Swapping

One of the hidden details of multithreaded programming is that the operating system has to go through a fair amount of work in order to run a separate thread. This is not much in the whole scheme of things (less than a millisecond, certainly), but it can add up when there are several threads involved.

For example, if you have ten threads all trying to do similar tasks at the same time and at the same priority, then the system will keep swapping the threads in and out trying to get the work done. This may not be as bad as swapping out each thread after just a couple of instructions in some round-robin fashion; we may get a fair chunk of work done in any given thread before we are swapped out. But the amount of work accomplished on that thread must be weighed against the work done to swap threads to know whether it was worthwhile having multiple threads to accomplish the task.

I wrote a test to see what thread-swapping overhead was. I ran in a tight loop, calling wait/notify to swap the threads back and forth. For 100,000 swaps, it took 1.3 seconds on that particular test system. This doesn't sound like much, but if you can imagine each thread trying to perform something simple like drawing a single line, the fact that we could only do 100,000 of these operations in that 1.3 seconds makes the thread swap overhead seem pretty significant. (For comparison purposes, I also timed calling a function 100,000 on the same system, which took only about 10 ms).

The cost of thread swapping overhead comes into play especially on systems where there are less computing resources available than there are threads than want those resources. This is an excellent, if obvious, segue into my next point...

3) Limited Thread Resources

The ideal case for multithreaded systems is having one processor per thread, or at least one processor available whenever a new thread needs processing power. For example, if you have a four CPU system and there are four threads all trying to run at the same time, they can each have a processor to themselves and get along swimmingly. There need be no thread-swapping overhead, as mentioned in point #2 above, because the threads do not have to be swapped out; they have full control over their CPU (at least while the process is running).

There are now hybrid systems where single chips can have multiple resources available for threads, such as the Hyper Threaded CPUs of various chips today. While these systems cannot dedicate an entire CPU to a particular thread, they have enough resources on any CPU to dedicate some of those resources to separate threads. There is still some overhead of thread swapping and contention here, but it is at least better than the single-CPU model.

The problem with the sample application above is that it does not take into account anything about the system when it creates a thread per Shape. Unless the application is running on a system that has as many CPUs (or at least an many thread resources, in the hyper threading case) as there are threads, then there is going to be contention in thread processing and thus overhead for swapping threads in and out.

Our Happy Developer might realize that the available thread processing resources could be a bottleneck. Suppose they know that, in general, their application will run on systems with at least 2 processors or one hyper-threaded processor, and they would still like to take advantage of that capability in multithreading their application. They may then change their app to use a model of exactly two rendering threads instead of the one-per-Shape model above. Now, instead of sending each Shape rendering operation through its own dedicated thread, it will queue up these operations on two separate threads. The code will be more complex for this approach, but still fairly straightforward. For brevity, I'll skip the example, but hopefully it's easy to picture this thread-sharing approach.

Again, the Happy Developer awaits patiently the results they know will stun the world. There is a slight faltering of their smile this time; they have met defeat too many times in the past. Still, they look forward to ... more failure.

Once again, the application fails to improve upon the original single-threaded approach. This time, they have eliminated much of the overhead in dealing with threads. And when running on a system than can process multiple threads simultaneously, they may even have eliminated thread-swapping overhead (or at least reduced it significantly).

Given the power of doing multiple tasks simultaneously, and the ability of the system to handle this simultaneity, what happened?

4) Graphics is Inherently Single-Threaded

Here's the sad reality of today's computing platforms; graphics hardware is inherently single-threaded.

This single-threaded approach is so ingrained in today's platforms that it is an assumption at the hardware, the driver, and even the API level (although some APIs are written to handle multi-threaded programming, they do not do a good job of compensating for the limitations below them and the hardware and driver level)..

While the hardware architectures, processors, operating systems, and languages have evolved to allow and encourage multi-threaded programming, the underlying graphics engines simply cannot do it.

This means that you may be able to easily write an application that performs graphics operations in multiple threads. And the system you are using (e.g., Java) may turn around and issue those graphics calls in separate threads. And the underlying systems that Java depends upon (e.g., X, DirectX, GDI, whatever) may be able to receive those calls from multiple threads. But when it finally gets down to the hardware, it has all been funneled through one pipe and there is no way to get any advantage by trying to use multiple threads at a higher level.

Just like the chain that is only as strong as its weakest link, an application is only as multithreaded as the systems it depends upon; in this case, if the graphics subsystem is single-threaded, then there is nothing you can do at the upper layers to make that system more multi-threaded-friendly.

Let's take a look at an example. I was working on this one just this week, playing around with various options in double-buffering. I wanted to play with the idea of writing to a buffer on one thread and copying that buffer to the onscreen window on a totally different thread. There were various reasons for this (and various possible gotchas), but it was at least worth an experiment.

I wrote my rendering loop to fill the buffer as fast as it could, with simple calls that mimicked a scrolling operation (copy part of the buffer to itself in a different location, fill in the rest with some color). After each operation, it would update a flag that told the system that the buffer contents were new (and should thus be copied to the onscreen window at some time).

I wrote my buffer-copying thread to occasionally wake up (every few milliseconds) and copy the buffer to the screen if the buffer had been changed since the last copy.

There are a few implementation details here (such as synchronizing on the update flag variable), but I have described the essential bits.

What I saw confused me at first. It ran pretty well on my development system. So I had someone else run it on their system. In that new environment, it basically froze the window for several seconds. It looked like we were not even getting our screen- updating loop, as if the system was not even kicking off the timer I had set.

In digging into it further, I found the artifact was more disturbing. We were being woken up correctly based on the timer, and were then attempting to update the screen. But the buffer-rendering loop had so completely filled the graphics pipeline with scroll/fill calls that we basically froze the system while those were being worked on by the underlying driver and hardware.

Here was a situation where:

• the language supported the threading approach
• the underlying subsystem (in this case, DirectDraw) supported rendering to/from multiple threads
• the operating system supported multiple threads

• the graphics rendering system was created as inherently single-threaded, so the fact that one thread could so completely fill the graphics pipeline meant that future threads that the operating system wanted to swap in would simply have to wait for the original thread.

The upshot of this whole diatribe on the graphics subsystem is that there is basically no gain to be had in changing the original sample application in the way that we did; since the underlying system is inherently single-threaded, we have nothing to gain and everything to lose by introducing potential thread overhead when everything will just end up in a single thread in the final rendering step in the hardware.

Conclusions, Thoughts, Powertool Accidents

The point of this article was to raise some issues to be aware of in multithreaded programming. Note that I am specifically not saying "Don't Do It!". There are lots of advantages to multithreaded approaches, some of them mentioned in the introduction above. For example, it is still a huge win to do time-consuming non-graphics tasks in a separate thread (such as IO or image loading) if you do not want to block the main thread (a big example in my desktop client world is the canonical "don't block the GUI thread" example; all blocking operations should happen elsewhere so that the user still sees a snappy GUI)..

And there are even cases where graphics operations can be effectively multi-threaded. For example, in a system that is using all software rendering (such as rendering destinations that are not hardware-accelerated, such as BufferedImage objects) running on systems that support multiple threads, there could be a huge win in having separate rendering threads. Imagine a multi-chip server that is producing separate images all in parallel, rendering to each with software loops, running each of those loops on separate processors; this is a major win for parallelism.

But what I am saying is: be aware of the issues in the platform and underlying systems when taking a multithreaded approach. And always test your application to see if you actually got the speedup you were anticipating.

Think of multithreaded programming kind of like a chainsaw. It can be an incredibly powerful tool that can dramatically reduce the time needed to perform some tasks. Or it can chop your hand off and cause a huge mess that's impossible to recover from. You need to know how to use it effectively to determine which one it will do for you.

ToolkitBufferedVolatileManagedImage Strategies

A common question seems to arise often from Java graphics developers about which image type or creation method to use. When exactly should you use VolatileImage? What is BufferedImage appropriate for? What about the old Toolkit images? And when is BufferStrategy more appropriate than one of these image types?

It's a pretty big topic, and the answer (like all truly great answers) is probably "It depends". But there are some general guidelines that can come in handy. And perhaps a description of what these different kinds of images and methods are all about might help.

1) Image Types

First of all, perhaps a short dictionary of image types might help:

• Toolkit Image: This is the oldest kind of image object in the Java API. These images are created and returned by the old 1.x APIs such as Applet.getImage() and Toolkit.createImage(). These images are created from a pointer to a data source, such as a GIF or JPG file, and return an object of type Image. They are useful for convenient loading and storage of image data for display, but getting at the actual pixel data or manipulating it is not as easy.
• BufferedImage: This is an image type that was created in the jdk1.2 API. They were created for easier and more powerful manipulation of the actual pixel data in an image. At first, there was no way to load an image from a data source directly into a BufferedImage; these images were used, instead, to create an arbitrary buffer for pixel data, which you could then write to, read from, or display conveniently. The main way to get actual pixel data into a BufferedImage object at first was through use of rendering operations (after getting a Graphics object for the BufferedImage), or by manually setting the pixel data through methods in BufferedImage, WritableRaster, and DataBuffer. With the advent of the ImageIO API (see below) in jdk1.4, it became possible to create a BufferedImage object directly from a data source, just like Toolkit images (only these BufferedImage objects are writable, unlike their Toolkit image cousins).
• VolatileImage: This image type was created in jdk 1.4 as a means of creating and managing accelerated image memory. One of the problems with hardware acceleration for images is that, on some platforms, accelerated memory can be deleted out from under you at any time. This is obviously not what you want for your typical image data. To work around that, the VolatileImage API was created to provide a notification mechanism so that you know when an image must be re-rendered due to data loss. VolatileImage objects are not loaded from image data, but are just created as empty pixel buffers (much as the initial BufferedImage objects were (see above)); to get loaded image data into a VolatileImage, applications must load the image data through some non-Volatile means, get the Graphics object for the VolatileImage, and then copy the data into the Graphics object using drawImage().
• Managed Images: These image objects are not specific objects or APIs in Java, but are rather a concept of how we accelerate image operations. A "managed image" is one that you create through any of the normal image creation/loading methods and which we try to accelerate for you internally, by creating an accelerated mirror copy of the image data. This type of image can benefit from hardware acceleration without falling prey to the "surface loss" issues mentioned above for VolatileImage. I'll talk more about managed images and hardware acceleration later in the article.

That's it for the basic image types. Now let's talk about how we actually create and use these image objects.

2) Who you gonna call?

Whenever I want to give myself a fright about the complexity of our APIs, I simply ponder the vast array of choices that face developers who simply want to create an image. I'm sure I'm missing some here, but let's see...

Applet:
	getImage(url)
	getImage(url, name)

BufferedImage:
	new BufferedImage(colorModel, raster, premultiplied, properties)
	new BufferedImage(width, height, type)
	new BufferedImage(width, height, type, colorModel)

BufferStrategy:
	new BufferStrategy()
	(Note: This method cannot be called directly)

Canvas:
	createBufferStrategy(numBuffers)
	createBufferStrategy(numBuffers, capabilities)
	
Component:
	createImage(imageProducer)
	createImage(width, height)
	createVolatileImage(width, height)
	createVolatileImage(width, height, capabilities)

GraphicsConfiguration:
	createCompatibleImage(width, height)
	createCompatibleImage(width, height, transparency)
	createCompatibleVolatileImage(width, height)
	createCompatibleVolatileImage(width, height, capabilities)

Image:
	new Image()
	(Note: This method cannot be called directly)

ImageIO:
	read(file)
	read(imageInputStream)
	read(inputStream)
	read(url)

ImageIcon:
	new ImageIcon(imageData[]).getImage()
	new ImageIcon(imageData[], description).getImage()
	new ImageIcon(image).getImage()
	new ImageIcon(image, description).getImage()
	new ImageIcon(filename).getImage()
	new ImageIcon(filename, description).getImage()
	new ImageIcon(url).getImage()
	new ImageIcon(url, description).getImage()

Toolkit:
	createImage(imagedata[])
	createImage(imagedata[], offset, length)
	createImage(producer)
	createImage(filename)
	createImage(url)
	getImage(filename)
	getImage(url)

Window:
	createBufferStrategy(numBuffers)
	createBufferStrategy(numBuffers, capabilities)

I'm sure there's more out there, especially using things like ImageIO (which is all about reading and writing images, as you might guess from the name...). But this list will do for now.

So it's a wrap. This article's pretty much finished; just use the above API calls to create your images. Left as an exercise to the reader. Q.E.D. It's obvious, isn't it?

Okay, so maybe it isn't obvious; there are a lot of methods above that all seem to need different parameters or that create different types of images.

Here's the trick: All of the above image creation methods (and any others that are not on the list) can be broken down into just a few categories. Then the plethora of ways of creating an image in one of those categories can just be seen as utility methods; different ways of getting the same result. The convenience methods may be because of logic (why do I have to get the GraphicsConfig to create an image associated with a Component? Why not use the Component directly?), or convenience (instead of using some InputStream mechanism for all image readers, we provide several ways to read the image directly including from filenames, URLs, and streams; just call the method appropriate for your situation).

So the real work in this article is to break down the categories of image types and describe which types of images and methods you may want to use in which situations. Once you get that down, the rest, as they say, is just implementation details.

3) Image Loading or Creation?

First of all, are you loading existing image data? Or are you creating an image buffer in memory? Image loading means that you have image data (either locally or across the network) that you want to load into your application, possibly to copy that image onto the screen or to read and operate on the data. Image creation means that you want some arbitrary image memory created for your application; perhaps you want to create a buffer for double-buffered animations, or you want a place to cache intermediate filtering results.

3.1) Image Loading

In the above method list, all of the methods that take filenames, urls, streams, producers, and data arrays are those intended for loading existing images. In particular, all of the methods listed above for Applet, ImageIO, ImageIcon, and Toolkit are intended for image loading:

Applet:
	getImage(url)
	getImage(url, name)

ImageIO:
	read(file)
	read(imageInputStream)
	read(inputStream)
	read(url)

ImageIcon:
	new ImageIcon(imageData[]).getImage()
	new ImageIcon(imageData[], description).getImage()
	new ImageIcon(filename).getImage()
	new ImageIcon(filename, description).getImage()
	new ImageIcon(url).getImage()
	new ImageIcon(url, description).getImage()

Toolkit:
	createImage(imageData[])
	createImage(imageData[], offset, length)
	createImage(producer)
	createImage(filename)
	createImage(url)
	getImage(filename)
	getImage(url)

There are at least four major things that differentiate these methods:

• the location of the image data
• the format of the image data
• the synchronous or asynchronous behavior of each method
• the type of image that is created (Image or BufferedImage)

When I'm talking about location, I'm mainly concerned with whether the file is local or across a network. Also, if it's packed into some resource file, such as a jar file, that also comes into play here.

Loading across the network

If you are accessing the data across a network, it's probably easiest to use the URL variations:

	Applet.getImage(url)
	Applet.getImage(url, name)
	ImageIO.read(url)
	new ImageIcon(url).getImage()
	new ImageIcon(url, description).getImage()
	Toolkit.createImage(url)
	Toolkit.getImage(url)

	URL dukeURL = new URL("http://java.sun.com/duke.gif");
	BufferedImage img = ImageIO.read(dukeURL);

Now suppose you have another image that you have saved locally in a file; just use the filename variation of the above. For example, let's say you loved one particular instantiation of the lovable-yet-quirky duke.gif file above so much that you downloaded and saved it for use in your application (see the above note on scary lawyers). Then you could use the following code to load that file from the directory where the program was launched:

	BufferedImage img = ImageIO.read(new File("duke.gif"));

	URL dukeURL = getClass().getResource("media/" + "duke.gif");
	BufferedImage img = ImageIO.read(dukeURL);

Another consideration is the format of your stored image. The old Toolkit/Applet loaders only understand GIF, JPEG, and PNG format files. (Okay, they also understand XBM and XPM2, old X11 image formats, but those are probably not formats you are terribly concerned about). These loaders works well for most web applications since these image types are traditional web image formats. But what if you have an image in some other format that the Toolkit/Applet loaders do not understand?

ImageIO currently has built-in readers for GIF, JEG, and PNG. In addition, it will have BMP and WBMP capability in the jdk1.5 release. Moreover, there will be more image readers/writers for ImageIO going forward, whereas there are no specific plans to support more formats for the old Toolkit/Applet loaders. And finally, ImageIO has a pluggable reader API, so if you have a custom image format, or some other format not yet supported by the core library, you can write your own loader for that format within ImageIO.

In fact, the JAI team has made available a package with additional ImageIO readers/writers at http://java.sun.com/products/java-media/jai/downloads/download-iio.html if you have requirements beyond the current ImageIO defaults.

So ImageIO could also be the right choice if you need to deal with formats beyond the basic web image formats.

3.1.3) Synchronicity

The Applet and Toolkit image loading methods came from the old days of Java 1.0, when Java was seen primarily as a networked application API and image data might come from any source, potentially one on an unreliable or slow network connection. To make networked applications more robust, it is reasonable to put network-dependent operations in separate threads to ensure that an application's main or GUI threads do not hang while waiting for a slow download. Because this was a common pattern for Java GUI applications at that time, the image loading operations were all created to run on a separate image loading thread. Thus when an application calls:

	Applet.getImage(url)

Note that this model of asynchronous loading does not apply solely to networked applications, or even to image loading specifically; any operation that takes a significant amount of time should not be done on the GUI thread, lest you run the chance of making your application appear hung while the operation is taking place. So, for example, if you are loading in a huge image from a local file, you may want that non-networked operation to happen in a separate worker thread to ensure that your GUI has no pauses during image loading.

This model works well enough for applications that create their images early for later use. The application simply may need to check whether the image has been loaded whenever it is required in the application.

When applications do need the data (for example, if they need image sizes in order to determine layout correctly, or if they need to display images in their final form), they may need to synchronize on the image loader and wait until the image loading is done. For example, an application may want to load local image data and be willing to wait for that data to load before proceeding (knowing that a local load will usually not take very long). In that case, the application might do something similar to the following:

	public Image loadImageSynchronously(Component comp, String filename) {
	    URL url = getClass().getResource("media/" + filename);
	    Image image = Toolkit.getDefaultToolkit().getImage(url);
	    MediaTracker tracker = new MediaTracker(comp);
	    int id = 0;
	    tracker.addImage(image, id);
	    try {
		tracker.waitForID(id);
	    } catch (InterruptedException e) {
		System.out.println("INTERRUPTED while loading Image");
	    }
	    return image;
	}

	public Image loadImageSynchronously(String filename) {
	    URL url = getClass().getResource("media/" + filename);
	    Image image = new ImageIcon(url).getImage();
	    return image;
	}

Meanwhile, ImageIO has synchronous loading methods that do not return until the image has been loaded and is ready to go. Note that some applications and situations may still need asynchronous loading behavior (for long image loads or to more efficiently multitask). For example, it does not take a huge amount of time to affect perceived GUI performance, so if an image load will take even as long as a tenth of a second, you may want to avoid loading that image synchronously on the Event Dispatch Thread (so don't load it in your paint() method). You can always spawn a new Thread yourself to call the ImageIO loading methods if necessary.

3.1.4) Resulting Java Image Type

Part of the decision over which image creation API you is in which image type you want to get back from the creation method. In particular, do you want a Toolkit Image or a BufferedImage?

Toolkit Images are created by the Applet, Toolkit, and ImageIcon methods listed above. The resulting images are easy to use for display purposes (just call drawImage(...) from this image into a Graphics destination and the image will be copied appropriately), but lack the power of BufferedImage objects for manipulation of the data.

BufferedImage objects are created by the ImageIO methods listed above. These objects offer a more powerful API, albeit with potentially more work involved to do some operations (such as displaying an animating GIF image).

Image or BufferedImage: What's in a Name?

Although both Image and BufferedImage have similar properties in terms of being displayable, BufferedImage has many more capabilities. For one thing, the Image objects created by the Toolkit, Applet, and ImageIcon load methods are read-only; you cannot get the Graphics of those Images and render to them. So if you want to modify the image data, you will need to do more work (such as creating another image that is modifiable and copying the loaded Image into that new image). Image has some very simple methods and is mostly intended to be a simple object that holds image data. But BufferedImage has many methods for modifying and extracting all kinds of data from an image; color models, pixel data, and more. Given a choice between the two, I would always opt for the one that gave me more power and flexibility.

But doesn't that increased capability mean increased overhead? Not at all; there is no extra processing involved in BufferedImages when these other powerful methods are not used. If all you do is load and image and display it, BufferedImage can do this just as easily as the more streamlined Image object.

So go ahead and use BufferedImage. It is, after all, better than butter.

Dirty Laundry

One good (and not entirely obvious) reason for using the ImageIO API for loading images is the unfortunate reality that the code is simply newer, cleaner, and more maintained (both now and in the future). Much of the old Applet and Toolkit image code was written years ago and has many assumptions and situations that it must account for and is therefore tricky to maintain and upgrade.

Our future image reading/writing direction is with ImageIO; yours should be too, because that's where the focus of our efforts will be in the future.

Having said all that wonderful stuff about ImageIO, there could be situations in which the old Toolkit/Applet/ImageIcon approach makes more sense for your particular application, including:

• Performance: Ideally, ImageIO would be more performant than the old code. But some code paths in ImageIO have not yet been optimized to the extent that the old code was, so you may find some situations (such as loading images in a particular format) which perform better using the old APIs than they do with ImageIO. The problems are (mostly) known, the bugs have (mostly) been filed, and these performance gaps will (definitely) be fixed. But if performance in these situations is important for your application, by all means use the old methods in the meantime if they work better for you.
• Ease of Use: One of the goals of ImageIO was to expose more capabilities to the developer. There is much information, such as metadata, in an image that is not exposed through the old image loading methods or the old Image class. The flipside of this approach (at least in the current API) is that it can be more work to do trivial tasks. A good example of this is loading and displaying an animated GIF.
  In the old image loading APIs, if you loaded an animated GIF file, you did not need to worry about the details of animating that image. Instead, you could just load the image and display it and we would automatically schedule the animations of that image appropriately. You could not access the information about the animation (how many frames? what animation rate? loop? halt?), but making it animate was quite simple.
  ImageIO, on the other hand, exposes all of that animation information to you through the API, but you must actually use that information to manage the image animation yourself; you cannot just tell us to draw the image and have us manage it for you. This is a known issue and we will do something about it Real Soon Now; look for an upcoming article about doing this, and look for future API changes to make this simpler in the future.
  
• Compatibility: Sometimes applications need to be as backward-compatible as possible across most or all jdk releases. In particular, some applications need to be able to run on old jdk1.1 or earlier releases. For example, applet games tend to need ubiquity of the runtime environment and cannot count on browsers having any later version of Java than the 1.1 APIs shipped with the MSVM. In this case, you may need to use the older APIs simply so that they will run across all of these platforms.

Note also that if you need to use the old APIs for some reason but you still want the power and flexibility of BufferedImage, it is easy enough to load the images in through whatever methods are appropriate, create a new BufferedImage object, and then simply copy the loaded images into the BufferedImage. For example:

	Image img = new ImageIcon(filename).getImage();
	BufferedImage bImg = new BufferedImage(img.getWidth(null),
					       img.getHeight(null).
					       BufferedImage.TYPE_INT_RGB);
	Graphics g = bImg.getGraphics();
	g.drawImage(img, 0, 0, null);

3.1.5) Hey! What about the other loading methods above?

The approaches above cover most of the loading methods I listed, but some are notably skipped. The *Stream methods of ImageIO are simply variations on a theme; if you happen to have your data in that format (versus a URL or file), go for it; it's just a convenience to use these alternatives.

As for the other skipped methods (one using an ImageProducer and some using data arrays), I hoped you wouldn't notice....
These other loading approaches are somewhat dated and come from the old days of Java image processing. There are still some situations which might require the ImageProducer/ImageConsumer approach, but in general it should be easier, more straightforward, and more performant to use the newer APIs such as ImageIO and BufferedImage. The uses and misuses of the older Consumer/Producer/PixelGrabber APIs could stand a separate article all by themselves. I'll try to tackle this one in the future.

As far as reading the image data from an array of data (see the methods above with the imageData[] parameter), this is really only appropriate if you've already read the data into the array to begin with. This could be necessary if you have some custom image storage mechanism, such as a database. But if the image existing in a regular file/URL/stream format, you should probably be using one of the other loading methods instead.

3.2) Image Creation

What if you do not have an existing image on the network or file system? What if you just want a buffer of pixel data that you can use in your application? This could be for creating sprites or icons with rendering calls instead of loaded image information (perhaps you've found this to be faster in your situation than reading image files). Or it could be a buffer that you can use for caching intermediate results or for providing double-buffered rendering for an animation.

For the purposes of this discussion, I'll break down this category of images into three types:

• Static Images: These are images that you render to infrequently (perhaps just once, when you create the image). The images are mainly used to copy from.
• Dynamic Images: These are images that you render to often, like an animating sprite.
• Back Buffers: These are like dynamic images, in that they are rendered to frequently (usually at least once per frame), but they are specifically intended to provide buffering support for the onscreen window, and are thus usually copied from often as well, like once per frame.

3.2.1) Static Images

Static images are ones that are created and rendered to once (or infrequently) but probably copied from often. Examples of this type of image include icons for a GUI or sprites for a game.

The best approach for this type of image is to create an image that is in the same format as the image or window that the image will be copied to; this ensures the most straightforward copy mechanism since the underlying software will not have to perform a conversion on the image data while copying to the destination.

You could, of course, create a BufferedImage object manually through one of its constructors; you could query the GraphicsDevice for its display information and then create a BufferedImage of the appropriate type:

	new BufferedImage(colorModel, raster, premultiplied, properties)
	new BufferedImage(width, height, type)
	new BufferedImage(width, height, type, colorModel)

But why go to the hassle of all of that when there are convenience mechanisms that do all of this for you? Specifically, take a look at:

	Component.createImage(width, height)
	GraphicsConfiguration.createCompatibleImage(width, height)
	GraphicsConfiguration.createCompatibleImage(width, height, transparency)

The best part about static images is that you can use very simple means to create the images and then we will try very hard internally to see that you get any available hardware acceleration for these images when they get copied around. We call these "managed images", because we manage the acceleration details for you. For more information on managed images, please see my blog on BufferedImage performance.

Note that we currently (in all jdk1.4.* releases) manage images that are created with the above APIs and some of the Toolkit image loading methods described previously, but in jdk 5.0 (available in Beta form now, and full release soon) we manage nearly all types of images and take advantage of hardware acceleration if it exists. So go ahead and create the type of image that is most convenient for you and we'll try to do the right thing under the hood.

3.2.2) Dynamic Images

This kind of image may be rendered to quite often, as in an animating icon, or a sprite that is modified on a frequent basis. You could certainly use the same image-creation APIs listed above for static images; these will work fine in most situations and are certainly the easiest way to go in general.

However, some developers interested in maximizing performance may want to know more about image management and how dynamic images can affect it.

We manage images by detecting when the application is copying from an image to a destination (either another image or an onscreen window) that lives in accelerated memory. If this copy is done successively when the source image has not changed, then we may decide to cache a copy of that image in accelerated memory and perform future copies from this cached version.

In the case of a dynamic image, if that image is being updated one or more times for every copy to the destination, then we will never create an accelerated version of it, and thus the image will never benefit from any hardware acceleration that we could otherwise provide.

(Aside: For the insatiably curious, the reason for this oddity in acceleration comes from "surface loss", where an accelerated version of an image may simple go away at any time due to operating-system or user-caused situations. To keep the original image data intact, we store the main image data (that which is modified by the application) in an unaccelerated location, and only accelerate a mirror copy of that image. That way, if the accelerated version gets wiped out, we still have the original data from which we can create a new accelerated copy. The problem here, in terms of performance, is that an "unaccelerated image" means that all rendering to and from that image is unaccelerated. And if an application is constantly modifying the image, all of that rendering will be unaccelerated and it is never appropriate for us to create and use an accelerated version of that image.)

Developers that care about top performance for these types of images may want to look into using VolatileImages instead. These images store the data in accelerated memory (when possible) and thus rendering to and from that image may be accelerated automatically. The downside is that these images require a bit more care and feeding, due to "surface loss" issues that arise with current video memory architectures and operating systems. Note that not all types of rendering to these images is accelerated, either, but simple types of rendering like lines and rectangular fills and copies can usually be accelerated, depending on the platform configuration.

I've already written about VolatileImages in past blogs (Part I and Part II), so I will not go into the details of their usage here; please check out those other articles for more information. But it is worth covering the APIs used to create the images, just for consistency's sake in this article:

	Component.createVolatileImage(width, height)
	Component.createVolatileImage(width, height, capabilities)
	GraphicsConfiguration.createCompatibleVolatileImage(width, height)
	GraphicsConfiguration.createCompatibleVolatileImage(width, height, capabilities)

	GraphicsConfiguration.createCompatibleVolatileImage(width, height, transparency)
	GraphicsConfiguration.createCompatibleVolatileImage(width, height, capabilities, transparency)

The use of the ImageCapabilities object in these methods gives you the ability to require certain attributes (such as hardware acceleration) from any image created with that method. In general, you probably will not need to use that variation, although as we enable more hardware acceleration features in our platform, we may expand the ImageCapabilities API to be more powerful and useful. (Note, too, that ImageCapabilities can be used effectively as a means of inquiring what capabilities an existing image has).

3.2.3) Back Buffers

By "back buffer" I mean an arbitrary offscreen image that is created for use in a double-buffering situation. Typically, an application that wishes to have smooth graphics, especially animations, will draw to a back buffer and then copy that buffer onto the screen instead of drawing directly to the screen. Swing does this by default, so that you do not see the various GUI elements in a Swing app flash as they are drawn to the screen. The buffer copy in these applications typically happens so fast that the graphics in the application are perceptibly smoother than if they were drawn one-by-one directly to the screen.

A developer could use any of the above static or dynamic image APIs that I listed for creating a back buffer. However, the following things should be taken into account when doing so:

• Use of managed images for buffers has the same performance implications as dynamic images; we will be unable to manage the images effectively, thus you will not be able to take advantage of any hardware acceleration either for rendering to or copying from the buffer. Since the copy from the back buffer to the front buffer involves a lot of pixels, being able to accelerate that operation can have a huge performance impact on your application.
• Use of VolatileImages for buffers is a better way to go in terms of hardware acceleration potential. But why go to all the bother of managing your VolatileImages when there is an easier route?

BufferStrategy: the preferred way of buffering in Java

In jdk1.4, we introduced the BufferStrategy API, which is a wrapper around VolatileImages. This API allows you to ask for an accelerated back buffer and avoid having to manage the details of surface loss associated with VolatileImages. It also ensures that you will get a buffer of the optimal type for your application. In particular, you will get either a FlipBuffer (which can only be used in fullscreen-exclusive mode on Windows) or a BltBuffer (which is used by default for windowed applications). A FlipBuffer performs a swap of the front and back buffers in video memory when you request BufferStrategy.show(). A BltBuffer copies the contents from the back buffer to the front (just as you would if you called drawImage() from a VolatileImage back buffer to the front buffer).

With this API, there is little need to create and manage VolatileImages directly; just let us manage the details for you inside the BufferStrategy implementation.

For more information on BufferStrategy, check out the javadocs; they're pretty clear on how the system works.

The APIs you will need when creating a BufferStrategy are:

	Canvas.createBufferStrategy(numBuffers)
	Canvas.createBufferStrategy(numBuffers, capabilities)
	Frame.createBufferStrategy(numBuffers)
	Frame.createBufferStrategy(numBuffers, capabilities)

4) Wrap-Up

So that's pretty much it. You have image loading methods and image (or buffer) creation methods. And in each category, you have various flavors depending on the location and type of the data, and the type of image you want returned to you. So even though there are a lot of methods listed at the top of this article, they all break down into just a few comprehensible categories and can be used effectively, once you understand the implications of each variation.

Although there is certainly more complexity here than we can cover with a simple table, it might help to break down some of the basic attributes of the image types we have talked about and the reasons to consider one type over another when writing your application:

• Toolkit Image:
  • Creation methods: Applet.getImage(...), Toolkit.createImage(...), Toolkit.getImage(...), ImageIcon.getImage()
  • Most useful for: Simple loading and display of image data, applications that need to use 1.x APIs, asynchronous usage models (except for ImageIcon), loading performance (versus some ImageIO loading operations in the current releases).
  • Less useful for: Image data manipulation, reading/writing individual pixel data within the image, synchronous loading operations (except for ImageIcon), more varied or custom source image formats..
• BufferedImage:
  • Creation methods: new BufferedImage(...), Component.createImage(...), GraphicsConfiguration.createCompatibleImage(...), ImageIO.read(...)
  • Most useful for: Image data manipulation, read/write access to individual pixel data, synchronous loading or creation, various image data formats (both source data that ImageIO can read and Java format (i.e., TYPE_INT_RGB, TYPE_INT_RGBA, etc.).
  • Less useful for: Applications that require jdk 1.x compatibility, easy display of animating GIF images.
• VolatileImage:
  • Creation methods: Component.createVolatileImage(...), GraphicsConfiguration.createVolatileImage(...)
  • Most useful for: Hardware acceleration of dynamic images, more fine-grained control over acceleration properties.
  • Less useful for: Simple image creation and display (applications must use the VolatileImage API to handle surface loss issues)
• BufferStrategy:
  • Creation methods: Canvas.createBufferStrategy(), Window.createBufferStrategy()
  • Most useful for: Easy creation and management of window buffering schemes
  • Less useful for: Access to actual image data of buffer

4.1) Hey! You forgot some methods!

There are still a couple of the creation methods up top that I have not covered yet:

BufferStrategy:
	new BufferStrategy()
Image:
	new Image()

4.1) What About Performance?

It is difficult or impossible for me to write a long block of text or code without thinking about performance. And since some of the users of the APIs above, and image operations in general, care a great deal about performance, I should spend a few words discussing some performance issues to be aware of.

Again, check out my blogs on BufferedImage ( Part I and Part II); I go into much more detail on image management there. Some important things to keep in mind with respect to managed images (and making sure they are benefiting from available acceleration):

• If you want copies from an image to be accelerated but you will be modifying the image frequently, think about VolatileImage instead (see the above discussion on Dynamic Images).
• Careful with that Raster, Eugene! This is a hidden gotcha in the current implementation. Part of the BufferedImage API is a method that allows you to get the Raster of the image (getRaster()). From that Raster, you can create a writable child raster (WritableRaster.createWritableChild()) or get the internal data array for the raster (Raster.getDataBuffer()). Both of these methods end up giving you a copy of the data that we have no control over. Specifically, we cannot tell when you have updated the contents of the image, so we cannot tell whether our cached/accelerated version of the image needs to be updated. So, when you call these Raster methods, we throw up our hands and give up on acceleration entirely.
  If you need to access the pixel data but you still want us to accelerate image copies, call methods which manipulate the data for you without giving you a reference to the data; e.g., BufferedImage.setRGB() and WritableRaster.setPixels() for setting arbitrary pixel values, or any of the Graphics.draw*() or Graphics.fill* methods for setting blocks of pixel values.
• If you are transforming the image while copying it around (say, scaling it to the destination, or rotating it), we will not be able to do that using hardware acceleration with the current implementation. If you anticipate using a particular transformation of an image (like a zoomed-in view or several rotations of a sprite) multiple times, then it might be worth creating a temporary version of that transformed image and letting us manage (accelerate) that transformed version for you.

4.2) What about 5.0?

Most of the APIs I discussed above are pre-5.0, so you can use everything above (except where noted) in the current releases available for download. If you are looking forward to using 5.0 (available in beta form today, in full release Real Soon Now), then I'll mention a couple of tweaks to the above:

• VolatileImage APIs: It will be possible in 1.5 to create a VolatileImage with a transparency parameter. Developers have been asking for that capability for some time; the original createVolatileImage() methods only allowed creation of opaque VolatileImages.
  That's the good news: the API is there. The bad news is that there is no hardware acceleration for these non-opaque images by default because our mechanisms for accelerating these types of images are still not ready for prime time. Translucent volatile images can achieve hardware acceleration by using the handy -Dsun.java2d.translaccel=true flag.
• Managed images everywhere! You will now get managed images by calling most of our image creation and loading mechanisms. In jdk1.4.*, only some of the image creation methods (such as Component.createImage() and GraphicsConfiguration.createCompatibleImage()) returned managed images. But in 1.5, you can even call new BufferedImage(...) with a type that is not in the screen bit depth and end up with an accelerated managed image.
• OpenGL Acceleration: The 5.0 release has an OpenGL-based rendering pipeline available on all platforms. This pipeline is not be enabled by default (due to various concerns including graphics hardware and driver issues), but there is a flag that you can use to experiment with it (sun.java2d.opengl=true). This pipeline provides our most comprehensive hardware acceleration so far on all our supported platforms, so it should be pretty exciting to see this one in action.

ImageIO: Just another example of better living by doing it yourself

I've always known that ImageIO was a good thing to use since its inception. It reads and writes more formats than the original Image loading APIs, it has a pluggable interface for new image formats, it's the way of the future, it's more robust than the previous APIs, it's synchronous without the need for MediaTracker or that hacky ImageIcon workaround, blah, blah, blah....

I knew all that without actually working with it because it was apparent from the API. It is the way of the future for image reading and writing, and it is more robust and all the rest of that stuff. It's just a great full-featured package that's written from the ground up to provide what developers have been asking us to provide after using the old Image APIs for years.

But I'd never actually used the darned package...

(Petty justification: The folks working on the libraries for Java don't usually get enough time to play around with using the APIs; we're too busy making the stuff work. That's not to say that the people who wrote ImageIO didn't use it substantially in implementing it; of course they did. But I wasn't involved in that part of the API, and I'm busy enough in my little rendering and performance closet that I didn't take the opportunity to play around with ImageIO since it first came out.)

This all changed in the last month.

See, JavaOne was coming, which meant that I had a few demos to write. And it turns out I needed a couple of image utilities, and I needed them quickly. For both utilities, I began thinking "Now where could I find some simple program to do this ..." and then I realized how easy it would be to accomplish it from scratch in ImageIO. A few lines of code later and, presto!, I had my applications.

Not only did I get exactly the functionality that I was looking for, but I got the satisfaction of actually having written the code myself, which is so darned gratifying to us geeks. As they say, "Just another example of better living by doing it yourself." Or at least that's what I say.

Anyway, on with the code.

ImageScaler

The first utility came from a need to scale some images. We had a demo that was going to show images in a particular size all the time. Rather than scaling them on the fly or caching the scaled versions in the application, we figured it would be better to simply load images that were already of the size needed.

Knowing how these projects tend to go, what I really wanted was the ability to scale our full-size images (always keep the original full-size versions of media around...) down to some arbitrary size. And then, when the designers changed their minds about the size at the last minute, be able to quickly re-scale the images to some other arbitrary size.

We could do this through some image manipulation program, but it would be a major hassle to keep doing it (including having to go through the person that knew the program that ran the macro that ... you get the picture). So what I wanted was a simple standalone application to scale the images.

What I wrote was ImageScaler. This application takes the pathname to a directory (or uses "." as the default) and width/height values. It reads in all image files from that directory, creates a subdirectory ("scaled/"), and writes out scaled versions of those images in the specified width and height.

Here's the code:

package jcg.lhdemo;

import java.awt.*;
import java.awt.image.*;
import javax.imageio.ImageIO;
import java.io.File;

/**
 * ImageScaler
 *
 * This class loads all images in a given directory and scales them to the
 * given sizes, saving the results as JPEG files in a new "scaled/" subdirectory
 * of the original directory.
 */
public class ImageScaler {
    
    // Default w/h values; overriden by command-line -width/-height parameters
    static int IMAGE_W = 150;
    static int IMAGE_H = 250;
    
    public static void main(String args[]) {
        // Default directory is current directory, overridden by -dir parameter
        String imagesDir = ".";
        for (int i = 0; i < args.length; ++i) {
            if (args[i].equals("-dir") && ((i + 1) < args.length)) {
                imagesDir = args[++i];
            } else if (args[i].equals("-width") && ((i + 1) < args.length)) {
                IMAGE_W = Integer.parseInt(args[++i]);
            } else if (args[i].equals("-height") && ((i + 1) < args.length)) {
                IMAGE_H = Integer.parseInt(args[++i]);
            }
        }
        // new subdirectory for scaled images
        String scaledImagesDir = imagesDir + File.separator + "scaled";
        // directory that holds original images
        File cwd = new File(imagesDir);
        // directory for scaled images
        File subdir = new File(scaledImagesDir);
        subdir.mkdir();
        File files[] = cwd.listFiles();
        // temporary image for every scaled instance
        BufferedImage scaledImg = new BufferedImage(IMAGE_W, IMAGE_H, 
                                                    BufferedImage.TYPE_INT_RGB);
        Graphics2D gScaledImg = scaledImg.createGraphics();
        // Note the use of BILNEAR filtering to enable smooth scaling
        gScaledImg.setRenderingHint(RenderingHints.KEY_INTERPOLATION,
                                    RenderingHints.VALUE_INTERPOLATION_BILINEAR);
        for (int i = 0; i < files.length; ++i) {
            try {
                // For every file in the directory, assume it's an image and
                // load it
                BufferedImage img = ImageIO.read(files[i]);
                // If we get here, we must have read the image file successfully.
                // Create a new File in the scaled subdirectory
                File scaledImgFile = new File(scaledImagesDir + File.separator + 
                                              files[i].getName());
                // Scale the original image into the temporary image
                gScaledImg.drawImage(img, 0, 0, IMAGE_W, IMAGE_H, null);
                // Save the scaled version out to the file
                ImageIO.write(scaledImg, "jpeg", scaledImgFile);
            } catch (Exception e) {
                System.out.println("Problem with " + files[i]);
            }
        }
    }
}

• It assumes that the images are in JPEG format. Actually, it does read in the files in any format, but it writes them out in JPEG format with the original filenames, so if the images weren't JPEG to begin with, things will get a bit confusing. We could, of course, use any format that ImageIO supports (or any for which we have plugins); in this case I knew that we were dealing with JPEG images so I made that simplifying assumption in the code. The code could be easily extended to handle other or multiple formats instead.
• Note the use of the BILINEAR rendering hint to the Graphics2D object; this tells Java 2D to do some smooth filtering on the image to make the scaled version look better than it would by using the default NEAREST_NEIGHBOR method. NEAREST_NEIGHBOR works great for many purposes (and is faster in general), but if you're trying to get higher quality and don't mind waiting just a tad longer for the result, use BILINEAR. And for those developers using jdk 5.0, note that the BICUBIC hint now works; you may get even better results with that value.
• Aspect ratio: This code assumes that the original images are in the same aspect ratio as that determined by the width/height that we are scaling to. This may not be the case in any given situation, and might cause unattractive artifacts with widely differing aspect ratios. A more robust application could provide the ability to preserve the aspect ratio of the original instead of using the width/height parameters blindly.
• A batch processing application like ImageScaler which acts on several files in the same format in a loop might be more performant by locating the appropriate reader and writer once, outside the for() loop, and then just calling the appropriate read() or write() functions inside the loop. I'll leave this as an exercise for the reader....

JpegConverter

A couple of weeks after I wrote and used ImageScaler, I ran into another problem where I needed a quick conversion utility. I had a series of BMP images that I wanted to convert to JPEG (in this case, because I needed a compressed image format). Once again I found myself writing a very quick program using ImageIO that did the job. In fact, I probably wrote and ran the program in less time than it would have taken me to find an appropriate utility out on the net, install it, and run it. And, once again, "it's another example of better living by doing it yourself".

This program takes a pointer to a directory (or uses "." by default), reads in all BMP image files in that directory (I constrained JpegConverter to convert only BMP files, but, like ImageScaler, this application could be easily extended to handle multiple formats) and then saves out a JPEG version of each file.

Here's the code:

import java.awt.*;
import java.awt.image.*;
import javax.imageio.ImageIO;
import java.io.File;

/**
 * JpegConverter
 *
 * This class loads all BMP images in a given directory and saves each as a JPEG
 * file in the same directory.  This code is specific to BMP, but it could be
 * easily extended to read images of any type that ImageIO handles.
 */
public class JpegConverter {
    
    public static void main(String args[]) {
        // Default directory is current directory, overridden by -dir parameter
        String imagesDir = ".";
        for (int i = 0; i < args.length; ++i) {
            if (args[i].equals("-dir") && ((i + 1) < args.length)) {
                imagesDir = args[++i];
            }
        }
        // directory that holds original images
        File cwd = new File(imagesDir);
        File files[] = cwd.listFiles();
        for (int i = 0; i < files.length; ++i) {
            String fileName = files[i].getName();
            // converstion to lower case just for ease of replacing the
            // filename extension
            String fileNameLC = fileName.toLowerCase();
            if (fileName.endsWith("bmp")) {
                try {
                    // Replace original "bmp" filename extension with "jpg"
                    int extensionIndex = fileNameLC.lastIndexOf("bmp");
                    String fileNameBase = fileName.substring(0, extensionIndex);
                    BufferedImage img = ImageIO.read(files[i]);
                    // create new JPEG file
                    File convertedImgFile = 
                            new File(imagesDir + File.separator + 
                                     fileNameBase + "jpg");
                    // store original file out in JPEG format
                    ImageIO.write(img, "jpeg", convertedImgFile);
                } catch (Exception e) {
                    System.out.println("Problem with " + files[i]);
                }
            }
        }
    }
}

Notes:

• Some of the notes for ImageScaler above apply to JpegConverter as well, so consider this note as a dereference to the notes above...

And so, in conclusion

Some readers might wonder: "What's the big deal?" After all, these applications are pretty simple, so why all the fuss?

Exactly my point; thanks for clarifying it. It is easy. So easy that I would encourage everyone to think about using ImageIO next time you have a simple (or hard) image input/output problem come up; see if the ImageIO API doesn't make solving that problem just a tad easier.

VolatileImage Q&A

I originally wrote this as one single blog, but as I found out with my previous BufferedImage articles, I tend to go on much longer than I inteded at first, so I had to split it up. So now there's 2 parts to this blog: last time's VolatileImage usage discussion, and this week's VolatileImage FAQ. This week, I thought I'd write down various reasonable questions about either VolatileImage objects and usage in general, or questions that might arise from last week's example. These are questions that come up all the time, or that might reasonably surface from the discussion so far. If you have questions of your own, feel free to post them in the talk-back section at the end.

First, let's see that sample code again:

import java.awt.Color;
import java.awt.Component;
import java.awt.Frame;
import java.awt.Graphics;
import java.awt.image.VolatileImage;

public class VImageDemo extends Component {
	
    VolatileImage backBuffer = null;

    void createBackBuffer() {
	if (backBuffer != null) {
	    backBuffer.flush();
	    backBuffer = null;
	}
	backBuffer = createVolatileImage(getWidth(), getHeight());
    }

    public void paint(Graphics g) {
	if (backBuffer == null) {
	    createBackBuffer();
	}
	do {
	    // First, we validate the back buffer
	    int valCode = backBuffer.validate(getGraphicsConfiguration());
	    if (valCode == VolatileImage.IMAGE_RESTORED) {
		// This case is just here for illustration
		// purposes.  Since we are
		// recreating the contents of the back buffer
		// every time through this loop, we actually
		// do not need to do anything here to recreate
		// the contents.  If our VImage was an image that
		// we were going to be copying _from_, then we
		// would need to restore the contents at this point
	    } else if (valCode == VolatileImage.IMAGE_INCOMPATIBLE) {
		createBackBuffer();
	    }
	    // Now we've handled validation, get on with the rendering

	    //
	    // rendering to the back buffer:
	    Graphics gBB = backBuffer.getGraphics();
	    gBB.setColor(Color.white);
	    gBB.fillRect(0, 0, getWidth(), getHeight());
	    gBB.setColor(Color.red);
	    gBB.drawLine(0, 0, getWidth(), getHeight());

	    // copy from the back buffer to the screen
	    g.drawImage(backBuffer, 0, 0, this);

	    // Now we are done; or are we?  Check contentsLost() and loop as necessary
	} while (backBuffer.contentsLost());
    }

    public static void main(String args[]) {
	Frame f = new Frame();
	f.setSize(500, 500);
	f.add(new VImageDemo());
	f.setVisible(true);
    }
}

And now, on with the questions:

Q: So when do I use VolatileImage objects? Should I use them instead of BufferedImages, or managed images?
A: Ah yes, the plethora of image types and the confusion that quickly sets in. The quick answer is "You probably do not need to ever use a VolatileImage object directly in your application." This answer assumes that most applications:

• Are Swing apps, and therefore are already using a VolatileImage back buffer because Swing does it for them
• Do not need or want the extra complexity inherent in using these images that must be managed.
• Do not need every ounce of performance and runtime details available

The main reason for anyone to ever use a VolatileImage object is when you need a dynamic (frequently updated) image that you would really like to be accelerated if possible, including both rendering to the image and copying from the image. This is the basic usage model of a back buffer; you want it accelerated and you are rendering to the image frequently (at least once per screen update, by definition). This is why Swing uses a VolatileImage for its back buffer.

Using a BufferedImage or a managed image in this kind of scenario will instead give you software-rendering performance. Because of the volatile nature of accelerated memory (particularly on Windows), any image that is not explicitly managed for loss (which means any image that is not a VolatileImage) must reside primarly in system memory. That way, any rendering operation to that image is captured in that system memory version and cannot be destroyed through the loss of an accelerated version of the image. This implies that all rendering to the image must occur first through software rendering loops (since we cannot use hardware acceleration to render to a system memory image). Then, if it makes sense, we may copy that image into accelerated memory for future copies from that image. There are two main problems with this for a back-buffer scenario:

• Performance: for simple rendering, hardware acceleration will generally win out over software rendering performance. There are various reasons for this, including the speed of the graphics processors, the inherent parallelism of offloading operations to the graphics chip, etc. But in the end, just remember that's why they call it "accelerated". So if your back buffer resides in system memory (if you are not using a VolatileImage), then you must make do with software rendering performance.
• Performance: Although the render-to performance is important for the back buffer, sometimes the bigger win comes in the simple copy-to-screen operation that back buffers do on every frame. Ideally, this copy happens between an accelerated back buffer living in VRAM to the screen, also living in VRAM, and can happen at amazing speeds due to the memory bandwidth in the graphics system. But if your back buffer lives in system memory, this copy must happen over the system bus: reading from system memory (hopefully the on-chip cache, at least) and copying to VRAM down the PCI or AGP bus. Imagine trying to render 60 frames per second in an animation application, with a screen size of 1280x1024 and 32bpp screen depth; that's about 5 megs of data per frame * 60 frames = 300 MBytes of data moving across the bus, just to copy the buffer to the screen, not including anything else that needs to happen over the bus or through the CPU/cache/memory. "But what if the back buffer is a managed image?" Unfortunately, this is no better than an un-managed image; because you are rendering to the back buffer every frame, there is no benefit to our copying that buffer down to a VRAM cached version; we'd have to do it every frame (since the contents of the primary version of the image are being updated every frame), which is no better than simply copying it directly to the screen every frame.

Q: Couldn't the contenstLost() rendering loop become an infinite loop?
A: No. At least, not in theory. validate() will attempt to allocate an accelerated surface. Failing that, it will allocate the surface memory in system memory. So when you get either an IMAGE_OK or and IMAGE_RESTORED, you can guarantee that there is something there to render to (and from). And if you get an INCOMPATIBLE error, you must recreate the image completely, which again will result in at least something you can render to and from. So if nothing happens to that surface between that validate and the contentsLost(), then contentsLost() will return false and you can continue. The only situation I can foresee where you might get stuck in this loop more than one time around would be some weird system thrashing case where we are able to allocate VRAM, but then it's clobbered by the time we get around to the end of the loop, then we can get the memory again by the next validate, then it gets clobbered, and so on; this is pretty unlikely in reality, and if this is really happening on the system, there are probably more disturbing things to worry about than spinning in this rendering loop too many times.

Q: What's with the extra null-check condition in createBackBuffer()?
A: This is an optimization tweak for those interested. There are a couple of things going on here:

	backBuffer.flush();

The second item is this condition is even tweakier:

	backBuffer = null;

Let us say that the operation of assigning the new object to backBuffer causes the collector to run to try to get more memory. It can only get memory from objects that are already dead; but since the old backBuffer object is still alive until this reassignment is done, that object is not available for collection.

As I said, this extra step is somewhat useless here; since we already nuked the video memory taken up by this image by using the more definitive and direct approach of calling flush(). The only possible memory savings here is that Java heap memory taken up by the image data structure. This is not enough to worry about. But I brought it up because I have run into problems with this in other image contexts. For example, let's say (for some unknown and perhaps suspect reason) that our rendering loop creates a new BufferedImage every time through the loop for some intermediate calculation. Images can get pretty big, so this one object can be enough to swamp the young generation of the collector; but if we have to have 2 of these objects live simultaneously, then there's a fair chance that we're going to start chucking objects into the tenured generation and thus requiring Full (aka, "expensive") GC operations to get that memory back. Much better to get rid of the old one before asking for another one.

I don't want to get too deeply into GarbageCollection issues and implementations (and I think I've reached the limit of my knowledge on the subject in any case), but I did want to mention the cases above because as graphics and image users, we tend to run into unique garbage and memory situation, so it's worth understanding more about this stuff.

Example time: Let's say we have one VImage sprite and a VIimage backBuffer. We could write a buggy rendering loop like this:

		int valCode = backBuffer.validate(comp.getGraphicsConfig());
		if (valCode == VolatileImage.IMAGE_RESTORED) {
		    // backBuffer does not need to be restored,
		    // but sprite does - initContents() is a method
		    // that re-renders the contents into the sprite
		    initSpriteContents();
		} else if (valCode == VolatileImage.IMAGE_INCOMPATIBLE) {
		    // recreate all volatile images
		    createBackBuffer();
		    createSprite();
		    // also restore contents of sprite since we now have a
		    // completely new image
		    initSpriteContents();
		}
		// rendering to the back buffer:
		Graphics gBB = backBuffer.getGraphics();
		gBB.drawImage(sprite, 0, 0, this);
		
		// copy from the back buffer to the screen
		g.drawImage(backBuffer, 0, 0, this);
		
		// Now we are done; or are we?  Check contentsLost() and loop as necessary
	    } while (!backBuffer.contentsLost());

Here's the problem: images only get reinstated (put into a reasonable state to render to/from) when you call validate() on that image. So if you get a IMAGE_RESTORED value from backBuffer.validate(), then chances are pretty good that the sprite is also lost ... and is staying lost. The code above calls initSpriteContents() to re-render the contents of the sprite, but these operations will fail because the sprite is not in a state that will accept rendering.

The single contentsLost() call will probably work for now, but as I said above, if any images are every selectively punted that could change and the above code would be wrong under the mythical future implementation.

A more correct version of the above looks something like this:

		int valCodeBB = backBuffer.validate(comp.getGraphicsConfig());
		int valCodeSprite = sprite.validate(comp.getGraphicsConfig());
		if (valCodeSprite == VolatileImage.IMAGE_RESTORED) {
		    // note that we do not care about the back buffer
		    // for restoration since the buffer will be 
		    // re-rendered here anyway.
		    initSpriteContents();
		}
		if (valCodeBuffer == VolatileImage.IMAGE_INCOMPATIBLE) {
		    createBackBuffer();
		}
		if (valCodeSprite == VolatileImage.IMAGE_INCOMPATIBLE) {
		    createSprite();
		    // also restore contents of sprite since we now have a
		    // completely new image
		    initSpriteContents();
		}
		// rendering to the back buffer:
		Graphics gBB = backBuffer.getGraphics();
		gBB.drawImage(sprite, 0, 0, this);
		
		// copy from the back buffer to the screen
		g.drawImage(backBuffer, 0, 0, this);
		
		// Now we are done; or are we?  Check contentsLost() and loop as necessary
	    } while (!backBuffer.contentsLost() && !sprite.contentsLost());

VolatileImage: Now you See it, Now you Don't

Way back when we were first implementing the VolatileImage API, I had asked to come to the Swing staff meeting so that I could explain about the new VolatileImage API and why Swing should start using it.

I got up to the whiteboard, drew some pictures (probably horrible, but it's just a crutch for me when I explain things). I described this new image type, boiling it down to its basics:

1. It's going to be hardware-accelerated (depending on the runtime platform)
2. Rendering-to and Copying-from should be way faster than the current BufferedImage objects
3. It can go away at any time
4. Swing should use it for their back buffer

Then they asked me to go back to point #3; the whole disappearing-image thing. Apparently, they thought I was kidding.

It's true: VolatileImages can (and quite often will) go away and the API for VolatileImage was developed specifically to deal with that problem. All of the loss situations currently occur only on Windows.

You would think that we would actually be notified prior to these problems, so that we could prevent the loss, or backup the image, or something. But you'd be wrong; we only find out from Windows that there is a problem the next time we actually try to use the image. Surface loss situations arise from situations such as:

• Another app going into fullscreen mode
• The display mode changing on your screen (whether caused by your application or the user)
• TaskManager being run
• A screensaver kicking in
• The system going into or out of StandBy or Hibernate mode
• Your kid just yanked the power cord to the machine

Let's look at how you use the API to manage this situation. Then we'll step back and talk about other issues regarding these lossy beasts.

There are two parts to the API that need to be used in dealing with image loss:

	int valCode = validate(GraphicsConfiguration gc);
	boolean contentsLost();

validate() is used prior to using a VolatileImage. This tells us that you are a