Chet Haase's Blog

September 2003 Archives

VolatileImage Q&A

I originally wrote this as one single blog, but as I found out with my previous BufferedImage articles, I tend to go on much longer than I inteded at first, so I had to split it up. So now there's 2 parts to this blog: last time's VolatileImage usage discussion, and this week's VolatileImage FAQ. This week, I thought I'd write down various reasonable questions about either VolatileImage objects and usage in general, or questions that might arise from last week's example. These are questions that come up all the time, or that might reasonably surface from the discussion so far. If you have questions of your own, feel free to post them in the talk-back section at the end.

First, let's see that sample code again:

import java.awt.Color;
import java.awt.Component;
import java.awt.Frame;
import java.awt.Graphics;
import java.awt.image.VolatileImage;

public class VImageDemo extends Component {
	
    VolatileImage backBuffer = null;

    void createBackBuffer() {
	if (backBuffer != null) {
	    backBuffer.flush();
	    backBuffer = null;
	}
	backBuffer = createVolatileImage(getWidth(), getHeight());
    }

    public void paint(Graphics g) {
	if (backBuffer == null) {
	    createBackBuffer();
	}
	do {
	    // First, we validate the back buffer
	    int valCode = backBuffer.validate(getGraphicsConfiguration());
	    if (valCode == VolatileImage.IMAGE_RESTORED) {
		// This case is just here for illustration
		// purposes.  Since we are
		// recreating the contents of the back buffer
		// every time through this loop, we actually
		// do not need to do anything here to recreate
		// the contents.  If our VImage was an image that
		// we were going to be copying _from_, then we
		// would need to restore the contents at this point
	    } else if (valCode == VolatileImage.IMAGE_INCOMPATIBLE) {
		createBackBuffer();
	    }
	    // Now we've handled validation, get on with the rendering

	    //
	    // rendering to the back buffer:
	    Graphics gBB = backBuffer.getGraphics();
	    gBB.setColor(Color.white);
	    gBB.fillRect(0, 0, getWidth(), getHeight());
	    gBB.setColor(Color.red);
	    gBB.drawLine(0, 0, getWidth(), getHeight());

	    // copy from the back buffer to the screen
	    g.drawImage(backBuffer, 0, 0, this);

	    // Now we are done; or are we?  Check contentsLost() and loop as necessary
	} while (backBuffer.contentsLost());
    }

    public static void main(String args[]) {
	Frame f = new Frame();
	f.setSize(500, 500);
	f.add(new VImageDemo());
	f.setVisible(true);
    }
}

And now, on with the questions:

Q: So when do I use VolatileImage objects? Should I use them instead of BufferedImages, or managed images?
A: Ah yes, the plethora of image types and the confusion that quickly sets in. The quick answer is "You probably do not need to ever use a VolatileImage object directly in your application." This answer assumes that most applications:

• Are Swing apps, and therefore are already using a VolatileImage back buffer because Swing does it for them
• Do not need or want the extra complexity inherent in using these images that must be managed.
• Do not need every ounce of performance and runtime details available

The main reason for anyone to ever use a VolatileImage object is when you need a dynamic (frequently updated) image that you would really like to be accelerated if possible, including both rendering to the image and copying from the image. This is the basic usage model of a back buffer; you want it accelerated and you are rendering to the image frequently (at least once per screen update, by definition). This is why Swing uses a VolatileImage for its back buffer.

Using a BufferedImage or a managed image in this kind of scenario will instead give you software-rendering performance. Because of the volatile nature of accelerated memory (particularly on Windows), any image that is not explicitly managed for loss (which means any image that is not a VolatileImage) must reside primarly in system memory. That way, any rendering operation to that image is captured in that system memory version and cannot be destroyed through the loss of an accelerated version of the image. This implies that all rendering to the image must occur first through software rendering loops (since we cannot use hardware acceleration to render to a system memory image). Then, if it makes sense, we may copy that image into accelerated memory for future copies from that image. There are two main problems with this for a back-buffer scenario:

• Performance: for simple rendering, hardware acceleration will generally win out over software rendering performance. There are various reasons for this, including the speed of the graphics processors, the inherent parallelism of offloading operations to the graphics chip, etc. But in the end, just remember that's why they call it "accelerated". So if your back buffer resides in system memory (if you are not using a VolatileImage), then you must make do with software rendering performance.
• Performance: Although the render-to performance is important for the back buffer, sometimes the bigger win comes in the simple copy-to-screen operation that back buffers do on every frame. Ideally, this copy happens between an accelerated back buffer living in VRAM to the screen, also living in VRAM, and can happen at amazing speeds due to the memory bandwidth in the graphics system. But if your back buffer lives in system memory, this copy must happen over the system bus: reading from system memory (hopefully the on-chip cache, at least) and copying to VRAM down the PCI or AGP bus. Imagine trying to render 60 frames per second in an animation application, with a screen size of 1280x1024 and 32bpp screen depth; that's about 5 megs of data per frame * 60 frames = 300 MBytes of data moving across the bus, just to copy the buffer to the screen, not including anything else that needs to happen over the bus or through the CPU/cache/memory. "But what if the back buffer is a managed image?" Unfortunately, this is no better than an un-managed image; because you are rendering to the back buffer every frame, there is no benefit to our copying that buffer down to a VRAM cached version; we'd have to do it every frame (since the contents of the primary version of the image are being updated every frame), which is no better than simply copying it directly to the screen every frame.

Q: Couldn't the contenstLost() rendering loop become an infinite loop?
A: No. At least, not in theory. validate() will attempt to allocate an accelerated surface. Failing that, it will allocate the surface memory in system memory. So when you get either an IMAGE_OK or and IMAGE_RESTORED, you can guarantee that there is something there to render to (and from). And if you get an INCOMPATIBLE error, you must recreate the image completely, which again will result in at least something you can render to and from. So if nothing happens to that surface between that validate and the contentsLost(), then contentsLost() will return false and you can continue. The only situation I can foresee where you might get stuck in this loop more than one time around would be some weird system thrashing case where we are able to allocate VRAM, but then it's clobbered by the time we get around to the end of the loop, then we can get the memory again by the next validate, then it gets clobbered, and so on; this is pretty unlikely in reality, and if this is really happening on the system, there are probably more disturbing things to worry about than spinning in this rendering loop too many times.

Q: What's with the extra null-check condition in createBackBuffer()?
A: This is an optimization tweak for those interested. There are a couple of things going on here:

	backBuffer.flush();

The second item is this condition is even tweakier:

	backBuffer = null;

Let us say that the operation of assigning the new object to backBuffer causes the collector to run to try to get more memory. It can only get memory from objects that are already dead; but since the old backBuffer object is still alive until this reassignment is done, that object is not available for collection.

As I said, this extra step is somewhat useless here; since we already nuked the video memory taken up by this image by using the more definitive and direct approach of calling flush(). The only possible memory savings here is that Java heap memory taken up by the image data structure. This is not enough to worry about. But I brought it up because I have run into problems with this in other image contexts. For example, let's say (for some unknown and perhaps suspect reason) that our rendering loop creates a new BufferedImage every time through the loop for some intermediate calculation. Images can get pretty big, so this one object can be enough to swamp the young generation of the collector; but if we have to have 2 of these objects live simultaneously, then there's a fair chance that we're going to start chucking objects into the tenured generation and thus requiring Full (aka, "expensive") GC operations to get that memory back. Much better to get rid of the old one before asking for another one.

I don't want to get too deeply into GarbageCollection issues and implementations (and I think I've reached the limit of my knowledge on the subject in any case), but I did want to mention the cases above because as graphics and image users, we tend to run into unique garbage and memory situation, so it's worth understanding more about this stuff.

Example time: Let's say we have one VImage sprite and a VIimage backBuffer. We could write a buggy rendering loop like this:

		int valCode = backBuffer.validate(comp.getGraphicsConfig());
		if (valCode == VolatileImage.IMAGE_RESTORED) {
		    // backBuffer does not need to be restored,
		    // but sprite does - initContents() is a method
		    // that re-renders the contents into the sprite
		    initSpriteContents();
		} else if (valCode == VolatileImage.IMAGE_INCOMPATIBLE) {
		    // recreate all volatile images
		    createBackBuffer();
		    createSprite();
		    // also restore contents of sprite since we now have a
		    // completely new image
		    initSpriteContents();
		}
		// rendering to the back buffer:
		Graphics gBB = backBuffer.getGraphics();
		gBB.drawImage(sprite, 0, 0, this);
		
		// copy from the back buffer to the screen
		g.drawImage(backBuffer, 0, 0, this);
		
		// Now we are done; or are we?  Check contentsLost() and loop as necessary
	    } while (!backBuffer.contentsLost());

Here's the problem: images only get reinstated (put into a reasonable state to render to/from) when you call validate() on that image. So if you get a IMAGE_RESTORED value from backBuffer.validate(), then chances are pretty good that the sprite is also lost ... and is staying lost. The code above calls initSpriteContents() to re-render the contents of the sprite, but these operations will fail because the sprite is not in a state that will accept rendering.

The single contentsLost() call will probably work for now, but as I said above, if any images are every selectively punted that could change and the above code would be wrong under the mythical future implementation.

A more correct version of the above looks something like this:

		int valCodeBB = backBuffer.validate(comp.getGraphicsConfig());
		int valCodeSprite = sprite.validate(comp.getGraphicsConfig());
		if (valCodeSprite == VolatileImage.IMAGE_RESTORED) {
		    // note that we do not care about the back buffer
		    // for restoration since the buffer will be 
		    // re-rendered here anyway.
		    initSpriteContents();
		}
		if (valCodeBuffer == VolatileImage.IMAGE_INCOMPATIBLE) {
		    createBackBuffer();
		}
		if (valCodeSprite == VolatileImage.IMAGE_INCOMPATIBLE) {
		    createSprite();
		    // also restore contents of sprite since we now have a
		    // completely new image
		    initSpriteContents();
		}
		// rendering to the back buffer:
		Graphics gBB = backBuffer.getGraphics();
		gBB.drawImage(sprite, 0, 0, this);
		
		// copy from the back buffer to the screen
		g.drawImage(backBuffer, 0, 0, this);
		
		// Now we are done; or are we?  Check contentsLost() and loop as necessary
	    } while (!backBuffer.contentsLost() && !sprite.contentsLost());

VolatileImage: Now you See it, Now you Don't

Way back when we were first implementing the VolatileImage API, I had asked to come to the Swing staff meeting so that I could explain about the new VolatileImage API and why Swing should start using it.

I got up to the whiteboard, drew some pictures (probably horrible, but it's just a crutch for me when I explain things). I described this new image type, boiling it down to its basics:

1. It's going to be hardware-accelerated (depending on the runtime platform)
2. Rendering-to and Copying-from should be way faster than the current BufferedImage objects
3. It can go away at any time
4. Swing should use it for their back buffer

Then they asked me to go back to point #3; the whole disappearing-image thing. Apparently, they thought I was kidding.

It's true: VolatileImages can (and quite often will) go away and the API for VolatileImage was developed specifically to deal with that problem. All of the loss situations currently occur only on Windows.

You would think that we would actually be notified prior to these problems, so that we could prevent the loss, or backup the image, or something. But you'd be wrong; we only find out from Windows that there is a problem the next time we actually try to use the image. Surface loss situations arise from situations such as:

• Another app going into fullscreen mode
• The display mode changing on your screen (whether caused by your application or the user)
• TaskManager being run
• A screensaver kicking in
• The system going into or out of StandBy or Hibernate mode
• Your kid just yanked the power cord to the machine

Let's look at how you use the API to manage this situation. Then we'll step back and talk about other issues regarding these lossy beasts.

There are two parts to the API that need to be used in dealing with image loss:

	int valCode = validate(GraphicsConfiguration gc);
	boolean contentsLost();

validate() is used prior to using a VolatileImage. This tells us that you are about to use the image, so we need to make sure that it is in a state ready to be used. In most cases, all we do is check that things are fine and return IMAGE_OK. But in some situations, we need to do more:

• SurfaceLoss: One of the situations above that cause surface loss may have occurred since the last time you called validate(), in which case the underlying native surface memory is no longer there. In this case, validate() tells us to reallocate that memory or, failing that (there may not be enough VRAM, or some other failure might occur), allocate something in system memory that can also do the job. One of these is guaranteed to succeed and we will then return IMAGE_RESTORED from validate.
• Surface incompatibility: the reason that we take a GraphicsConfiguration argument to validate is that we need to know what you intend to do with the image before we can do the right thing to validate it. For example, you may have a dual-head system. Perhaps you created the VolatileImage on the primary graphics device. Later, you wish to copy that image to a Component on the secondary device. Surfaces are stored in graphics-device-specific memory, so we cannot copy an image from the VRAM on one device to another device. Or if we can, you probably don't want us to, because it would be a very slow operation. Instead, we ask you to tell us the target device for the operation and then we will tell you whether this is possible for the given image. If the GraphicsConfiguration that you pass in is compatible with the one that the image was created with, then we return IMAGE_OK, otherwise we return IMAGE_INCOMPATIBLE.

When you get a RESTORED error, you know that the image was in a funky state, but has now been reinstated and can be used. If you are about to write to the image and clobber the contents (as you would a back buffer, for example), this is not an error that you need to do anything about; the image memory exists and that's all you need to know. But if it's an image that you are copying from, or in some other way depending on the contents, then you need to restore those contents prior to depending on them.

For example, let's say you have an icon stored in a VolatileImage. At some point in your app, the image memory gets lost. The next time you go to copy from that image to the back buffer, you get the RESTORED code. Before doing the copy that you wanted to do, you need to recreate the icon using whatever means you need to do do that.

When you get an INCOMPATIBLE error, you know that your image is not in an appropriate state to do whatever operation you want to do with it. So, for example, if you are about to copy that image to a window with the GraphicsConfiguration that you passed in, that operation may not succeed. The only way to handle this error is to simply recreate the image on the new GraphicsConfiguration.

Now that you know what to do prior to using your image (call validate() and handle the return code appropriately), let's look at what you do afterwards. ("Afterwards? I'm done rendering, aren't I? How much do I have to do here? I thought Java was supposed to be easy, dangit!")

Notice that you had to call validate() prior to doing any rendering in case anything happened to your image since the last time you used it. But what happens if something happened to your image during your use of it? You might just want to know that some failure occurred. For example, let's say you are using a VolatileImage for a back buffer in some double-buffered application. You're in your rendering loop, you've called and handled the validate() call for the buffer, and you've done all the rendering to that buffer, and then copied the buffer to the screen. You're done, right?

Nope; what about if your image became lost during either the rendering to that buffer or the rendering from the buffer to the screen? Then chances are pretty good that either the copy to the screen simply didn't happen, or that some of the rendering either to or from that buffer is undefined. Do you want to just let it go and catch it later? Probably not, especially if this is some static application that only renders its contents based on changes in the app (versus some animation that's rendering one frame after another constantly); you don't want garbage hanging out on the screen or some required paint event just dropped on the floor. Instead, you want to make sure you get the right bits to the user as soon as possible.

This is where contentsLost() comes in; at the end of your rendering loop (or wherever you are doing rendering operations to and from your VolatileImage(s), you need to call contentsLost(). The boolean return value from this function will tell you whether the rendering operations from the iamge in question happened successfully, or whether anything went amiss during rendering. If the function returns false, everything is cool and you can just continue about your normal routine. if the function returns true, this means that the contents on that image are undefined and you need to:

• re-validate() that image
• redo whatever rendering you had just done to make sure it gets done correctly

So these are the two crucial pieces of the VolatileImage puzzle: you call validate() before you do anything with the image and contentsLost() after you are supposedly finished. And based on the return values from those functions, you do some other stuff to handle the errors.

We can put this into a simple loop which should probably look very similar in most situations. At first, I wrote some code that just showed the inner rendering loop. But to avoid confusion and allow easy copy/paste/compile/test, I've posted an entire (and functional) test app below. Sure it's boring (unless you really dig apps that draw single lines to the screen), but it shows the basic VolatileImage management we're after:

import java.awt.Color;
import java.awt.Component;
import java.awt.Frame;
import java.awt.Graphics;
import java.awt.image.VolatileImage;

public class VImageDemo extends Component {
	
    VolatileImage backBuffer = null;

    void createBackBuffer() {
	if (backBuffer != null) {
	    backBuffer.flush();
	    backBuffer = null;
	}
	backBuffer = createVolatileImage(getWidth(), getHeight());
    }

    public void paint(Graphics g) {
	if (backBuffer == null) {
	    createBackBuffer();
	}
	do {
	    // First, we validate the back buffer
	    int valCode = backBuffer.validate(getGraphicsConfiguration());
	    if (valCode == VolatileImage.IMAGE_RESTORED) {
		// This case is just here for illustration
		// purposes.  Since we are
		// recreating the contents of the back buffer
		// every time through this loop, we actually
		// do not need to do anything here to recreate
		// the contents.  If our VImage was an image that
		// we were going to be copying _from_, then we
		// would need to restore the contents at this point
	    } else if (valCode == VolatileImage.IMAGE_INCOMPATIBLE) {
		createBackBuffer();
	    }
	    // Now we've handled validation, get on with the rendering

	    //
	    // rendering to the back buffer:
	    Graphics gBB = backBuffer.getGraphics();
	    gBB.setColor(Color.white);
	    gBB.fillRect(0, 0, getWidth(), getHeight());
	    gBB.setColor(Color.red);
	    gBB.drawLine(0, 0, getWidth(), getHeight());

	    // copy from the back buffer to the screen
	    g.drawImage(backBuffer, 0, 0, this);

	    // Now we are done; or are we?  Check contentsLost() and loop as necessary
	} while (backBuffer.contentsLost());
    }

    public static void main(String args[]) {
	Frame f = new Frame();
	f.setSize(500, 500);
	f.add(new VImageDemo());
	f.setVisible(true);
    }
}

Stay tuned for my next blog; now that we've gone over the basics in how to use the VolatileImage API, we'll address some questions that you may have about using VolatileImages, or about this sample code in particular.

This work is licensed under a Creative Commons License.