Skip to main content

VolatileImage Q&A

Posted by chet on September 17, 2003 at 7:57 AM PDT

I originally wrote this as one single blog, but as I found out with
my previous BufferedImage articles, I tend to go on much longer
than I inteded at first, so I had to split it up. So now there's
2 parts to this blog: last time's VolatileImage usage discussion,
and this week's VolatileImage FAQ. This week, I thought I'd write
down various reasonable questions about either VolatileImage
objects and usage in general, or questions that might arise from
last week's example. These are questions that come up all the time,
or that might reasonably surface from the discussion so far. If you have questions
of your own, feel free to post them in the talk-back section at the end.

First, let's see that sample code again:


import java.awt.Color;
import java.awt.Component;
import java.awt.Frame;
import java.awt.Graphics;
import java.awt.image.VolatileImage;

public class VImageDemo extends Component {

    VolatileImage backBuffer = null;

    void createBackBuffer() {
if (backBuffer != null) {
    backBuffer.flush();
    backBuffer = null;
}
backBuffer = createVolatileImage(getWidth(), getHeight());
    }

    public void paint(Graphics g) {
if (backBuffer == null) {
    createBackBuffer();
}
do {
    // First, we validate the back buffer
    int valCode = backBuffer.validate(getGraphicsConfiguration());
    if (valCode == VolatileImage.IMAGE_RESTORED) {
// This case is just here for illustration
// purposes.  Since we are
// recreating the contents of the back buffer
// every time through this loop, we actually
// do not need to do anything here to recreate
// the contents.  If our VImage was an image that
// we were going to be copying _from_, then we
// would need to restore the contents at this point
    } else if (valCode == VolatileImage.IMAGE_INCOMPATIBLE) {
createBackBuffer();
    }
    // Now we've handled validation, get on with the rendering

    //
    // rendering to the back buffer:
    Graphics gBB = backBuffer.getGraphics();
    gBB.setColor(Color.white);
    gBB.fillRect(0, 0, getWidth(), getHeight());
    gBB.setColor(Color.red);
    gBB.drawLine(0, 0, getWidth(), getHeight());

    // copy from the back buffer to the screen
    g.drawImage(backBuffer, 0, 0, this);

    // Now we are done; or are we?  Check contentsLost() and loop as necessary
} while (backBuffer.contentsLost());
    }

    public static void main(String args[]) {
Frame f = new Frame();
f.setSize(500, 500);
f.add(new VImageDemo());
f.setVisible(true);
    }
}

And now, on with the questions:


Q: So when do I use VolatileImage objects? Should I use them instead of
BufferedImages, or managed images?



A: Ah yes, the plethora of image types and the confusion that
quickly sets in. The quick answer is "You probably do not need
to ever use a VolatileImage object directly in your application."
This answer assumes that most applications:

  • Are Swing apps, and therefore are already using a VolatileImage
    back buffer because Swing does it for them

  • Do not need or want the extra complexity inherent in using these
    images that must be managed.

  • Do not need every ounce of performance and runtime details
    available

So much for the quick answer. The long and much more descriptive and rewarding answer is "You might want to use VolatileImage objects sometimes."
(Actually, this long answer is much shorter than the quick answer
above, but you'll see that the underlying explanation behind this
glib response is not so brief).

The main reason for anyone to ever use a VolatileImage object is
when you need a dynamic (frequently updated) image that you would
really like to be accelerated if possible, including both
rendering to the image and copying from the image. This is the
basic usage model of a back buffer; you want it accelerated and
you are rendering to the image frequently (at least once per
screen update, by definition). This is why Swing uses a VolatileImage
for its back buffer.

Using a BufferedImage or a managed image in this kind of scenario will instead give you software-rendering performance. Because of the volatile nature of accelerated memory (particularly on Windows), any image that is not explicitly managed for loss (which means any image that is not a VolatileImage) must reside primarly in system memory. That way, any rendering operation to that image is captured in that system memory version and cannot be destroyed through the loss of an accelerated version of the image. This implies that all rendering to the image must occur first through software rendering loops (since we cannot use hardware acceleration to render to a system memory image). Then, if it makes sense, we may copy that image into accelerated memory for future copies from that image. There are two main problems with this for a back-buffer scenario:

  • Performance: for simple rendering, hardware acceleration will generally win out over software rendering performance. There are various reasons for this, including the speed of the graphics processors, the inherent parallelism of offloading operations to the graphics chip, etc. But in the end, just remember that's why they call it "accelerated". So if your back buffer resides in system memory (if you are not using a VolatileImage), then you must make do with software rendering performance.
  • Performance: Although the render-to performance is important for the back buffer, sometimes the bigger win comes in the simple copy-to-screen operation that back buffers do on every frame. Ideally, this copy happens between an accelerated back buffer living in VRAM to the screen, also living in VRAM, and can happen at amazing speeds due to the memory bandwidth in the graphics system. But if your back buffer lives in system memory, this copy must happen over the system bus: reading from system memory (hopefully the on-chip cache, at least) and copying to VRAM down the PCI or AGP bus. Imagine trying to render 60 frames per second in an animation application, with a screen size of 1280x1024 and 32bpp screen depth; that's about 5 megs of data per frame * 60 frames = 300 MBytes of data moving across the bus, just to copy the buffer to the screen, not including anything else that needs to happen over the bus or through the CPU/cache/memory. "But what if the back buffer is a managed image?" Unfortunately, this is no better than an un-managed image; because you are rendering to the back buffer every frame, there is no benefit to our copying that buffer down to a VRAM cached version; we'd have to do it every frame (since the contents of the primary version of the image are being updated every frame), which is no better than simply copying it directly to the screen every frame.



Q: Couldn't the contenstLost() rendering loop become an infinite loop?




A: No. At least, not in theory. validate() will attempt to allocate
an accelerated surface. Failing that, it will allocate the surface
memory in system memory. So when you get either an IMAGE_OK or
and IMAGE_RESTORED, you can guarantee that there is something
there to render to (and from). And if you get an INCOMPATIBLE
error, you must recreate the image completely, which again will
result in at least something you can render to and from.
So if nothing happens to that surface between that validate and
the contentsLost(), then contentsLost() will return false and
you can continue. The only situation I can foresee where you
might get stuck in this loop more than one time around would be
some weird system thrashing case where we are able to allocate
VRAM, but then it's clobbered by the time we get around to the
end of the loop, then we can get the memory again by the next
validate, then it gets clobbered, and so on; this is pretty
unlikely in reality, and if this is really happening on the system, there are probably more disturbing things to worry about than spinning in this rendering loop too many times.



Q: What's with the extra null-check condition in createBackBuffer()?




A: This is an optimization tweak for those interested. There
are a couple of things going on here:


backBuffer.flush();

This call tells us to release any memory associated with this image.
This will happen anyway when we reassign backBuffer to a new
value; the old object will eventually be GC'd (Garbage Collected). But the key word
here is "eventually"; we do not know exactly how long it will
take for that GC operation to kick in. When dealing with the
constraint of small and finite video memory, we need to be as
proactive as possible and make sure that we do not have
anything wasting valuable VRAM space. For example, what if there was
space for exactly one back buffer? If we had an old version
of the back buffer around and then went to create a new one without
first removing the old one, we would not have space in video memory
for the new one and would be forced to create it in system memory.
Eventually, the old one would go away and we would re-create the
new one in accelerated memory, but why not get it right the first
time?

The second item is this condition is even tweakier:


backBuffer = null;

This one is more superfluous, given the flush() call above. I put it
in just to bring up an interesting point (and one that was a complete
mystery to me until I ran into it). In general, the garbage collector
will not collect any objects until they are dead; that is, until there
is no reference to them. By assigning null to backBuffer, we ensure
that the next time the garbage collector runs, the object formerly
pointed at by backBuffer (sounds like the monicker of a rock star...)
is up for collection. If we took the more obvious and usual
approach of simply reassigning backBuffer to the new object
we create, the end result is the same, but the timing is not.

Let us say that the operation of assigning the new object to
backBuffer causes the collector to run to try to get more memory.
It can only get memory from objects that are already dead; but
since the old backBuffer object is still alive until this
reassignment is done, that object is not available for
collection.

As I said, this extra step is somewhat useless here; since we
already nuked the video memory taken up by this image by using the
more definitive and direct approach of calling flush(). The only
possible memory savings here is that Java heap memory taken
up by the image data structure. This is not enough to worry about.
But I brought it up because I have run into problems with this in
other image contexts. For example, let's say (for some unknown
and perhaps suspect reason) that our rendering loop creates a
new BufferedImage every time through the loop for some intermediate
calculation. Images can get pretty big, so this one object can
be enough to swamp the young generation of the collector; but if
we have to have 2 of these objects live simultaneously, then there's
a fair chance that we're going to start chucking objects into the
tenured generation and thus requiring Full (aka, "expensive") GC
operations to get that memory back. Much better to get rid of the
old one before asking for another one.

I don't want to get too deeply into GarbageCollection issues and
implementations (and I think I've reached the limit of my knowledge
on the subject in any case), but I did want to mention the cases above because
as graphics and image users, we tend to run into unique garbage and memory
situation, so it's worth understanding more about this stuff.

Q: If I have several VolatileImage objects, can I just call the
image-loss methods on one of them? Won't I get the same answers from
all of them and thus working with just one of them will be just as good.




A: Yes, but NO.

I say NO with Capital Letters because I have, in fact, run
into this exact bug in a simple demo app I was writing recently.

Whoops!


It is correct that the images will probably all return the same
values (if one image gets lost, all will be lost). At least this
is probably true now, but it is not guaranteed to always
be true. For example, at some point we will implement image
priorities and may actually force some images to be punted out
of memory, thus causing selective loss instead of the wholesale
loss we now see due to operating-system funkiness.
However, even if we could always count on all of the image attributes
being the same, that does not mean that reinstating any one image
will automatically cause all images to be reinstated.

Example time: Let's say we have one VImage sprite and a VIimage
backBuffer. We could write a buggy rendering loop like this:


int valCode = backBuffer.validate(comp.getGraphicsConfig());
if (valCode == VolatileImage.IMAGE_RESTORED) {
    // backBuffer does not need to be restored,
    // but sprite does - initContents() is a method
    // that re-renders the contents into the sprite
    initSpriteContents();
} else if (valCode == VolatileImage.IMAGE_INCOMPATIBLE) {
    // recreate all volatile images
    createBackBuffer();
    createSprite();
    // also restore contents of sprite since we now have a
    // completely new image
    initSpriteContents();
}
// rendering to the back buffer:
Graphics gBB = backBuffer.getGraphics();
gBB.drawImage(sprite, 0, 0, this);

// copy from the back buffer to the screen
g.drawImage(backBuffer, 0, 0, this);

// Now we are done; or are we?  Check contentsLost() and loop as necessary
    } while (!backBuffer.contentsLost());

This code is exactly the same as the code way above except that
we also use the return values from validate() on
the backBuffer to do whatever is necessary to the sprite.

Here's the problem: images only get reinstated (put into a reasonable
state to render to/from) when you call validate() on that image.
So if you get a IMAGE_RESTORED value from backBuffer.validate(), then
chances are pretty good that the sprite is also lost ... and
is staying lost. The code above calls initSpriteContents()
to re-render the contents of the sprite, but these operations
will fail because the sprite is not in a state that will accept
rendering.

The single contentsLost() call will probably work for now, but as I
said above, if any images are every selectively punted that could
change and the above code would be wrong under the mythical future
implementation.

A more correct version of the above looks something like this:


int valCodeBB = backBuffer.validate(comp.getGraphicsConfig());
int valCodeSprite = sprite.validate(comp.getGraphicsConfig());
if (valCodeSprite == VolatileImage.IMAGE_RESTORED) {
    // note that we do not care about the back buffer
    // for restoration since the buffer will be
    // re-rendered here anyway.
    initSpriteContents();
}
if (valCodeBuffer == VolatileImage.IMAGE_INCOMPATIBLE) {
    createBackBuffer();
}
if (valCodeSprite == VolatileImage.IMAGE_INCOMPATIBLE) {
    createSprite();
    // also restore contents of sprite since we now have a
    // completely new image
    initSpriteContents();
}
// rendering to the back buffer:
Graphics gBB = backBuffer.getGraphics();
gBB.drawImage(sprite, 0, 0, this);

// copy from the back buffer to the screen
g.drawImage(backBuffer, 0, 0, this);

// Now we are done; or are we?  Check contentsLost() and loop as necessary
    } while (!backBuffer.contentsLost() && !sprite.contentsLost());
Related Topics >>