Skip to main content

ToolkitBufferedVolatileManagedImage Strategies

Posted by chet on August 11, 2004 at 5:11 AM PDT

A common question seems to arise often from Java graphics
developers about which image type
or creation method to use. When exactly should you use VolatileImage?
What is BufferedImage appropriate for? What about the old
Toolkit images? And when is BufferStrategy more appropriate
than one of these image types?

It's a pretty big topic, and the answer (like all truly great answers)
is probably "It depends". But there are some general guidelines
that can come in handy. And perhaps a description of what these
different kinds of images and methods are all about might help.


1) Image Types

First of all, perhaps a short dictionary of image types might help:

  • Toolkit Image: This is the oldest kind of image object in the Java API. These images
    are created and returned by the old 1.x APIs such as Applet.getImage() and
    Toolkit.createImage(). These images are created from a pointer to a data
    source, such as a GIF or JPG file, and return an object of type Image.
    They are useful for convenient loading and storage of image data for display, but
    getting at the actual pixel data or manipulating it is not as easy.
  • BufferedImage: This is an image type that was created in the jdk1.2 API. They were
    created for easier and more powerful manipulation of the actual pixel data in
    an image. At first, there was no way to load an image from a data source directly
    into a BufferedImage; these images were used, instead, to create an arbitrary buffer
    for pixel data, which you could then write to, read from, or display conveniently.
    The main way to get actual pixel data into a BufferedImage object at first was
    through use of rendering operations (after getting a Graphics object
    for the BufferedImage), or by manually setting the pixel data through methods
    in BufferedImage, WritableRaster, and DataBuffer. With the advent of the
    ImageIO API (see below) in jdk1.4, it became possible to create a
    BufferedImage object directly from a data source, just like Toolkit images (only
    these BufferedImage objects are writable, unlike their Toolkit image cousins).
  • VolatileImage: This image type was created in jdk 1.4 as a means of creating and
    managing accelerated image memory. One of the problems with hardware acceleration for
    images is that, on some platforms, accelerated memory can be deleted out from under
    you at any time. This is obviously not what you want for your typical image data.
    To work around that, the VolatileImage API was created to provide a notification
    mechanism so that you know when an image must be re-rendered due to data loss.
    VolatileImage objects are not loaded from image data, but are just created as empty
    pixel buffers (much as the initial BufferedImage objects were (see above)); to get
    loaded image data into a VolatileImage, applications must load the image data
    through some non-Volatile means, get the Graphics object for the
    VolatileImage, and then copy the data into the Graphics object using drawImage().
  • Managed Images: These image objects are not specific objects or APIs in Java, but are
    rather a concept of how we accelerate image operations. A "managed image" is one
    that you create through any of the normal image creation/loading methods and which
    we try to accelerate for you internally, by creating an accelerated mirror copy of the
    image data. This type of image can benefit from hardware acceleration without falling
    prey to the "surface loss" issues mentioned above for VolatileImage. I'll talk
    more about managed images and hardware acceleration later in the article.

That's it for the basic image types. Now let's talk about how we actually
create and use these image objects.


2) Who you gonna call?

Whenever I want to give myself a fright about the complexity of our
APIs, I simply ponder the vast array of choices that face developers
who simply want to create an image. I'm sure I'm missing some here,
but let's see...


Applet:
	getImage(url)
	getImage(url, name)

BufferedImage:
	new BufferedImage(colorModel, raster, premultiplied, properties)
	new BufferedImage(width, height, type)
	new BufferedImage(width, height, type, colorModel)

BufferStrategy:
	new BufferStrategy()
	(Note: This method cannot be called directly)

Canvas:
	createBufferStrategy(numBuffers)
	createBufferStrategy(numBuffers, capabilities)
	
Component:
	createImage(imageProducer)
	createImage(width, height)
	createVolatileImage(width, height)
	createVolatileImage(width, height, capabilities)

GraphicsConfiguration:
	createCompatibleImage(width, height)
	createCompatibleImage(width, height, transparency)
	createCompatibleVolatileImage(width, height)
	createCompatibleVolatileImage(width, height, capabilities)

Image:
	new Image()
	(Note: This method cannot be called directly)

ImageIO:
	read(file)
	read(imageInputStream)
	read(inputStream)
	read(url)

ImageIcon:
	new ImageIcon(imageData[]).getImage()
	new ImageIcon(imageData[], description).getImage()
	new ImageIcon(image).getImage()
	new ImageIcon(image, description).getImage()
	new ImageIcon(filename).getImage()
	new ImageIcon(filename, description).getImage()
	new ImageIcon(url).getImage()
	new ImageIcon(url, description).getImage()

Toolkit:
	createImage(imagedata[])
	createImage(imagedata[], offset, length)
	createImage(producer)
	createImage(filename)
	createImage(url)
	getImage(filename)
	getImage(url)

Window:
	createBufferStrategy(numBuffers)
	createBufferStrategy(numBuffers, capabilities)

I'm sure there's more out there, especially using things like ImageIO (which is
all about reading and writing images, as you might guess from the name...). But
this list will do for now.

So it's a wrap. This article's pretty much finished; just use the above API calls to
create your images. Left as an exercise to the reader. Q.E.D. It's obvious, isn't it?

Okay, so maybe it isn't obvious; there are a lot of methods above that
all seem to need different parameters or that create different types
of images.

Here's the trick: All of the above image creation methods (and any
others that are not on the list) can be broken down into just a few
categories. Then the plethora of ways of creating an image in
one of those categories can just be seen as utility methods; different
ways of getting the same result. The convenience methods may be because
of logic (why do I have to get the GraphicsConfig to create an image
associated with a Component? Why not use the Component directly?), or
convenience (instead of using some InputStream mechanism for all
image readers, we provide several ways to read the image directly
including from filenames, URLs, and streams; just call the method
appropriate for your situation).

So the real work in this article is to break down the categories of
image types and describe which types of images and methods you may want to use in
which situations. Once you get that down, the rest, as they say,
is just implementation details.


3) Image Loading or Creation?

First of all, are you loading existing image data? Or are you creating
an image buffer in memory? Image loading means that you have image
data (either locally or across the network) that you want to load
into your application, possibly to copy that image onto the screen
or to read and operate on the data. Image creation means that you want some
arbitrary image memory created for your application; perhaps you want to
create a buffer for double-buffered animations, or you want a place to
cache intermediate filtering results.


3.1) Image Loading

In the above method list, all of the methods that take filenames, urls,
streams, producers, and data arrays are those intended for loading existing
images. In particular, all of the methods listed above for Applet,
ImageIO, ImageIcon, and Toolkit are intended for image loading:


Applet:
getImage(url)
getImage(url, name)

ImageIO:
read(file)
read(imageInputStream)
read(inputStream)
read(url)

ImageIcon:
new ImageIcon(imageData[]).getImage()
new ImageIcon(imageData[], description).getImage()
new ImageIcon(filename).getImage()
new ImageIcon(filename, description).getImage()
new ImageIcon(url).getImage()
new ImageIcon(url, description).getImage()

Toolkit:
createImage(imageData[])
createImage(imageData[], offset, length)
createImage(producer)
createImage(filename)
createImage(url)
getImage(filename)
getImage(url)

There are at least four major things that differentiate these methods:

  • the location of the image data
  • the format of the image data
  • the synchronous or asynchronous behavior of each method
  • the type of image that is created (Image or BufferedImage)


3.1.1) Location, Location, Location

When I'm talking about location, I'm mainly concerned with whether the
file is local or across a network. Also, if it's packed into some resource
file, such as a jar file, that also comes into play here.


Loading across the network

If you are accessing the data across a network, it's probably easiest
to use the URL variations:


Applet.getImage(url)
Applet.getImage(url, name)
ImageIO.read(url)
new ImageIcon(url).getImage()
new ImageIcon(url, description).getImage()
Toolkit.createImage(url)
Toolkit.getImage(url)

For example, let's say you just have to have a copy of Duke in your
application (barring the legal ramifications of shipping someone else's image
in your application, of course. I'm no lawyer, but I've seen enough of them
on TV to be very, very afraid). You could, of course, copy that image locally and
read it from a file. But what if that particular Duke image is modified
constantly, preened and pruned to use the very latest imaging technologies
and updated to the latest clothing fashions (assuming Duke actually wore
clothes, but since he's a California native (I suspect Santa Cruz) he is
apparently very comfortable in the buff).
Suppose there is always an up-to-date duke.gif file on the java.sun.com
site. Then you could use something like the following code to load this
image into your application:

URL dukeURL = new URL("http://java.sun.com/duke.gif");
BufferedImage img = ImageIO.read(dukeURL);


Loading from the local file system

Now suppose you have another image that you have saved locally in a file;
just use the filename variation of the above. For example, let's say
you loved one particular instantiation of the lovable-yet-quirky duke.gif
file above so much that you downloaded and saved it for use in your application
(see the above note on scary lawyers). Then you could use the following code
to load that file from the directory where the program was launched:


BufferedImage img = ImageIO.read(new File("duke.gif"));


Loading from jarfiles

It is perhaps more common to bundle up your application and media into
jarfiles. In this case, the image will be downloaded with your application,
but will not be accessible via a simple filename. You will probably want to
use a URL in this case also, where you create a URL from the jarfile
resource. For example, suppose you took your downloaded version of
duke.gif and put him in a subdirectory of your application hierarchy
called "media". Then you could load the image with the following code:


URL dukeURL = getClass().getResource("media/" + "duke.gif");
BufferedImage img = ImageIO.read(dukeURL);


3.1.2) Image format

Another consideration is the format of your stored image. The old
Toolkit/Applet loaders only understand GIF, JPEG, and PNG format files.
(Okay, they also understand XBM and XPM2, old X11 image formats, but
those are probably not formats you are terribly concerned about).
These loaders works well for most web applications since these image types
are traditional web image formats. But what if you have an image
in some other format that the Toolkit/Applet loaders
do not understand?

ImageIO currently has built-in readers for GIF, JEG, and PNG.
In addition, it will have BMP and WBMP capability in the jdk1.5 release.
Moreover, there will be more image readers/writers for ImageIO going
forward, whereas there are no specific plans to support more formats for the
old Toolkit/Applet loaders. And finally, ImageIO has a pluggable
reader API, so if you have a custom image format, or some other
format not yet supported by the core library, you can write
your own loader for that format within ImageIO.

In fact, the JAI team has made available a package with additional ImageIO
readers/writers at

http://java.sun.com/products/java-media/jai/downloads/download-iio.html

if you have requirements beyond the current ImageIO defaults.

So ImageIO could also be the right choice if you need to deal with
formats beyond the basic web image formats.


3.1.3) Synchronicity

The Applet and Toolkit image loading methods came from the old days
of Java 1.0, when Java was seen primarily as a networked application
API and image data might come from any source, potentially
one on an unreliable or slow network connection. To make networked
applications more robust, it is reasonable to put network-dependent
operations in separate threads to ensure that an application's
main or GUI threads do not hang while waiting for a slow download.
Because this was a common pattern for Java GUI applications at that time,
the image loading operations were all created to run on a separate
image loading thread. Thus when an application calls:


Applet.getImage(url)

that call will return immediately. That doesn't mean that the image
has been loaded; in fact, in most cases the image load may not have
even started yet. Internally, getImage() does not load the image at
all. Instead, image loading is deferred until some operation
requires the image data, at which time a separate Thread
processes the image loading.

Note that this model of asynchronous loading does not apply solely to
networked applications, or even to image loading specifically; any operation
that takes a significant amount of time should not be done on the GUI
thread, lest you run the chance of making your application appear hung
while the operation is taking place. So, for example, if you are loading
in a huge image from a local file, you may want that non-networked
operation to happen in a separate worker thread to ensure that your
GUI has no pauses during image loading.

This model works well enough for applications that create their images early for
later use. The application simply may need to check whether
the image has been loaded whenever it is required in the application.

When applications do need the data (for example, if they need image sizes in
order to determine layout correctly, or if they need to display images in
their final form), they may need to synchronize on the image
loader and wait until the image loading is done. For example,
an application may want to load local image data and be willing to
wait for that data to load before proceeding (knowing that a local
load will usually not take very long). In that case, the application might do
something similar to the following:


public Image loadImageSynchronously(Component comp, String filename) {
    URL url = getClass().getResource("media/" + filename);
    Image image = Toolkit.getDefaultToolkit().getImage(url);
    MediaTracker tracker = new MediaTracker(comp);
    int id = 0;
    tracker.addImage(image, id);
    try {
tracker.waitForID(id);
    } catch (InterruptedException e) {
System.out.println("INTERRUPTED while loading Image");
    }
    return image;
}

Note that you can use ImageIcon to load the image synchronously; you could rewrite
the above method as follows:

public Image loadImageSynchronously(String filename) {
    URL url = getClass().getResource("media/" + filename);
    Image image = new ImageIcon(url).getImage();
    return image;
}

ImageIcon is simply a wrapper around this functionality (in fact, I stole part
of the above sample code directly from the ImageIcon implementation: don't
tell the Swing team!). But if your intention is to load an image
synchronously as quickly as possible, why cause us to go through the process
of spawning a separate thread and then synchronizing on that thread?

Meanwhile, ImageIO has synchronous loading methods that do not return
until the image has been loaded and is ready to go. Note that some
applications and situations may still need asynchronous loading behavior
(for long image loads or to more efficiently multitask). For example,
it does not take a huge amount of time to affect perceived GUI
performance, so if an image load will take even as long as a tenth
of a second, you may want to avoid loading that image synchronously on
the Event Dispatch Thread (so don't load it in your paint()
method). You can always spawn a new Thread yourself to call the
ImageIO loading methods if necessary.


3.1.4) Resulting Java Image Type

Part of the decision over which image creation API you is in which image
type you want to get back from the creation method. In particular,
do you want a Toolkit Image or a BufferedImage?

Toolkit Images are created by the Applet, Toolkit, and ImageIcon methods
listed above. The resulting images are easy to use for display purposes
(just call drawImage(...) from this image into a Graphics
destination and the image will be copied appropriately), but lack the
power of BufferedImage objects for manipulation of the data.

BufferedImage objects are created by the ImageIO methods listed above.
These objects offer a more powerful API, albeit with potentially more
work involved to do some operations (such as displaying an animating
GIF image).


Image or BufferedImage: What's in a Name?

Although both Image and BufferedImage have similar
properties in terms of being displayable, BufferedImage has many
more capabilities. For one thing, the Image objects created by the
Toolkit, Applet, and ImageIcon load methods are read-only; you cannot
get the Graphics of those Images and render to them. So if you want to
modify the image data, you will need to do more work (such as creating
another image that is modifiable and copying the loaded Image
into that new image). Image has some very simple methods and is mostly
intended to be a simple object that holds image data. But BufferedImage
has many methods for modifying and extracting all kinds of data from an image;
color models, pixel data, and more. Given a choice between the two,
I would always opt for the one that gave me more power and flexibility.

But doesn't that increased capability mean increased overhead? Not at
all; there is no extra processing involved in BufferedImages when these
other powerful methods are not used. If all you do is load and image and
display it, BufferedImage can do this just as easily as the more
streamlined Image object.

So go ahead and use BufferedImage. It is, after all,

better than butter
.


Dirty Laundry

One good (and not entirely obvious) reason for using the ImageIO API for loading images
is the unfortunate reality that the code is simply newer, cleaner, and more
maintained (both now and in the future). Much of the old
Applet and Toolkit image code was written years ago
and has many assumptions and situations that it must account for and is
therefore tricky to maintain and upgrade.

Our future image reading/writing direction is with ImageIO; yours should be
too, because that's where the focus of our efforts will be in the future.

Having said all that wonderful stuff about ImageIO, there could be situations
in which the old Toolkit/Applet/ImageIcon approach makes more sense for
your particular application, including:

  • Performance: Ideally, ImageIO would be more performant than the old
    code. But some code paths in ImageIO have not yet been optimized to the
    extent that the old code was, so you may find some situations (such as
    loading images in a particular format) which perform better using the old
    APIs than they do with ImageIO. The problems are (mostly) known, the bugs
    have (mostly) been filed, and these performance gaps will (definitely) be fixed. But if
    performance in these situations is important for your application, by
    all means use the old methods in the meantime if they work better for you.
  • Ease of Use: One of the goals of ImageIO was to expose more capabilities
    to the developer. There is much information, such as metadata,
    in an image that is
    not exposed through the old image loading methods or the old Image class.
    The flipside of this approach (at least in the current API) is that it
    can be more work to do trivial tasks. A good example of this is
    loading and displaying an animated GIF.

    In the old image loading APIs, if you loaded an animated GIF file, you did
    not need to worry about the details of animating that image. Instead, you
    could just load the image and display it and we would automatically
    schedule the animations of that image appropriately. You could not
    access the information about the animation (how many frames? what
    animation rate? loop? halt?), but making it animate was quite simple.

    ImageIO, on the other hand, exposes all of that animation information to
    you through the API, but you must actually use that information to manage the image
    animation yourself; you cannot just tell us to draw the image and
    have us manage it for you. This is a known issue and we will
    do something about it Real Soon Now; look for an upcoming article
    about doing this, and look for future API changes to make this simpler
    in the future.
  • Compatibility: Sometimes applications need to be as backward-compatible
    as possible across most or all jdk releases. In particular, some applications
    need to be able to run on old jdk1.1 or earlier releases. For example, applet
    games tend to need ubiquity of the runtime environment and cannot count on
    browsers having any later version of Java than the 1.1 APIs shipped with the
    MSVM. In this case, you may need to use the older APIs simply so that they
    will run across all of these platforms.

Certainly, if you need the kind of functionality that the old APIs
give you, go ahead and use those APIs. That's what they're (still) there
for...

Note also that if you need to use the old APIs for some reason but
you still want the power and flexibility of BufferedImage, it is
easy enough to load the images in through whatever methods are
appropriate, create a new BufferedImage object, and then simply
copy the loaded images into the BufferedImage. For example:


Image img = new ImageIcon(filename).getImage();
BufferedImage bImg = new BufferedImage(img.getWidth(null),
       img.getHeight(null).
       BufferedImage.TYPE_INT_RGB);
Graphics g = bImg.getGraphics();
g.drawImage(img, 0, 0, null);


3.1.5) Hey! What about the other loading methods above?

The approaches above cover most of the loading methods I listed, but
some are notably skipped. The *Stream methods of ImageIO are simply variations
on a theme; if you happen to have your data in that format (versus a
URL or file), go for it; it's just a convenience to use these alternatives.

As for the other skipped methods (one using an ImageProducer and
some using data arrays), I hoped you wouldn't notice....


These other loading approaches are somewhat dated and come from the old days
of Java image
processing. There are still some situations which might require the
ImageProducer/ImageConsumer approach, but in general it should be
easier, more straightforward, and more performant to use the newer APIs
such as ImageIO and BufferedImage. The uses and misuses of the older
Consumer/Producer/PixelGrabber APIs could stand a separate article
all by themselves. I'll try to tackle this one in the future.

As far as reading the image data from
an array of data (see the methods above with the imageData[] parameter),
this is really only appropriate if you've already read
the data into the array to begin with. This could be necessary if
you have some custom image storage mechanism, such as a database.
But if the image
existing in a regular file/URL/stream format, you should probably
be using one of the other loading methods instead.


3.2) Image Creation

What if you do not have an existing image on the network or file system? What
if you just want a buffer of pixel data that you can use in your application?
This could be for creating sprites or icons with rendering calls instead
of loaded image information (perhaps you've found this to be faster in your situation
than reading image files). Or it could be a buffer that you can use for
caching intermediate results or for providing double-buffered rendering
for an animation.

For the purposes of this discussion, I'll break down this category of
images into three types:

  • Static Images: These are images that you render to
    infrequently (perhaps just once, when you create the image). The images
    are mainly used to copy from.
  • Dynamic Images: These are images that you render to often, like
    an animating sprite.
  • Back Buffers: These are like dynamic images, in that they are
    rendered to frequently (usually at least once per frame), but they are
    specifically intended to provide buffering support for the
    onscreen window, and are thus usually copied from often
    as well, like once per frame.


3.2.1) Static Images

Static images are ones that are created and rendered to once (or
infrequently) but probably copied from often. Examples of this type of image
include icons for a GUI or sprites for a game.

The best approach for this type of image is to create an image that is in
the same format as the image or window that the image will be copied to;
this ensures the most straightforward copy mechanism since the underlying
software will not have to perform a conversion on the image data while
copying to the destination.

You could, of course, create a BufferedImage object manually through one
of its constructors; you could query the GraphicsDevice for its display
information and then create a BufferedImage of the appropriate type:


new BufferedImage(colorModel, raster, premultiplied, properties)
new BufferedImage(width, height, type)
new BufferedImage(width, height, type, colorModel)

But why go to the hassle of all of that when there are convenience
mechanisms that do all of this for you? Specifically, take a look
at:


Component.createImage(width, height)
GraphicsConfiguration.createCompatibleImage(width, height)
GraphicsConfiguration.createCompatibleImage(width, height, transparency)

These methods examine the display resolution for the given
Component or GraphicsConfiguration and create an image of an
appropriate type. The Component variation is specified to return an
object of type Image, but currently it is actually a BufferedImage
internally. If you need a BufferedImage object instead (you won't
necessarily need one; you can get a Graphics object from an Image and
render to it just the same as you can to a BufferedImage), then
you can perform an instanceof and cast to a BufferedImage in your code,
or simply call the GraphicsConfiguration variation).

The best part about static images is that you can use very simple
means to create the images and then we will try very hard internally to
see that you get any available hardware acceleration for these images
when they get copied around. We call these "managed images", because
we manage the acceleration details for you. For more information on
managed images, please see my
blog on BufferedImage performance
.

Note that we currently (in all jdk1.4.* releases) manage images
that are created with the above APIs and some of the Toolkit image
loading methods described previously, but in jdk 5.0 (available in Beta
form now, and full release soon) we manage nearly
all types of images and take advantage of hardware acceleration if
it exists. So go ahead and create the type of image that is most
convenient for you and we'll try to do the right thing under the hood.


3.2.2) Dynamic Images

This kind of image may be rendered to quite often, as in an animating
icon, or a sprite that is modified on a frequent basis. You could
certainly use the same image-creation APIs listed above for static images;
these will work fine in most situations and are certainly the easiest way
to go in general.

However, some developers interested in maximizing performance
may want to know more about image management and how dynamic images
can affect it.

We manage images by detecting when the application is copying from an
image to a destination (either another image or an onscreen window) that
lives in accelerated memory. If this copy is done successively when the
source image has not changed, then we may decide to cache a copy of that image
in accelerated memory and perform future copies from this cached version.

In the case of a dynamic image, if that image is being updated one or more
times for every copy to the destination, then we will never create an
accelerated version of it, and thus the image will never benefit from
any hardware acceleration that we could otherwise provide.

(Aside: For the insatiably
curious, the reason for this oddity in acceleration comes from "surface loss",
where an accelerated version of an image may simple go away at any time due
to operating-system or user-caused situations. To keep the original image data
intact, we store the main image data (that which is modified by the application)
in an unaccelerated location, and only accelerate a mirror copy of that image.
That way, if the accelerated version gets wiped out, we still have the original
data from which we can create a new accelerated copy. The problem here, in
terms of performance, is that an "unaccelerated image" means that all rendering
to and from that image is unaccelerated. And if an application is constantly
modifying the image, all of that rendering will be unaccelerated and it is
never appropriate for us to create and use an accelerated version of that image.)

Developers that care about top performance for these types of
images may want to look into using VolatileImages instead. These images
store the data in accelerated memory (when possible) and thus rendering to
and from that image may be accelerated automatically. The downside
is that these images require a bit more care and feeding, due to
"surface loss" issues that arise with current video memory architectures
and operating systems. Note that not all types of rendering to
these images is accelerated, either, but simple types of rendering
like lines and rectangular fills and copies can usually be accelerated, depending
on the platform configuration.

I've already written about VolatileImages in past blogs
(Part I and
Part II), so I will not
go into the details of their usage here; please check out those
other articles for more information. But it is worth covering the
APIs used to create the images, just for consistency's sake in this
article:


Component.createVolatileImage(width, height)
Component.createVolatileImage(width, height, capabilities)
GraphicsConfiguration.createCompatibleVolatileImage(width, height)
GraphicsConfiguration.createCompatibleVolatileImage(width, height, capabilities)

and new in jdk5.0:


GraphicsConfiguration.createCompatibleVolatileImage(width, height, transparency)
GraphicsConfiguration.createCompatibleVolatileImage(width, height, capabilities, transparency)

Note that some of these methods are duplicated
in GraphicsConfiguration and Component just for consistency with the
pre-existing createImage() and createCompatibleImage() methods.
Calling Component.createVolatileImage(w,h) is exactly like calling
Component.getGraphicsConfiguration().createCompatibleVolatileImage(w, h).

The use of the ImageCapabilities object in these methods gives you the ability
to require certain attributes (such as hardware acceleration) from any
image created with that method. In general, you probably will not need to
use that variation, although as we enable more hardware acceleration features
in our platform, we may expand the ImageCapabilities API to be more powerful
and useful. (Note, too, that ImageCapabilities can be used effectively
as a means of inquiring what capabilities an existing image has).


3.2.3) Back Buffers

By "back buffer" I mean an arbitrary offscreen image that is created for use in
a double-buffering situation. Typically, an application that wishes to have
smooth graphics, especially animations, will draw to a back buffer and
then copy that buffer onto the screen instead
of drawing directly to the screen. Swing does this by default, so that
you do not see the various GUI elements in a Swing app flash as they are
drawn to the screen. The buffer copy in these applications typically happens
so fast that the graphics in the application are perceptibly smoother than
if they were drawn one-by-one directly to the screen.

A developer could use any of the above static or dynamic image APIs that
I listed for creating a back buffer. However, the following things should
be taken into account when doing so:

  • Use of managed images for buffers has the same performance
    implications as dynamic images; we will be unable to manage
    the images effectively, thus you will not be able to take advantage
    of any hardware acceleration either for rendering to or copying from
    the buffer. Since the copy from the back buffer to the front buffer
    involves a lot of pixels, being able to accelerate that
    operation can have a huge performance impact on your application.
  • Use of VolatileImages for buffers is a better way to go in
    terms of hardware acceleration potential. But why go to all the
    bother of managing your VolatileImages when there is an easier
    route?


BufferStrategy: the preferred way of buffering in Java

In jdk1.4, we introduced the BufferStrategy API, which is a wrapper around
VolatileImages. This API allows you to ask for an accelerated back buffer
and avoid having
to manage the details of surface loss associated with VolatileImages. It
also ensures that you will get a buffer of the optimal type for your application.
In particular, you will get either a FlipBuffer (which can only be used
in fullscreen-exclusive mode on Windows) or a BltBuffer (which is used
by default for windowed applications). A FlipBuffer performs a swap of the
front and back buffers in video memory when you request BufferStrategy.show().
A BltBuffer copies the contents from the back buffer to the front (just as
you would if you called drawImage() from a VolatileImage back buffer to the
front buffer).

With this API, there is little need to create and manage
VolatileImages directly; just let us manage the details for you inside
the BufferStrategy implementation.

For more information on BufferStrategy, check out the javadocs; they're
pretty clear on how the system works.

The APIs you will need when creating a BufferStrategy are:


Canvas.createBufferStrategy(numBuffers)
Canvas.createBufferStrategy(numBuffers, capabilities)
Frame.createBufferStrategy(numBuffers)
Frame.createBufferStrategy(numBuffers, capabilities)

For the most part, you will do this on a Frame. I know the API says
that the methods are in Window, but a Window is really just a pseudo-frame
and Frames are more useful in general. Take my advice; use Frame (or JFrame, for
the Swing developers).


4) Wrap-Up

So that's pretty much it. You have image loading methods and image (or buffer)
creation methods. And in each category, you have various flavors depending
on the location and type of the data, and the type of image you want
returned to you. So even though there are a lot of methods listed at the
top of this article, they all break down into just a few comprehensible
categories and can be used effectively, once you understand the implications
of each variation.

Although there is certainly more complexity here than we can cover with
a simple table, it might help to break down some of the basic attributes
of the image types we have talked about and the reasons to consider one
type over another when writing your application:


  • Toolkit Image:

    • Creation methods:
      Applet.getImage(...), Toolkit.createImage(...),
      Toolkit.getImage(...), ImageIcon.getImage()
    • Most useful for: Simple loading and display of image data, applications
      that need to use 1.x APIs, asynchronous usage models (except for ImageIcon),
      loading performance (versus
      some ImageIO loading operations in the current releases).
    • Less useful for: Image data manipulation, reading/writing individual pixel
      data within the image, synchronous loading operations (except for
      ImageIcon), more varied or custom source image formats..

  • BufferedImage:

    • Creation methods:
      new BufferedImage(...),
      Component.createImage(...),
      GraphicsConfiguration.createCompatibleImage(...),
      ImageIO.read(...)
    • Most useful for: Image data manipulation, read/write access to
      individual pixel data, synchronous loading or creation, various image
      data formats (both source data that ImageIO can read and Java format
      (i.e., TYPE_INT_RGB, TYPE_INT_RGBA, etc.).
    • Less useful for: Applications that require jdk 1.x compatibility,
      easy display of animating GIF images.

  • VolatileImage:

    • Creation methods:
      Component.createVolatileImage(...),
      GraphicsConfiguration.createVolatileImage(...)
    • Most useful for: Hardware acceleration of dynamic images, more
      fine-grained control over acceleration properties.
    • Less useful for: Simple image creation and display (applications must
      use the VolatileImage API to handle surface loss issues)

  • BufferStrategy:

    • Creation methods:
      Canvas.createBufferStrategy(),
      Window.createBufferStrategy()
    • Most useful for: Easy creation and management of window buffering
      schemes
    • Less useful for: Access to actual image data of buffer


4.1) Hey! You forgot some methods!

There are still a couple of the creation methods up top that I have not
covered yet:


BufferStrategy:
new BufferStrategy()
Image:
new Image()

These classes are abstract base classes and you cannot create an instances
of them directly (and couldn't do anything useful with it if you could).
BufferStrategy objects must be created on the component which they will
buffering and Image objects are created from one of the load/create methods
explained above.


4.1) What About Performance?

It is difficult or impossible for me to write a long block of text or code
without thinking about performance. And since some of the users of the APIs
above, and image operations in general, care a great deal about performance,
I should spend a few words discussing some performance issues to be aware of.

Again, check out my blogs on BufferedImage (
Part I and
Part II); I go into much more detail
on image management there. Some important things to keep in mind with
respect to managed images (and making sure they are benefiting from
available acceleration):

  • If you want copies from an image to be accelerated
    but you will be modifying the image
    frequently, think about VolatileImage instead (see the above
    discussion on Dynamic Images).
  • Careful with that Raster, Eugene! This is a hidden gotcha in the current
    implementation. Part of the BufferedImage API is a method that
    allows you to get the Raster of the image (getRaster()). From that Raster,
    you can create a writable child raster (WritableRaster.createWritableChild())
    or get the internal data array for the raster (Raster.getDataBuffer()).
    Both of these methods end up giving you a copy of the data that we
    have no control over. Specifically, we cannot tell when you have updated
    the contents of the image, so we cannot tell whether our cached/accelerated
    version of the image needs to be updated. So, when you call these Raster
    methods, we throw up our hands and give up on acceleration entirely.


    If you need to access the pixel data but you still want us to accelerate
    image copies, call methods which manipulate the data for you without giving
    you a reference to the data; e.g., BufferedImage.setRGB() and
    WritableRaster.setPixels() for setting arbitrary pixel values, or
    any of the Graphics.draw*() or Graphics.fill* methods for setting
    blocks of pixel values.
  • If you are transforming the image while copying it around (say, scaling
    it to the destination, or rotating it), we will not be able to do that
    using hardware acceleration with the current implementation. If you anticipate
    using a particular transformation of an image (like a zoomed-in view or
    several rotations of a sprite) multiple
    times, then it might be worth creating a temporary version of that
    transformed image and letting us manage (accelerate)
    that transformed version for you.


4.2) What about 5.0?

Most of the APIs I discussed above are pre-5.0, so you can use everything above
(except where noted)
in the current releases available for download. If you are looking forward
to using 5.0 (available in beta form today, in full
release Real Soon Now), then I'll mention a couple of tweaks to the above:

  • VolatileImage APIs: It will be possible in 1.5 to create a VolatileImage
    with a transparency parameter. Developers have been asking for that
    capability for some time; the original createVolatileImage() methods
    only allowed creation of opaque VolatileImages.

    That's the good news: the API is there. The bad news is
    that there is no hardware acceleration for these non-opaque
    images by default
    because our mechanisms for accelerating these types of images are still
    not ready for prime time. Translucent volatile images can achieve hardware
    acceleration by using the handy -Dsun.java2d.translaccel=true flag.
  • Managed images everywhere! You will now get managed images
    by calling most of our image creation
    and loading mechanisms. In jdk1.4.*, only some of the image creation
    methods (such as Component.createImage() and
    GraphicsConfiguration.createCompatibleImage()) returned managed images.
    But in 1.5, you can even call new BufferedImage(...) with a type that
    is not in the screen bit depth and end up with an accelerated
    managed image.
  • OpenGL Acceleration: The 5.0 release has an OpenGL-based
    rendering pipeline available on all platforms. This pipeline is not
    be enabled by default (due to various concerns including
    graphics hardware and driver issues), but there is a flag that you
    can use to experiment with it (sun.java2d.opengl=true).
    This pipeline provides our most
    comprehensive hardware acceleration so far on all our supported
    platforms, so it should be pretty exciting to see this one in action.
Related Topics >>