Skip to main content

Write a Phony Application ... Or Dial Trying

Posted by chet on January 16, 2008 at 1:29 PM PST

A few weeks ago, in a quest for more performance benchmarks, the scene graph team
asked for a demo that was representative of some of the graphics and animations that
might be typical in a consumer-oriented application.

I had run across an interesting video on the Apple site recently,
iPhone: A guided tour
. The device in the video had ideas of what we were looking for; fades,
moves, scales, transitions... all the whizzy animations that consumers love. So
I took a whack at doing something similar with the scene graph.

Introducing: JPhone:

jPhoneLg.png

Okay, so it's not really the same thing. For one thing, I just used some icons I
had lying around which don't look as good at the large size required for this interface.
I'd love to use the iPhone icons instead, but I'm still waiting for the Apple lawyers
to call me back. And waiting. And waiting. (Steve?)

Also, I guess I have to admit it: the jPhone demo is not a phone. Even if
you pick up your monitor and hold it next to your ear, all you'll hear is the sound
of your brain screaming in pain from the pixel radiation. (And the ocean. Isn't it
funny how you can always hear the ocean? Or maybe it's just sound waves.)
But mimicking a phone
wasn't the point; it was all about the user interface.

Finally, it'll become obvious when you start to use it that, well, there are no
'applications' behind the icons; it's just the same dummy screen that comes up again
and again. But once more, my petty rationalization comes in handy; the demo was
supposed to be about GUI animations, not actual functionality.

But hey, It's A Demo!

Anyway, on with the article. Note that my discussion below is all about the jPhone
demo. I don't actually know how things operate under the hood on that phone thingie
from Apple; all I know I learned from watching that video. But I do know how the
jPhone demo works, so I'll stick to that.

The GUI: Main menu, tray menu, and applications

There are three different GUI areas in the display, used at different times. What
I call the "main menu" is the grid of icons arrayed out across the first screen,
starting at the top. Each of these icons accesses a different application (each
of which looks uncannily similar in JPhone) when clicked. The "tray menu" at the bottom has four additional icons, which are much the same as the
icons in the main menu, but for common functionality that the user might want to access more frequently.
Finally, the "application" screens are those displays that come
up after the user clicks an icon. For example, when the user clicks on the Calculator
button, they probably expect a calculator application screen to become active.

Main Menu

The main menu of the application consists of a grid of icons, four columns wide.
The menu is created in the cleverly-named createMenu() method.

Each "icon" consists of both an image and a text caption. So for each icon object
in the scene graph, we create an SGGroup to hold the children,
an SGImage to hold
the image of the icon, and an SGText object to hold the caption.

The group is created in a single line as follows:

        SGGroup imageAndCaption = new SGGroup();

The image node takes a few more lines, as we need to scale the image appropriately
and then set the image on the node:

        SGImage icon = new SGImage();
        try {
            BufferedImage originalImage =
                    ImageIO.read(getImageURL(mainMenuIcons[iconIndex]));
            BufferedImage iconImage = new BufferedImage(ICON_SIZE,
                            ICON_SIZE, BufferedImage.TYPE_INT_ARGB);
            Graphics2D gImage = iconImage.createGraphics();
            gImage.setRenderingHint(RenderingHints.KEY_INTERPOLATION,
                    RenderingHints.VALUE_INTERPOLATION_BILINEAR);
            gImage.drawImage(originalImage, 0, 0, ICON_SIZE, ICON_SIZE,
                    null);
            gImage.dispose();
            icon.setImage(iconImage);
            imageAndCaption.add(icon);
        } catch (Exception e) {
            System.out.println("Error loading image: " + e);
        }

(Note that our image scaling assumes that a one-step bilinear scale will give us
sufficient quality, which it does in the case of up-scaling the smaller images we're
using for icons. For a more general scaling solution that gives dependable quality and decent
performance, check out Chris
Campbell
's article on
The Perils of Image.getScaledInstance()
).

The text node takes a few lines to set up the text rendering and location attributes
appropriately:

        SGText textNode = new SGText();
        textNode.setText(iconCaptions[iconIndex]);
        textNode.setFont(captionFont);
        textNode.setFillPaint(Color.LIGHT_GRAY);
        textNode.setAntialiasingHint(RenderingHints.VALUE_TEXT_ANTIALIAS_ON);
        Rectangle2D rect = textNode.getBounds();
        textNode.setLocation(new Point2D.Double(
                (ICON_SIZE - rect.getWidth())/2, ICON_SIZE + 10));
        imageAndCaption.add(textNode);

To position each icon in the menu, and to allow the icon to be moved later when it
animates, we create a transform node as the parent of the icon and add that node
to the scene graph:

        SGTransform.Translate transformNode = SGTransform.createTranslation(xOffset, 
                yOffset, imageAndCaption);
        rootNode.add(transformNode);

Tray Menu

The tray menu, initialized in the createTray() method, consists of just four icons at the bottom of the screen. These icons
are mostly like the main menu icons, although they have a different background
and the animation they undergo during transitions is different, so there are differences in their setup.

First, the tray needs to be positioned on the screen, so the tray group needs an
overall transform. Also, we will fade the tray in and out during transitions from
and to the application screen, so the tray group also needs a filter node
to handle fades. These nodes are set up as follows:

        final SGGroup trayGroup = new SGGroup();
        SGComposite opacityNode = new SGComposite();
        opacityNode.setOpacity(1f);
        SGTransform trayTransform = SGTransform.createTranslation(0,
                SCREEN_H - (1.5 * ICON_SIZE), trayGroup);
        opacityNode.setChild(trayTransform);
        rootNode.add(opacityNode);

Next, we have an interesting background for the tray that consists of a basic gray
gradient with a solid darker gray underneath the captions. We set this up with a
couple of shape nodes and a transform node to position the darker area appropriately:

        // Set up the basic tray background
        SGShape trayBackground = new SGShape();
        trayBackground.setShape(new Rectangle(SCREEN_W,
                (int)(ICON_SIZE * 1.5)));
        trayBackground.setMode(SGShape.Mode.FILL);
        trayBackground.setFillPaint(new GradientPaint(0f, 0f, Color.DARK_GRAY,
                0f, (float)(ICON_SIZE * 1.5), Color.LIGHT_GRAY));
        trayGroup.add(trayBackground);
       
        // Set up the darker background for the captions
        SGShape captionBackground = new SGShape();
        captionBackground.setShape(new Rectangle(SCREEN_W, 20));
        captionBackground.setMode(SGShape.Mode.FILL);
        captionBackground.setFillPaint(new GradientPaint(0f, 0f, Color.DARK_GRAY,
                0f, 10f, Color.GRAY));
        SGTransform captionBGTransform = SGTransform.createTranslation(0,
                (1.5 * ICON_SIZE) - 22, captionBackground);
        trayGroup.add(captionBGTransform);

The icons themselves are set up just like those for the main menu, adding themselves
to the trayGroup object created above; I'll skip the details since the code is similar
to what we saw earlier for the main menu icons.

Application

The application objects, created in the createApp() method, are simply images. The
only interesting part about them is the animation of scaling and fading in and out
as they become active and inactive. We load and scale the application image just
like we did for the icons in createMenu() above, so I won't show that code here.
We then add filter nodes for opacity and for scaling, so that we can fade and scale
the application screen during animations:

        // App screen is just an image, scaled/faded in when it becomes active
        final SGImage photo = new SGImage();
        // ... code to load/scale/set image removed for brevity ...
        photo.setVisible(false);
        SGComposite opacityNode = new SGComposite();
        opacityNode.setOpacity(0f);
        opacityNode.setChild(photo);
        AffineTransform fullScale = new AffineTransform();
        AffineTransform smallScale =
                AffineTransform.getTranslateInstance(
                SCREEN_W/2, SCREEN_H/2);
        smallScale.scale(.1, .1);
        smallScale.translate(-SCREEN_W/2, -SCREEN_H/2);
        SGTransform scaleNode = SGTransform.createAffine(smallScale, opacityNode);
        fullScale = new AffineTransform();
        rootNode.add(scaleNode);

Note that the application node starts out invisible (because it is hidden until
triggered by a mouse click on one of the menu icons), completely transparent (until
it is faded in), and scaled to 10% of its true size (until it is scaled in during
a later animation).

Animations

The objects set up above were necessary, but the fun part is really the animations
that drive the application. The animations are all triggered based on user clicks. A click on
the main menu icons will run animations on the main menu, the tray menu, and the
application screen simultaneously. A click on the application screen will run
all of the same animations - in reverse. Let's see how we set up and run these animations.
There are two Timelines create to run these animations (recall from
an
earlier blog entry
that Timeline is a convenient grouping mechanism for animations):

        Timeline menuOutTimeline = new Timeline();
        Timeline menuInTimeline = new Timeline();

Main Menu Animation

The interesting part in the main menu animation is in trying to guess what's going
on in the Apple video (and the iPhone interface). As an engineer, I would expect the icons to move in a linear
fashion, sliding horizontally or vertically, perhaps the same every time, or maybe
with the direction set based on which icon was clicked. In fact, one of the engineers
on the team once rewrote my animation code to do just that, assuming that there
was a bug in my code and I must have meant to have this straight-line animation instead of the effect I had implemented.

But if you look closely at the video (or, heck, at that iPhone you have in your
pocket), you'll see that the icons move in diagonal trajectories off of the screen,
all shooting off in different directions. For example, here's a stop-action view
captured from the video:

iPhoneAnim1crop.jpg

iPhoneAnim2crop.jpg

iPhoneAnim3crop.jpg

iPhoneAnim4crop.jpg

iPhoneAnim5crop.jpg

iPhoneAnim6crop.jpg

Also, it looks the same every time, no
matter which icon is clicked. I would then assume (and have implemented the code
this way in jPhone) that the icons all shoot away from one central point on
the screen. But that doesn't appear to be the case.

Anyway, I think the animation for jPhone's main menu looks pretty good, if not exactly
what they happen to do on that other device.

The basic idea in jPhone is to calculate the movement vector based on some movement
"center" (xCenter, yCenter) and the location of each icon (xOffset, yOffset):

        double xDir = xOffset - xCenter;
        double yDir = yOffset - yCenter;

We can then calculate the new offscreen position of the icon (xOffscreen, yOffscreen)
using this direction vector (I won't show that code here for brevity, but it basically
sets an offscreen value in one coordinate (x or y) and then calculates the other
coordinate based on the direction vector).

Finally, we can create an animation that will move the icon from its position in
the main menu to this offscreen position as follows:

        Clip iconClip = Clip.create(MENU_OUT_TIME,
                transformNode, "translateX", xOffscreen);
        iconClip.addTarget(KeyFrames.create(
                new BeanProperty(transformNode, "translateY"),
                yOffscreen));
        menuOutTimeline.schedule(iconClip);

The transformNode object being animated is the filter node in charge of the icon
location, so this animation will alter the translateX and translateY properties
of that object during the animation. Similarly, we create the opposite animation
to move the icon back to its original location from where it's residing offscreen:

        iconClip = Clip.create(MENU_IN_TIME,
                transformNode, "translateX", xOffset);
        iconClip.addTarget(KeyFrames.create(
                new BeanProperty(transformNode, "translateY"),
                yOffset));
        menuInTimeline.schedule(iconClip);

Tray Menu Animation

Tray menu animation is simpler; we just fade the tray out and back in when applications
become active or inactive. To fade the tray out, we create an animation on the opacity
property of the opacityNode object that we created earlier:

        Clip fader = Clip.create((int)MENU_OUT_TIME, opacityNode, 
                "opacity", 1f, 0f);
        fader.addTarget(new TimingTargetAdapter() {
            public void end() {
                trayGroup.setVisible(false);
            }
        });
        menuOutTimeline.schedule(fader);

Note that we're actually doing two things here; we're fading out the node from opaque
to completely transparent, and we're setting the visibility of the node to false
when the animation ends. The visibility property of nodes controls whether the nodes
process events or try to render themselves; since the node will be completely transparent
when it is faded out completely, it doesn't make sense for it to participate in
either events or rendering.

When an application becomes inactive, we run the reverse animation on the tray menu
to make it visible and fade it in:

        fader = Clip.create((int)MENU_IN_TIME, opacityNode, 
                "opacity", 0f, 1f);
        fader.addTarget(new TimingTargetAdapter() {
            public void begin() {
                trayGroup.setVisible(true);
            }
        });
        menuInTimeline.schedule(fader);

Application Animation

Finally, we need to animate the application screen. These animations are similar
to what we've seen before, although in this case we are both fading and scaling
the application screen:

        final Timeline appAnims = new Timeline();
        Clip fader = Clip.create((int)MENU_OUT_TIME, opacityNode,
                "opacity", .1f, 1f);
        Clip scaler = Clip.create((int)MENU_OUT_TIME, scaleNode,
                "affine", smallScale, fullScale);
        appAnims.schedule(fader);
        appAnims.schedule(scaler);
        fader.addTarget(new TimingTargetAdapter() {
            public void begin() {
                photo.setVisible(true);
            }
        });

The application animation is also how we start tying the different animations and
events together. First of all, we kick off the overall menuOutTimeline animation
from the application animation by scheduling it as a dependent animation
of the application's fader Clip, as follows:

        fader.addBeginAnimation(menuOutTimeline);

Next, we add an attribute to each icon that tells it which animation it is associated
with:

        appIcon.putAttribute("startAnim", appAnims);

Finally, we add a mouse listener to each icon that will listen for clicks and start
the animation appropriate for that icon:

        appIcon.addMouseListener(appStartListener);

where the mouse listener is defined to start the animation that we stored as an attribute
on the icon in question:

        SGMouseListener appStartListener = new SGMouseAdapter() {
            @Override
            public void mouseClicked(MouseEvent e, SGNode n) {
                Animation a = (Animation) n.getAttribute("startAnim");
                a.start();
            }
        };

We do similarly for the reverse animation for the application (I'll skip that code
for brevity and added surprise and excitement).

TransformComposer

Chris might prefer that I not mention this detail, since the final API for the scene
graph will hopefully make this step irrelevant, but for now the only way to animate
transforms such as the moves and scales shown above is for the animation engine
to know how to interpolate AffineTransform objects. It does not know how to do this
by default (because, frankly, it's not typically the way you would want to interpolate
between positions and orientations; it can produce unexpected results, thus the
need for better functionality in the API eventually). So we need to add this capability
to the application. We do this by creating a TransformComposer object
that tells the system
how to interpolate between AffineTransform objects, and we register the composer
as follows:

        static {
            Composer.register(AffineTransform.class, TransformComposer.class);
        }

I won't go into the details of TransformComposer here, but see
my earlier blog entry on the animation system
and the Composer
JavaDocs on the Scene Graph project site
to understand more about Composer.
Basically, a custom Composer simply converts between an arbitrary type and an array
of double values, which the base Composer class then knows how to linearly interpolate
between.

Runtime

That's mostly it. If you run the application and click on the icons you can
see the fading, moving, and scaling animations all working together to show a nice,
smooth transition between the menu and application screens.

Of course, in demos enough is never enough. So we decided to put in one more element
for fun.

In the Apple video, you'll notice that many of the demos they show are run by this
disembodied hand. It could be the Hand of God, but I don't think that Steve was
in the video. Besides, the hand isn't wearing a black turtleneck.

iPhoneHand.jpg

It seemed like our demo needed an element like that, so we created
the handCursor node.

Handy Cursor

Custom cursors are fairly easy in Java, but they are also fairly limited. In particular,
your cursor image is limited to 32x32, which doesn't really give us the effect we
were looking for. I want a hand, not a hand-shaped wart. We need a friggin' huge cursor.

In a traditional Swing application, we could manage this using the glass pane, displaying
an arbitrary image in that overlay on top of the application GUI. But the scene graph makes this
even easier; we just need a shape node.

First things first: we need to manage the real Swing cursor. Since we cannot use
the actual cursor as our hand, we will instead make the real cursor invisible with
the following code, so that if we can't make the cursor do what we really want,
we can at least get it out of the way:

        BufferedImage emptyImage = new BufferedImage(32, 32,
                BufferedImage.TYPE_INT_ARGB);
        invisibleCursor = Toolkit.getDefaultToolkit().
                createCustomCursor(emptyImage, new Point(0, 0), "empty");

Next, we will create our handCursor object, parent it to the root node, and add
it as a MouseMotionListener on the application as follows:

        handNode = new HandCursor(rootNode);
        addMouseMotionListener(handNode);

I won't show the entire HandCursor class (it's frankly not interesting enough), but
the basics are as follows: First, we load the hand image and scale it to an appropriate
size (using code similar to that shown earlier for the menu icons).
Next, we create an SGImage node and a transform node (for moving it), and parent the transform node to the root node. We also make the shape invisible
at first, since the hand cursor is not showing
by default:

        handNode = new SGImage();
        handNode.setImage(largerHand);
        handNode.setVisible(false);
        handTransform = SGTransform.createTranslation(0, 0, handNode);
        rootNode.add(handTransform);

When the application frame detects that the key "h" has been typed, it makes the
default cursor invisible and the hand cursor visible with the following:

        setCursor(invisibleCursor);
        handNode.setVisible(true);

Now, all we have to do is track the mouse
position and display the hand node appropriately:

        public void mouseMoved(MouseEvent me) {
            mouseX = me.getX();
            mouseY = me.getY();
            handTransform.setTranslation(mouseX - 160, mouseY - 5);
        }

(where the hard-coded numbers in setTranslation() position the "hotspot" of the hand
cursor at the tip of the index finger).

return;

That's pretty much it. There's more code in JPhone, but I think I've covered
all of the interesting scene graph-related pieces above. Play with it, check out
the code
, and write a phony scene graph application of your own.

jPhoneAnim1.jpg
jPhoneAnim2.jpg
jPhoneAnim3.jpg
jPhoneAnim4.jpg
jPhoneAnim5.jpg

Related Information

JPhone Demo: a Java Web Start application

JPhone.java: the source code for the main demo class

TransformComposer.java: the source code for the custom composer helper class

Scene Graph Demos: The project site for all of the Scene Graph demos posted by the
team (including JPhone)

Related Topics >>

Comments

mikeazzi: Thanks - I was mistakenly hosting the JNLP file on the wrong server. Try it now...

I'm guessing that the webstart link doesn't work because it is being served as text/xml.

So I guess in my case, you could probably say I Dialed Trying :)

Hi Chet, the demo's Web Start (.jnlp) link does not work with either FF or IE on Win XP, under jre6 Update 10. The browser just tries to open the page as an XML document. Thanks.

Very, very impressive! I love a lot of the animation transformations in the iPhone and I'm pretty excited to put them into my own projects.