A few weeks ago, in a quest for more performance benchmarks, the scene graph team asked for a demo that was representative of some of the graphics and animations that might be typical in a consumer-oriented application.

I had run across an interesting video on the Apple site recently, iPhone: A guided tour. The device in the video had ideas of what we were looking for; fades, moves, scales, transitions... all the whizzy animations that consumers love. So I took a whack at doing something similar with the scene graph.

Introducing: JPhone:

Okay, so it's not really the same thing. For one thing, I just used some icons I had lying around which don't look as good at the large size required for this interface. I'd love to use the iPhone icons instead, but I'm still waiting for the Apple lawyers to call me back. And waiting. And waiting. (Steve?)

Also, I guess I have to admit it: the jPhone demo is not a phone. Even if you pick up your monitor and hold it next to your ear, all you'll hear is the sound of your brain screaming in pain from the pixel radiation. (And the ocean. Isn't it funny how you can always hear the ocean? Or maybe it's just sound waves.) But mimicking a phone wasn't the point; it was all about the user interface.

Finally, it'll become obvious when you start to use it that, well, there are no 'applications' behind the icons; it's just the same dummy screen that comes up again and again. But once more, my petty rationalization comes in handy; the demo was supposed to be about GUI animations, not actual functionality.

But hey, It's A Demo!

Anyway, on with the article. Note that my discussion below is all about the jPhone demo. I don't actually know how things operate under the hood on that phone thingie from Apple; all I know I learned from watching that video. But I do know how the jPhone demo works, so I'll stick to that.

The GUI: Main menu, tray menu, and applications

There are three different GUI areas in the display, used at different times. What I call the "main menu" is the grid of icons arrayed out across the first screen, starting at the top. Each of these icons accesses a different application (each of which looks uncannily similar in JPhone) when clicked. The "tray menu" at the bottom has four additional icons, which are much the same as the icons in the main menu, but for common functionality that the user might want to access more frequently. Finally, the "application" screens are those displays that come up after the user clicks an icon. For example, when the user clicks on the Calculator button, they probably expect a calculator application screen to become active.

Main Menu

The main menu of the application consists of a grid of icons, four columns wide. The menu is created in the cleverly-named createMenu() method.

Each "icon" consists of both an image and a text caption. So for each icon object in the scene graph, we create an SGGroup to hold the children, an SGImage to hold the image of the icon, and an SGText object to hold the caption.

The group is created in a single line as follows:

        SGGroup imageAndCaption = new SGGroup(); 

The image node takes a few more lines, as we need to scale the image appropriately and then set the image on the node:

        SGImage icon = new SGImage();
        try {
            BufferedImage originalImage =
                    ImageIO.read(getImageURL(mainMenuIcons[iconIndex]));
            BufferedImage iconImage = new BufferedImage(ICON_SIZE,
                            ICON_SIZE, BufferedImage.TYPE_INT_ARGB);
            Graphics2D gImage = iconImage.createGraphics();
            gImage.setRenderingHint(RenderingHints.KEY_INTERPOLATION,
                    RenderingHints.VALUE_INTERPOLATION_BILINEAR);
            gImage.drawImage(originalImage, 0, 0, ICON_SIZE, ICON_SIZE,
                    null);
            gImage.dispose();
            icon.setImage(iconImage);
            imageAndCaption.add(icon);
        } catch (Exception e) {
            System.out.println("Error loading image: " + e);
        }

(Note that our image scaling assumes that a one-step bilinear scale will give us sufficient quality, which it does in the case of up-scaling the smaller images we're using for icons. For a more general scaling solution that gives dependable quality and decent performance, check out Chris Campbell's article on The Perils of Image.getScaledInstance()).

The text node takes a few lines to set up the text rendering and location attributes appropriately:

        SGText textNode = new SGText();
        textNode.setText(iconCaptions[iconIndex]);
        textNode.setFont(captionFont);
        textNode.setFillPaint(Color.LIGHT_GRAY);
        textNode.setAntialiasingHint(RenderingHints.VALUE_TEXT_ANTIALIAS_ON);
        Rectangle2D rect = textNode.getBounds();
        textNode.setLocation(new Point2D.Double(
                (ICON_SIZE - rect.getWidth())/2, ICON_SIZE + 10));
        imageAndCaption.add(textNode);

To position each icon in the menu, and to allow the icon to be moved later when it animates, we create a transform node as the parent of the icon and add that node to the scene graph:

        SGTransform.Translate transformNode = SGTransform.createTranslation(xOffset, 
                yOffset, imageAndCaption);
        rootNode.add(transformNode);

Tray Menu

The tray menu, initialized in the createTray() method, consists of just four icons at the bottom of the screen. These icons are mostly like the main menu icons, although they have a different background and the animation they undergo during transitions is different, so there are differences in their setup.

First, the tray needs to be positioned on the screen, so the tray group needs an overall transform. Also, we will fade the tray in and out during transitions from and to the application screen, so the tray group also needs a filter node to handle fades. These nodes are set up as follows:

        final SGGroup trayGroup = new SGGroup();
        SGComposite opacityNode = new SGComposite();
        opacityNode.setOpacity(1f);
        SGTransform trayTransform = SGTransform.createTranslation(0, 
                SCREEN_H - (1.5 * ICON_SIZE), trayGroup);
        opacityNode.setChild(trayTransform);
        rootNode.add(opacityNode);

Next, we have an interesting background for the tray that consists of a basic gray gradient with a solid darker gray underneath the captions. We set this up with a couple of shape nodes and a transform node to position the darker area appropriately:

        // Set up the basic tray background
        SGShape trayBackground = new SGShape();
        trayBackground.setShape(new Rectangle(SCREEN_W, 
                (int)(ICON_SIZE * 1.5)));
        trayBackground.setMode(SGShape.Mode.FILL);
        trayBackground.setFillPaint(new GradientPaint(0f, 0f, Color.DARK_GRAY,
                0f, (float)(ICON_SIZE * 1.5), Color.LIGHT_GRAY));
        trayGroup.add(trayBackground);
        
        // Set up the darker background for the captions
        SGShape captionBackground = new SGShape();
        captionBackground.setShape(new Rectangle(SCREEN_W, 20));
        captionBackground.setMode(SGShape.Mode.FILL);
        captionBackground.setFillPaint(new GradientPaint(0f, 0f, Color.DARK_GRAY,
                0f, 10f, Color.GRAY));
        SGTransform captionBGTransform = SGTransform.createTranslation(0, 
                (1.5 * ICON_SIZE) - 22, captionBackground);
        trayGroup.add(captionBGTransform);

The icons themselves are set up just like those for the main menu, adding themselves to the trayGroup object created above; I'll skip the details since the code is similar to what we saw earlier for the main menu icons.

Application

The application objects, created in the createApp() method, are simply images. The only interesting part about them is the animation of scaling and fading in and out as they become active and inactive. We load and scale the application image just like we did for the icons in createMenu() above, so I won't show that code here. We then add filter nodes for opacity and for scaling, so that we can fade and scale the application screen during animations:

        // App screen is just an image, scaled/faded in when it becomes active
        final SGImage photo = new SGImage(); 
        // ... code to load/scale/set image removed for brevity ...
        photo.setVisible(false);
        SGComposite opacityNode = new SGComposite();
        opacityNode.setOpacity(0f);
        opacityNode.setChild(photo);
        AffineTransform fullScale = new AffineTransform();
        AffineTransform smallScale = 
                AffineTransform.getTranslateInstance(
                SCREEN_W/2, SCREEN_H/2);
        smallScale.scale(.1, .1);
        smallScale.translate(-SCREEN_W/2, -SCREEN_H/2);
        SGTransform scaleNode = SGTransform.createAffine(smallScale, opacityNode);
        fullScale = new AffineTransform();
        rootNode.add(scaleNode);

Note that the application node starts out invisible (because it is hidden until triggered by a mouse click on one of the menu icons), completely transparent (until it is faded in), and scaled to 10% of its true size (until it is scaled in during a later animation).

Animations

The objects set up above were necessary, but the fun part is really the animations that drive the application. The animations are all triggered based on user clicks. A click on the main menu icons will run animations on the main menu, the tray menu, and the application screen simultaneously. A click on the application screen will run all of the same animations - in reverse. Let's see how we set up and run these animations. There are two Timelines create to run these animations (recall from an earlier blog entry that Timeline is a convenient grouping mechanism for animations):

        Timeline menuOutTimeline = new Timeline();
        Timeline menuInTimeline = new Timeline();

Main Menu Animation

The interesting part in the main menu animation is in trying to guess what's going on in the Apple video (and the iPhone interface). As an engineer, I would expect the icons to move in a linear fashion, sliding horizontally or vertically, perhaps the same every time, or maybe with the direction set based on which icon was clicked. In fact, one of the engineers on the team once rewrote my animation code to do just that, assuming that there was a bug in my code and I must have meant to have this straight-line animation instead of the effect I had implemented.

But if you look closely at the video (or, heck, at that iPhone you have in your pocket), you'll see that the icons move in diagonal trajectories off of the screen, all shooting off in different directions. For example, here's a stop-action view captured from the video:

Also, it looks the same every time, no matter which icon is clicked. I would then assume (and have implemented the code this way in jPhone) that the icons all shoot away from one central point on the screen. But that doesn't appear to be the case.

Anyway, I think the animation for jPhone's main menu looks pretty good, if not exactly what they happen to do on that other device.

The basic idea in jPhone is to calculate the movement vector based on some movement "center" (xCenter, yCenter) and the location of each icon (xOffset, yOffset):

        double xDir = xOffset - xCenter;
        double yDir = yOffset - yCenter;

We can then calculate the new offscreen position of the icon (xOffscreen, yOffscreen) using this direction vector (I won't show that code here for brevity, but it basically sets an offscreen value in one coordinate (x or y) and then calculates the other coordinate based on the direction vector).

Finally, we can create an animation that will move the icon from its position in the main menu to this offscreen position as follows:

        Clip iconClip = Clip.create(MENU_OUT_TIME,
                transformNode, "translateX", xOffscreen);
        iconClip.addTarget(KeyFrames.create(
                new BeanProperty(transformNode, "translateY"), 
                yOffscreen));
        menuOutTimeline.schedule(iconClip);

The transformNode object being animated is the filter node in charge of the icon location, so this animation will alter the translateX and translateY properties of that object during the animation. Similarly, we create the opposite animation to move the icon back to its original location from where it's residing offscreen:

        iconClip = Clip.create(MENU_IN_TIME,
                transformNode, "translateX", xOffset);
        iconClip.addTarget(KeyFrames.create(
                new BeanProperty(transformNode, "translateY"), 
                yOffset));
        menuInTimeline.schedule(iconClip);

Tray Menu Animation

Tray menu animation is simpler; we just fade the tray out and back in when applications become active or inactive. To fade the tray out, we create an animation on the opacity property of the opacityNode object that we created earlier:

        Clip fader = Clip.create((int)MENU_OUT_TIME, opacityNode, 
                "opacity", 1f, 0f);
        fader.addTarget(new TimingTargetAdapter() {
            public void end() {
                trayGroup.setVisible(false);
            }
        });
        menuOutTimeline.schedule(fader);

Note that we're actually doing two things here; we're fading out the node from opaque to completely transparent, and we're setting the visibility of the node to false when the animation ends. The visibility property of nodes controls whether the nodes process events or try to render themselves; since the node will be completely transparent when it is faded out completely, it doesn't make sense for it to participate in either events or rendering.

When an application becomes inactive, we run the reverse animation on the tray menu to make it visible and fade it in:

        fader = Clip.create((int)MENU_IN_TIME, opacityNode, 
                "opacity", 0f, 1f);
        fader.addTarget(new TimingTargetAdapter() {
            public void begin() {
                trayGroup.setVisible(true);
            }
        });
        menuInTimeline.schedule(fader);

Application Animation

Finally, we need to animate the application screen. These animations are similar to what we've seen before, although in this case we are both fading and scaling the application screen:

        final Timeline appAnims = new Timeline();
        Clip fader = Clip.create((int)MENU_OUT_TIME, opacityNode, 
                "opacity", .1f, 1f);
        Clip scaler = Clip.create((int)MENU_OUT_TIME, scaleNode, 
                "affine", smallScale, fullScale);
        appAnims.schedule(fader);
        appAnims.schedule(scaler);
        fader.addTarget(new TimingTargetAdapter() {
            public void begin() {
                photo.setVisible(true);
            }
        });

The application animation is also how we start tying the different animations and events together. First of all, we kick off the overall menuOutTimeline animation from the application animation by scheduling it as a dependent animation of the application's fader Clip, as follows:

        fader.addBeginAnimation(menuOutTimeline); 

Next, we add an attribute to each icon that tells it which animation it is associated with:

        appIcon.putAttribute("startAnim", appAnims);

Finally, we add a mouse listener to each icon that will listen for clicks and start the animation appropriate for that icon:

        appIcon.addMouseListener(appStartListener);

where the mouse listener is defined to start the animation that we stored as an attribute on the icon in question:

        SGMouseListener appStartListener = new SGMouseAdapter() {
            @Override
            public void mouseClicked(MouseEvent e, SGNode n) {
                Animation a = (Animation) n.getAttribute("startAnim");
                a.start();
            }
        };

We do similarly for the reverse animation for the application (I'll skip that code for brevity and added surprise and excitement).

TransformComposer

Chris might prefer that I not mention this detail, since the final API for the scene graph will hopefully make this step irrelevant, but for now the only way to animate transforms such as the moves and scales shown above is for the animation engine to know how to interpolate AffineTransform objects. It does not know how to do this by default (because, frankly, it's not typically the way you would want to interpolate between positions and orientations; it can produce unexpected results, thus the need for better functionality in the API eventually). So we need to add this capability to the application. We do this by creating a TransformComposer object that tells the system how to interpolate between AffineTransform objects, and we register the composer as follows:

        static {
            Composer.register(AffineTransform.class, TransformComposer.class);
        }

I won't go into the details of TransformComposer here, but see my earlier blog entry on the animation system and the Composer JavaDocs on the Scene Graph project site to understand more about Composer. Basically, a custom Composer simply converts between an arbitrary type and an array of double values, which the base Composer class then knows how to linearly interpolate between.

Runtime

That's mostly it. If you run the application and click on the icons you can see the fading, moving, and scaling animations all working together to show a nice, smooth transition between the menu and application screens.

Of course, in demos enough is never enough. So we decided to put in one more element for fun.

In the Apple video, you'll notice that many of the demos they show are run by this disembodied hand. It could be the Hand of God, but I don't think that Steve was in the video. Besides, the hand isn't wearing a black turtleneck.

It seemed like our demo needed an element like that, so we created the handCursor node.

Handy Cursor

Custom cursors are fairly easy in Java, but they are also fairly limited. In particular, your cursor image is limited to 32x32, which doesn't really give us the effect we were looking for. I want a hand, not a hand-shaped wart. We need a friggin' huge cursor.

In a traditional Swing application, we could manage this using the glass pane, displaying an arbitrary image in that overlay on top of the application GUI. But the scene graph makes this even easier; we just need a shape node.

First things first: we need to manage the real Swing cursor. Since we cannot use the actual cursor as our hand, we will instead make the real cursor invisible with the following code, so that if we can't make the cursor do what we really want, we can at least get it out of the way:

        BufferedImage emptyImage = new BufferedImage(32, 32,
                BufferedImage.TYPE_INT_ARGB);
        invisibleCursor = Toolkit.getDefaultToolkit().
                createCustomCursor(emptyImage, new Point(0, 0), "empty");

Next, we will create our handCursor object, parent it to the root node, and add it as a MouseMotionListener on the application as follows:

        handNode = new HandCursor(rootNode);
        addMouseMotionListener(handNode);

I won't show the entire HandCursor class (it's frankly not interesting enough), but the basics are as follows: First, we load the hand image and scale it to an appropriate size (using code similar to that shown earlier for the menu icons). Next, we create an SGImage node and a transform node (for moving it), and parent the transform node to the root node. We also make the shape invisible at first, since the hand cursor is not showing by default:

        handNode = new SGImage();
        handNode.setImage(largerHand);
        handNode.setVisible(false);
        handTransform = SGTransform.createTranslation(0, 0, handNode);
        rootNode.add(handTransform);

When the application frame detects that the key "h" has been typed, it makes the default cursor invisible and the hand cursor visible with the following:

        setCursor(invisibleCursor);
        handNode.setVisible(true);

Now, all we have to do is track the mouse position and display the hand node appropriately:

        public void mouseMoved(MouseEvent me) {
            mouseX = me.getX();
            mouseY = me.getY();
            handTransform.setTranslation(mouseX - 160, mouseY - 5);
        }

(where the hard-coded numbers in setTranslation() position the "hotspot" of the hand cursor at the tip of the index finger).

return;

That's pretty much it. There's more code in JPhone, but I think I've covered all of the interesting scene graph-related pieces above. Play with it, check out the code, and write a phony scene graph application of your own.

Related Information

JPhone Demo: a Java Web Start application

JPhone.java: the source code for the main demo class

TransformComposer.java: the source code for the custom composer helper class

Scene Graph Demos: The project site for all of the Scene Graph demos posted by the team (including JPhone)