campbell's Blog

Effects in JavaFX: Chaining

Picking up where I left off six (cough, cough) months ago, in my series on the effects framework in JavaFX...

In JavaFX, you can chain effects together, contrary to a couple blog entries I've come across recently. This feature may not be entirely obvious from the API documentation, so let's look at chaining in a bit more detail here.

Just like Nodes in the JavaFX scene graph, Effects can be linked together in a tree-like structure. Most Effect subclasses expose one or more "input" variables that allow you to chain Effect instances together. By default, the input variable is set to null, indicating that the rendered contents of the Node to which the Effect is attached will be used as the input to the effect. For example, in the following code snippet, since we do not set the BoxBlur.input variable, the blur effect will be applied to the Text node content:

    Text {
        effect: BoxBlur {}
        content: "Hello!"
        ...
    }

If we instead explicitly set the BoxBlur.input variable to, say, a SepiaTone instance, the sepia effect will first be applied to the node content, and the result of that operation will then be blurred:

    Text {
        effect: BoxBlur {
            input: SepiaTone {}
        }
        content: "Hello!"
        ...
    }

The following is a simple applet that demonstrates what a Text node looks like when one or more effects are chained to it:

(The source bundle for this demo is available here.)

Note that in some situations, the order of the chained effects can be quite significant. One such scenario is in "cover flow" style components, where each node has a Reflection effect and is positioned in (faux) 3D using a PerspectiveTransform. In this case, it is important to apply the reflection effect first, and then transform the result last. If you try it the other way around, the result will be goofily [goofily, really?] non-realistic. The applet above demonstrates the difference in behavior of the two orderings.

Unfortunately, due to a minor technical limitation, we have not yet exposed an input variable on the DropShadow and InnerShadow classes as of the 1.2 release. This will certainly be resolved in an upcoming release. In the meantime, you can achieve the same result (albeit with a bit more code) by using nested Groups. For example, we can apply a Lighting effect to a Text, and then wrap that in a Group that has a DropShadow applied to it:

    Group {
        effect: DropShadow {
            radius: 14
            offsetX: 4
            offsetY: 4
        }
        content: Text {
            effect: Lighting {
                light: DistantLight { azimuth: -135 }
                surfaceScale: 5
            }
        }
    }

In the next few installments, I'll look at individual effects such as Lighting and Reflection in more detail. (Let's hope this time I get to it sooner than six months from now!)

In my ears: Charles Mingus, The Black Saint And The Sinner Lady


In my eyes: Ricky Gervais and Karl Pilkington, The World of Karl Pilkington

Effects in JavaFX: The Basics

Still Alive

In early 2008, I wrote my only blog entry of the year, on the subject of the filter effects framework that serves as part of the foundation of JavaFX. It has been a fun project to work on, and I'm fairly proud of the result. There was so much more I wanted to write about on the subject, but I wanted to wait until we first had a stable API and delivered an actual product.

It was a long road to JavaFX 1.0, but now that it has shipped, I can finally talk more about using filter effects in JavaFX and what's going on behind the scenes. Since it's a big subject, I'll try to break things up into smaller, more focused blog entries instead of my usual monolithic tomes. (At minimum, it will provide the illusion that I'm more prolific in 2009 than I was in 2008.)

Like any good graphics API, the filter effects framework in JavaFX is easy to use, provides high-quality visual results, and offers top-notch performance. In this entry, I'll just focus on the ease-of-use angle.

Bunny Jump, Bunny Jump

The best place to start is to look at the javafx.scene.effect package API. All of the core filter effect classes extend the abstract javafx.scene.effect.Effect base class. Any Effect instance can be applied to a scene graph Node by setting the Node.effect variable. Each Effect subclass exposes a small number of variables that control the visual appearance of the Node. In addition, most Effect subclasses provide one or more input variables that can be used to "chain" effects (more on that in a future blog entry).

To take a specific example, the DropShadow class provides 5 variables, analogous to knobs on a control panel: color, offsetX, offsetY, radius, and spread. Each "knob" is generally documented with default, minimum, maximum, and identity values, which makes it easy to see the acceptable range of values. Here's a really simple example that shows how to apply a DropShadow effect to a rounded Rectangle and control its appearance through the magic of the bind operator.

And here's the relevant code (or download the complete sources here):

<pre>
            Rectangle {
                effect: DropShadow {
                    radius: bind radius
                }
                x: 50 y: 30 width: 150 height: 100
                arcWidth: 40 arcHeight: 40
                fill: Color.RED
            }
</pre>

That's it! Pretty easy, isn't it? You, the developer, only need to concern yourself with turning these simple knobs. The effects framework does all the heavy lifting for you, such as painting the Node into an offscreen image at the appropriate resolution, manipulating the pixels in one of the CPU and/or GPU accelerated backends, and automatically repainting the affected portion of the scene.

For more interactive examples of the built-in Effect subclasses, check out the EffectsPlayground applet featured in the JavaFX Samples Gallery. Stay tuned for entries on quality, performance, and more...

In my ears: No Age, Nouns


In my eyes: Woody Allen, Mere Anarchy

Faster Java 2D Via Shaders

We've been doing a lot of work over the past couple years to accelerate many complex Java 2D operations on the GPU via OpenGL fragment shaders. (Fragment shaders are little programs that operate on each pixel being rendered on the screen; they're infinitely more flexible than the fixed functionality that's historically been available in OpenGL.) The sky's the limit when it comes to the kind of effects one can achieve by writing shader programs.

The first GPUs with fully programmable shader support were made available from Nvidia and ATI in 2002. It took a couple years for the new hardware to penetrate into the consumer market, but now shader-capable GPUs are extremely prevalent (thanks in part to the hefty graphical requirements of Mac OS X and Windows Vista), so much so that we can reliably take advantage of shaders to accelerate complex Java 2D operations. Even those first-generation boards are capable of providing huge performance gains, and each new generation of hardware seems to give an order of magnitude improvement over the last. This should be quite evident from the charts that follow. While CPU speeds seem to be nearing an asymptote, GPU performance continues to rocket, and now Java 2D is able to benefit from that power. Not only does this mean improved performance for your Swing or Java 2D application, but also reduced CPU consumption, thus freeing up your CPU to crunch on application logic rather than getting bogged down with rendering tasks.

[This is one of those blog entries that could be novel length, but no one would actually read the words, because there are too many pretty bar charts to distract the reader. Blah blah blah, words words words. See? No one's reading. So let's skip the prose and get on with it... Oh, but first, I have to tell you how to read these charts. I generated these numbers on a couple different machines, using J2DBench on Windows XP with the latest graphics drivers (ATI Catalyst 7.3 and Nvidia 93.71). Since the machines vary slightly in processor performance and bus speed (the GeForce 7800 is a PCI-E board, the rest are AGP), I decided to use our software pipeline as a baseline, and then compare the OGL pipeline numbers to that baseline. For example, if you see a result that lines up with the number 2000, it means that test is 2000% of baseline, or in other words, it is about 20 times faster on the GPU than on the CPU. Your mileage may vary, but the big takeaway is that most operations are many times faster when executed on the GPU...]

Text Rendering
(Bug 6274813: Available in JDK 6)

LCD-Optimized Text

I already discussed this a bit in a blog entry from about a year ago. Since then, we've enabled this by default in JDK 7 (and soon in a JDK 6 update) when the OGL pipeline is enabled. It's cool to see how software performance improves little over time, but each new generation of GPUs brings big performance improvements.

BufferedImageOps
(Bug 6514990: Available in JDK 7 b08)

ConvolveOp

ConvolveOp is commonly used for modern UI effects such as blurs and drop shadows. Due to limitations in first-generation shader-level hardware, we are currently only accelerating ConvolveOp for 3x3 and 5x5 sized Kernels. These are fairly common kernel sizes, but most drop shadow and glow effects require larger kernels, so we are working to loosen these restrictions and accelerate a wider range of kernel sizes.

LookupOp

LookupOp is often used to perform simple brightness and contrast adjustments on images. To simplify our code, we are currently only accelerating LookupOp for ByteLookupTables and ShortLookupTables with a maximum length of 256 elements, and for 1-, 3-, and 4-band sRGB images only.

RescaleOp

RescaleOp is basically a degenerate case of LookupOp that can be accelerated very efficiently in shaders; after all, it's just a multiply and an add. We are currently accelerating RescaleOp for 1-, 3-, and 4-band sRGB images only.

Multi-stop Gradient Paints
(Bug 6521533: Available in JDK 7 b10)

LinearGradientPaint

For 2-stop linear gradients (NO_CYCLE or REFLECT), we can delegate to our existing GradientPaint codepath, which is already ridiculously fast via OpenGL's fixed functionality. For all other linear gradients (for all CycleMethods and ColorSpaceTypes, up to a maximum of 12 color stops), we can accelerate the operation using shaders, for both antialiased and non-antialiased rendering.

RadialGradientPaint

The same restrictions for linear gradients also apply to radial gradients (maximum of 12 color stops, etc). For gradients with more than 12 stops, we simply fall back on our existing software routines.

Extended Composite Modes
(Bugs 6531647, 5094232: On the way)

Antialiased Painting (with Non-SrcOver AlphaComposite)

Historically the OGL pipeline has been able to accelerate the compositing step of an antialiased painting operation only when AlphaComposite.SrcOver (or AlphaComposite.Src, if the paint is opaque) due to the math involved. (This is described in the quasi-official guide to the OpenGL-based Java 2D pipeline, but that document could use a refresh for JDK 6 and beyond.) But now with shaders we're able to accelerate antialiased rendering for any arbitrary AlphaComposite mode.

Coming Soon: PhotoComposite

It's not officially approved (or integrated) yet, but I've been working on adding more blending modes to Java 2D, in addition to those already provided by AlphaComposite. Many of these modes come from traditional photography techniques, thus the name "PhotoComposite". Some modes are simple (Add, Multiply) and can be accelerated easily using OpenGL's built-in blending rules, others are more involved (ColorBurn, SoftLight) and benefit greatly from the use of shaders for efficient rendering.

What's Next?

There are plenty more optimizations that can be made to common Java 2D operations by leveraging shader technology; we'll keep working on this. Also, there have been some discussion on the interest list recently about the use of shaders in Java 2D. Some folks would like to be able to write arbitrary shaders and have them work on Java 2D content. I think it would be hard to come up with a general solution (in the public API) to make this work everywhere, and would shift an unreasonable burden to Java 2D (which is designed with WORA in mind).

However, I do recognize that it would be great if it were easy for developers to make use of shaders in their applications (as Romain has demonstrated) without getting bogged down in the OpenGL/GLSL learning curve. To that extent, I've been working on a few utility classes for JOGL in the vein of TextureIO and related classes. This is another way to make the transition easier for existing Swing developers to leverage JOGL in their applications. More on that in a future blog entry.

Finally, it's worth mentioning that all the shader-based optimizations I've described above are currently only available for the OpenGL-based Java 2D pipeline, but that will soon change. For JDK 7 (maybe sooner?), we have a newly redesigned Direct3D-9-based pipeline in the works that will share much of the architecture (and code) of the OpenGL-based pipeline. We fully expect that all of these shader-based optimizations will be available for most Windows users in the near future. Stay tuned.

In my ears: Panda Bear, "Person Pitch"


In my eyes: JPG, Issue 9

Swing and GTK: What a LAF!

[If it's not readily apparent, I wrote this blog entry months ago because I was waiting around for the bug fixes described here to make it into JDK 7. So just as a brief diversion from your day, pretend for a few moments that you've traveled back to the distant past, specifically the trail end of the year 2006...]

How I Spent My Summer


The big problem with my job at Sun is that I often simultaneously have ten different toes dipped into ten different puddles. This past spring I made one final splash in my primary puddle, the OpenGL-based Java 2D pipeline, to fix the remaining big ticket items and make it as solid as possible for the JDK 6 release. By and large those goals were met (for more info, go crazy, go crazier, or um, go more crazier), but along the way I experienced a bit of burnout. After all, I'd been hacking on the OGL pipeline since late 2001 (and pondering it as early as 1999 as evidenced in this posting by some clever, dare I say dashing, intern) so it's surprising that the burnout didn't happen sooner.

Anyway, with JDK 6 in a frozen state and summer rapidly approaching, I figured that it would be a good time for a change, but (oh!), what to work on? I started with a blank slate, but quickly found my toes overdipped. Here's a partial list of my non-OGL dabblings over the last six or so months:

• Finished implementing the new Java Plugin startup animation (check it out soon in JDK 6u1 and JDK 7, with more tweaks to follow)
• Kicked off my "Trickery" blog series (here and here) with some techniques used in the aforementioned animation work
• Toyed around with the idea of reimplementing our native rendering loops in pure Java code (yes, believe it or not there are advantages; definitely worth revisiting)
• Played with the SwingX Painters API and Timing Framework (partial success documented in my PhotoCube blog)
• Started investigating new icon-related APIs in Swing (for L&F provided icons and multi-res icons)
• Installed Ubuntu and quickly found that, yes indeed, Swing's GTK L&F could use some work (despite the big improvements already seen in JDK 6 in that area)

This list serves dual purposes. First, it's a pathetic attempt to prove to my manager that I did accomplish something this summer besides, um, building an Adirondack chair and paddling around Shook's Pond. More relevantly, it demonstrates that I have issues with focusing on one concrete project. However, the one item that showed the most potential (and had well-defined constraints) was the set of GTK L&F deficiencies. Ubuntu wasn't a tier-one testing platform for us in JDK 6 (something I hope will change for JDK 7), and it turns out that Ubuntu's 6.06 release introduced a new GTK engine (ubuntulooks) and theme (Human). The ubuntulooks engine is different enough from the clearlooks engine (from which it was derived) to warrant a close look, and it would require a number of special fixes to make Swing's GTK L&F look great on that distro.

Another reason to focus on a concrete project like the GTK L&F is that there's been an effort underway for some time by the NetBeans team to enable the GTK L&F by default on Linux and Solaris, just as Mac OS X and Windows use the platform L&F by default. After all, if Sun doesn't set an example by enabling the platform L&F in all of our desktop-focused software, how can we expect developers to do the same in their apps? Again, despite all the major improvements to Swing's GTK L&F in JDK 6, there were a few issues that slipped through the cracks, and we wanted to make sure those are addressed soon (ideally in the upcoming JDK 6 update releases). Expect to see NetBeans make this switch in their 6.0 release (or thereabouts) in Spring 2007.

[Update: The NetBeans folks just this week flipped the switch in NetBeans 6.0 M7 so that the GTK L&F is now enabled by default on Solaris and Linux (on JDK 6u1 and above). Check out the "new and noteworthy" page for a few sample screenshots and more information. To get things looking good in time for the NetBeans 6.0 release, a lot of energy was invested by the NetBeans, Solaris, and Swing teams. Thanks guys!]

Before and After Science


All right, all right, enough words already. As the saying (rarely) goes, a screenshot's worth a thousand words, so it's time for some before-and-after shots (click to enlarge):

Fig. 1 -- Native "twf" demo application.







Fig. 2 -- Java/Swing mockup of "twf", courtesy of Elliott Hughes, running on JDK 6.







Fig. 3 -- Same as above, but running on JDK 7-b08; can you spot the 6 (or 25) differences?

These particular screenshots were taken on an Ubuntu 6.06 system with the default Human theme, but similar improvements can be seen on many other GTK themes such as Nimbus (the new default on Solaris), Clearlooks (the most popular on various Linux distros), and so on. In total, over 30 GTK-specific bugs have been fixed so far in JDK 7, and there are plenty more fixes on the way. For more details on these bug fixes, refer to this list. As you can probably see from Figure 3, there's still a bit more work to be done (e.g. table headers, non-editable comboboxes, tabs), but we're getting closer.

Help Wanted


Here's the thing about the GTK L&F... It's not rocket science. Granted, some of the issues are tricky to fix, which is why it's taken quite a while to get it to where it is today. It's not for lack of smarts, but rather a lack of time. It can take a number of hours just to track down exactly why something is broken in the GTK L&F, and it can take up to a couple days to devise a robust and well-understood fix.

I hope these statements don't come across as whining. The fact of the matter is that at Sun we have anywhere from 1.33 to 2.117 persons available to work on the GTK L&F at any given time. Since time is the limiting factor, there's only so many fixes we can get in within a given year, maybe 50 tops. But what if we took a page from the book of CMT and increased the number of threads crunching on these issues in parallel? I think we could see some quick progress.

There are a number of people (you know who you are, Elliott Hughes; nudge nudge, Brian Harry; wink wink Roman Kennke) who are actively interested in the success of the GTK L&F. That's great, but I'd like to see more activity in this area, perhaps in addition to the aforementioned/illustrious names. And not just in the form of an independent project. We'd be happy to work closely with 1 or 2 (or 3) developers out there in the OpenJDK community, collaborating on fixes, and so on. Eventually it would be nice to see a couple folks in the community with the same level of expertise as we have in Kirill K. and Peter Z. (our resident GTK experts on the Swing team at Sun).

At the very least it would be a fun experiment as we move towards completely opening up the JDK library source code in the next few months. As has been stated elsewhere, we still have a lot of work to do w.r.t. opening up our internal processes (bug evaluations, code reviews, etc) to the outside world, but I think it would help the transition if we had a few external folks get more involved with this process now. Send me email if you'd like to get involved. I can't promise you fame or fortune, but I can promise you a beer or two (*, **, ***, ****, *****) in return for your hard work.

* Valid only in the Bay Area, California, unless you want to fly me to your location. (I'd gladly oblige, especially if you're located outside the U.S.)


** Under 21 receives R.C. Cola or equivalent.


*** Pabst Blue Ribbon or equivalent only.


**** Must be happy hour or half-off equivalent.


***** You must buy me a beer in return, because that's the way the system works.

In my ears: Deerhoof, "Reveille" [digging through old favorites...]


In my eyes: Woody Allen, "Without Feathers"