Java Media Components
Making Java a Player
Video used to be a feature that we lacked:
Now, it seems sometimes that it is the feature that we lack:
In the current world of YouTube, Flash, and iTunes, video has gone from a niche
capability to a checkbox item that all platforms must support.
So what are we doing about it, besides pointing out the obvious issue that it
isn't there now?
If you attended the Java Technology Desktop overview session at JavaOne this
year, you learned, along 500+ of your closest friends in the room,
that we're working on a new project that we call Java Media Components(JMC).
This new feature, hopefully shipping in Java SE 7, is intended to support basic
playback support for Java applications. JMC is also, eventually, intended
to address capture and streaming capabilities.
There are actually two levels of playback that we would like to support:
The native players on most platforms have substantial support for multiple video
formats. It would not make sense (read: it would cost serious $$$) for Java to
implement players for some of these existing formats. So why not leverage the
native platform capabilities instead?
The idea is to expose a simple API for playback control (e.g., play, stop,
and rewind), and to allow a Java application to thus control playback of
specified content through whatever was available on the native system. For
example, playback of an AVI file on Windows might launch Windows Media Player.
Moreover, the native player would be exposed through a GUI component that you
could integrate into the rest of your application's interface. There
would also, depending on the underlying native support, be options to
either expose the native controls for the player (e.g., the Play button), or to
allow the developer to skin their own controls around the player component.
This native-wrapper functionality is the easiest part of the whole JMC
project. The main work is in the simple API design and the glue code to the
various players on different platforms. In fact, we already have prototypes of
this working for several players, including Windows Media Player and Flash.
Curious readers familiar with lightweight/heavyweight mixing issues in
Swing applications might wonder at this point how we will deal with a native
player component in a Swing GUI. Good question, thanks for asking. This
issue is one of the reasons that we also plan to support mixed
heavyweight/lightweight components in Java SE 7. In case I don't write a
blog on this feature soon, here's the quick scoop: We can enable this through
clipping the heavyweight components against the shapes of the overlaying
lightweight components. There are some cases that do not yet work with this
mechanism, such as shaped and translucent lightweights, but the system works
pretty well in general for other cases.
Playing Nice with Java
Supporting common formats through the native player mechanism will
solve a key part of the problem. The ability for a Java application
to easily play video that resides on a user's local system through
a standard local player is an important case that will
be handled through this simple mechanism.
But native player support is not a complete solution. For one thing,
if a developer wants a specific video to play through the application, they
will have to ensure that the video is available in whatever formats will
be supported on all of their users' native platforms. For example, a Quicktime
movie would not be playable on Windows unless Quicktime happened to
be installed on that system. Similarly, an AVI file may not have much
luck playing on Linux.
To solve this problem, we need one common format, and an accompanying codec
(coder-decoder), so that any Java application can count on being able to play
back any content in that format, regardless of the runtime platform.
This cross-platform format support is a much more involved issue, since it
immediately gets into harder issues of:
Licensing technology from existing player developers, and/or
Creating or integrating new technology into the platform
We plan to tackle this part of the JMC project after the native players are well
in hand. Meanwhile, we are trying to figure out the right solution (which
formats, which technologies, etc.).
Playback versus Pixel Access
There is an important consideration in both the native and Java player solutions
above: How much do players need to simply display their pixels on the screen,
and how much do they need to actually expose the raw pixel data? To a great
extent, simply playing on the screen is probably good enough. If an application
just wants to play a video, then blasting it to the user's screen is fine. But
there are subtle cases here that make the issue a bit more complicated.
For example, what if you wanted to do something more interesting with the video
content, like play it on a lightweight Swing widget? Remember that I said that
we plan to allow mixing of heavyweight and lightweight components, which should
allow a JMC player to at least cooperate with Swing widgets. But that's not the
same as a JMC component actually being a Swing component. For example,
what if you want to use the video as the icon in a JButton? Or what if you want
the video playing in a translucent component? Or what if you want to get really
crazy and display video in a 3D object, via JOGL?
For any of these cases where you want to do anything with the pixels besides
simply show them on the screen, you will need access to the pixel data from the
video stream. Many of the native players do not offer this capability, but
merely allow the controls of playing, rewinding, and stopping the
video. Some native systems, such as DirectShow, do allow more flexibility, so
we plan to use these approaches when we can. And the Java Player will certainly
allow this capability. Since we will provide the decoder for the video through
the Java player, we can have those pixels appear on the screen, or in an image,
or wherever the application specifies.
What about Capture and Streaming?
Capturing video from an input device and streaming between Java
clients are two areas that we would also like to support eventually.
Having the Java player is actually a good initial step toward these
capabilities, since we will be able to encode and decode into our
cross-platform format. But we are taking baby steps here, and feel that the top
priority is solid playback support. We would like to get rolling on that
playback project first before we take on even more, and harder, issues.
What about Java Media Framework (JMF)?
JMF is an existing media library, provided as an optional package on top of Java
SE. JMF is a very large and powerful media development platform that is
capable of doing much more than I've discussed here. In particular, while
it's possible to write a complete video editing solution in JMF, we are clearly
going nowhere near that level of functionality with the JMC project, but rather
focusing squarely on playback at first. Again, we are concentrating on
attacking the pain points in our existing video support (or lack thereof), and
we feel that playback is simply much more in demand than full-on video editing
One solution here might be to simply revamp our investment in and support of JMF
itself, and make it the platform for future video endeavours. However, the
level of effort that we would need to invest to bring JMF up to the
current level of requirements, including providing codecs for modern formats,
would be huge. Also, at about a 2 Megabyte download size now, we're not sure
that simply adding in the current JMF to the core platform is the right
approach. It seems better to solve the top priority of simple playback by a
targeted solution that is possible with a much smaller footprint impact.
We may actually implement parts of the JMC stack on portions of JMF, which would
give us the advantage of re-use of some of JMFs capabilities without taking on
the entire library. Also, work on codecs for JMC could end up also benefitting
the JMF library eventually in any case.
That's the Plan
Plans change, of course, but this writeup represents what we would like to
see happen in Java SE 7. Does it sound reasonable to you? Would having simple
playback functionality, through both native players and a new
Java player, satisfy your craving for media in Java? And if we spin
up a new project on SwingLabs soon,
would you be interested in helping out in the design and implementation of