Fabrizio Giudici's Blog

Let's discuss about Java imaging

I was happy to see that Slav Boleslawski's article about a specialized component for rendering photos made its way to the featured articles of this week. Indeed if Java wants to gain attention on the desktop, the whole media segment can't be ignored.

Looking around I think we can notice some early warnings that people is indeed working on this area. Today I finished editing a blog post about a component for photo management and rendering, and I included in the article a quick panorama of some existing Java imaging applications (both commercial and opensource) and of the most important Java imaging APIs. I think it would be a good thing if people working both on commercial and opensource imaging application could meet in some specialized forum - not aimed at a particular technology (so I'm not thinking of the Java2D or JAI standard forums), but for sharing the experience and the open problems.

Yes, because in my opinion there are some open problems (I discussed some of them in my post). Apart from a general lack of information (or a difficulty in retrieving that information) about how to use the imaging APIs (a recurring problem is about high-quality image resizing), there are some inconsistencies in how the APIs behave in different systems. For instance, take this table with a few compared performance results about some basic operations performed with Java2D (keep them with a pinch of salt, but I'm starting to be confident with them):

The performance has been measured on a single Mac Mini Dual Core with a triple-boot, so you can actually compare results from the different operating systems. As you can see there are some wild differences, pointed out by the red color. Operations marked with "opt" are performed on optimized images, that is an image made with GraphicsConfiguration.createCompatibleImage(). A "compatible" image "has a layout and color model that is closest to this native device configuration and can therefore be optimally blitted to this device" (from the javadoc of GraphicsConfiguration) and using them is one of the good practices of working with Swing. In facts, as you can see, using them leads to dramatic performance improvements on Linux and Windows (OTOH the bad result is several magnitudos worse, and it's clearly related to some bugs). But one operation miserably fails on Linux with an exception. And... surprise, on Mac OS X the optimized image is not always faster (BTW, the test image is a JPEG converted from a 6 megapixels image taken with a Nikon D100 and loaded by standard Image I/O - both the test image and test sources are in Mistral unit tests). On Mac OS X, by default the used imaging core is not the same as Sun, as this partially explains the difference.

I talked about this only twice with other persons on some random forums, and each time I had the impression that this is part of a dubious, common but unshared knowledge. So, my point is: why don't we setup a specific forum where we can discuss this? An excellent opensource project would be to set up a well-written set of tests similar to the one I've just presented, so that they can be authoritative - it would be a great value for all the imaging Java programmers!

Comments

• It would be very useful if you mentioned the Java version you used on each platform, as well as what was the "exception" you got on linux. What was the desktop bit depth on all systems (hopefully it was the same)

  Also, regarding the proper way to scale images, there will be an article by Chris Campbell on java.net soon on this very topic.

  And, there is a Java2D benchmark (J2DBench) in the j2se source code (will be open sourced soon along with the rest of the platform). You can get to it now from jdk7.dev.java.net (it's in j2se/src/share/demo/java2d/J2DBench directory). This benchmark is actually a framework which allows for adding new tests, it would be nice if the community expanded the benchmark coverage instead of inventing a new benchmark every time.

  Dmitri
  Java2D Team
  

  Posted by: trembovetski on March 28, 2007 at 05:29 PM

• HiThis is a great table of performance. It would be excellent if you posted the code which does these operations. With benchmarking Java2D there are many little things to remember like Toolkit.sync(). There is a J2D benchmark class somewhere which Sun has released which may help.Are you actually rendering these these images to the screen? In that case, it would be interesting if you compared the different J2D pipelines such as sun.java2d.opengl=True and sun.java2d.noddraw=true. In the javagaming.org J2D forums we discuss this stuff and I've found that software (noddraw) rendering is the most predictable of the pipelines on Windows. The default pipeline can be quite slow on some machines.Great article, Keith

  Posted by: commanderkeith on March 28, 2007 at 05:36 PM

• The code can be checked out from Mistral unit tests; I'm copying the most significant code excerpts related with the table above.

  This is about AffineTransform:

  
  protected long runTest (EditableImage image)
  {
  long accTime = 0;
  
  for (double scale = 0.1; scale <= 1; scale += 0.1)
  {
  long time = System.currentTimeMillis();
  Java2DUtils.scaleWithAffineTransform(bufferedImage, scale, scale, Quality.INTERMEDIATE);
  accTime += System.currentTimeMillis() - time;
  }
  
  return accTime;
  }

  ...

  public static BufferedImage scaleWithAffineTransform (final BufferedImage bufferedImage, final double xScale,
  final double yScale, final Quality quality)
  throws IllegalArgumentException
  {
  final AffineTransform transform = AffineTransform.getScaleInstance(xScale, yScale);
  final int interpolation = findAffineTransformInterpolation(quality);
  final AffineTransformOp op = new AffineTransformOp(transform, interpolation);
  return op.filter(bufferedImage, null);
  }

  
  and this is for DrawImage:

  
  protected long runTest (EditableImage image)
  {
  long accTime = 0;
  
  for (double scale = 0.1; scale <= 1; scale += 0.1)
  {
  long time = System.currentTimeMillis();
  Java2DUtils.scaleWithDrawImage(bufferedImage, scale, scale, Quality.INTERMEDIATE);
  accTime += System.currentTimeMillis() - time;
  }
  
  return accTime;
  }

  ...

  public static BufferedImage scaleWithDrawImage (final BufferedImage bufferedImage, final double xScale,
  final double yScale, final Quality quality)
  throws IllegalArgumentException
  {
  int newWidth = (int)Math.round(bufferedImage.getWidth() * xScale);
  int newHeight = (int)Math.round(bufferedImage.getHeight() * yScale);
  BufferedImage result = createCompatibleImage(bufferedImage, newWidth, newHeight);

  Graphics2D g2d = (Graphics2D)result.getGraphics();
  Object interpolation = findRenderingHintsInterpolation(quality);
  g2d.setRenderingHint(RenderingHints.KEY_INTERPOLATION, interpolation);

  try
  {
  g2d.drawImage(bufferedImage, 0, 0, newWidth, newHeight, null);
  }
  finally
  {
  g2d.dispose();
  }

  return result;
  }

  ....

  public static BufferedImage createCompatibleImage (final BufferedImage bufferedImage, final int newWidth,
  final int newHeight)
  {
  SampleModel sampleModel = bufferedImage.getSampleModel().createCompatibleSampleModel(newWidth, newHeight);
  WritableRaster newRaster = Raster.createWritableRaster(sampleModel, null);
  ColorModel colorModel = bufferedImage.getColorModel();

  return new BufferedImage(colorModel, newRaster, false, Java2DUtils.getProperties(bufferedImage));
  }

  
  </code>

  The methods find****Interpolation() returns AffineTransformOp.TYPE_BILINEAR in the former case and RenderingHints.VALUE_INTERPOLATION_BILINEAR in the latter.

  Posted by: fabriziogiudici on March 28, 2007 at 11:03 PM

• In any case, the JDK used is 1.5.0 - at the moment of the test that produced the above results it should have been _08 for Windows and Linux; _06 or _07 on Mac OS X. The test image is https://mistral.dev.java.net/svn/mistral/trunk/www/images/20030701-0043.jpg. As soon as I find some time (unfortunately those very high numbers mean that the test last about one hour) I'll run the tests again with the latest JDKs.

  The exception I got (sorry at the moment I don't have the Linux machine at hand) is something related to a java.awt.image.ImagingOpException: Unable to transform src image, being the implementation of the image sun.awt.X11.PeerImage or such.

  No, during tests the images are not rendered on the screen, this is just a measurement of back-end performance.

  In any case, the purpose of my post wasn't to focus on the concrete problem, rather to talk about the general perspective of imaging. I'm longing to see Chris' post, and sure I'm going to check out jdk7.dev.java.net - having a single test suite is precisely my point. Is it possible to run that test suite also on JDK 5? I suppose it works also on JVM not made by Sun, such as Apple's.

  Posted by: fabriziogiudici on March 28, 2007 at 11:21 PM

• Final note: the "createCompatibleImage()" nome above is misleading, as "compatibile images" are another thing. I've changed it right now.

  At last, the images converted to the "opt" format are created with the following (usual) code:

  
  GraphicsEnvironment ge = GraphicsEnvironment.getLocalGraphicsEnvironment();
  GraphicsDevice gs = ge.getDefaultScreenDevice();
  GraphicsConfiguration gc = gs.getDefaultConfiguration();

  return gc.createCompatibleImage(width, height);
  

  Posted by: fabriziogiudici on March 29, 2007 at 12:10 AM

• "What was the desktop bit depth on all systems"

  32 bits in every system. I was wondering whether the exception could be related to the X11 video card drivers? Configuring them on Linux is always a tricky thing (even though the video card looked fine just right the default Ubuntu installation and I hadn't to do anything more).
  

  Posted by: fabriziogiudici on March 29, 2007 at 12:15 AM

• Hi Fabrizio,

  I like the idea of a forum / user community dedicated to Java imaging. Especially the idea of founding a project for best-practice Java imaging coding is very appealing. If you need contributors, you can count me in.

  Moritz.

  Posted by: m_petersen on March 29, 2007 at 03:37 AM

• Good. I'd like to have feedback about this from the Sun guys - if this initiative starts it must be a complement to their work, and neither a duplicate nor a disturbance.

  Posted by: fabriziogiudici on March 29, 2007 at 03:43 AM

• @trembovetski: I've downloaded the JDK7 sources and I have opened J2DBench with NetBeans. I see a problem: the test suite is (quite obviously) targeted at Java 6 (and I presume it will be at Java 7 when/if there will be changes). This makes it completely unusable for JDK 5, and I think that a lot of people will still work in Java 5 for some time.

  I'm dubious (also because - grrr - Apple still didn't release Java 6 final and I can't try them).

  Posted by: fabriziogiudici on March 29, 2007 at 05:53 AM

• @fabrizio: What makes you say "quite obviously"? J2DBench is designed to run on old JDKs (going back to 1.2, and it can even handle 1.1 with some slight tweaks). Check out the build.xml file: we compile with source/target=1.2. The code does test newer APIs (e.g. Linear/RadialGradientPaint), which means it needs to be built with JDK 6, but it does the appropriate runtime checks so that it can run on all older JDKs. So if you're tied to the Mac and you don't have JDK 6 installed, you can build the JAR once on some other machine and then use it perfectly fine on the Mac.

  Posted by: campbell on March 29, 2007 at 08:34 AM

• Ok, I'll try this evening. Thanks for the information.

  Posted by: fabriziogiudici on March 29, 2007 at 09:01 AM

• One comment about the code you posted: it appears that it's timing pure rendering time, it also includes the time it takes to create a destination image - which it for some reason does on every iteration.

  Dmitri
  

  Posted by: trembovetski on March 29, 2007 at 11:18 AM

• Yes. If you have to process a batch of pictures (i.e. resizing and rotating them) it looks to me the most useful test (in contrast to reusing the same destination image when you're doing some animation as the target is always the same and the processing synchronous). Of course more tests would be needed, including reuse of the target image

  In any case, I think that the comparison between AffineTransform and DrawImage is still fair, since op.filter(bufferedImage, null) also creates a new BufferedImage with the same characteristics, right? The most important point with those tests IMO is not the absolute but the relative values.

  PS I've started looking at J2DBench, it's interesting. I've not inspected the whole source, but I think it needs some integration with tests that manipulates different kind of photographs (including 16-bit ones). Where is the right place to discuss about J2DBench?

  Thanks.

  Posted by: fabriziogiudici on March 29, 2007 at 11:39 AM

• I think having a place to share such information is a good idea. You might look around the net and see what model for sharing and posting information works for you--I say this because I find forums and mailing lists are good for discussion, but frustrating to retrieve information from after the fact (outside of the fact that they are usually well-indexed). I personally don't find wikis very useful unless the community is very active. One source I've returned to over the years has been Java almanac--which isn't a community site, but I think is a good example of how to share small useful tidbits of information. At one time I also thought JGuru was pretty useful. Anyway, I'm bringing this up because the idea is good, but I'm not sure that a forum or discussion group is the best way to capture and share the good ideas and recommendations you find. Good luck! I like your blog. Patrick

  Posted by: pdoubleya on March 29, 2007 at 01:52 PM

• @pdoubleya: I agree, forum and mailing lists are not good, my idea is a place where people finds documents well structured and (as people work on them) complete. So they don't have to search and reassemble a lot of tiny pieces of information.

  I'm not enthusiastic about wikis too - I prefer CMSs - but it's probably the way to go since most people like them. Thank you for the pointers (I'll look at them) and for the appreciation.

  Posted by: fabriziogiudici on March 29, 2007 at 04:56 PM

• Thanks for the information.

  
  存倉
  辦公室搬運
  搬屋公司
  搬屋

  Posted by: xuefeng8 on August 02, 2007 at 03:18 AM

This work is licensed under a Creative Commons License.