Kirill Grouchnikov's Blog

Java on the desktop - trail behind or lead forward?

Subpixel rendering has been around since 1988. Microsoft stepped on the scene circa 1998, applying for the patent on ClearType in 1999 and getting it in 2001. This technology has been integrated in Windows XP when LCD displays (both desktop and laptop) have been far outnumbered by CRT displays. However, when the time came, XP was ready. What about Java? The first implementation was available only in June 2005, with the official release a few months ago (good 8 years after it has been introduced by Microsoft).

What's the next big thing in desktop typography and general layout? High resolution displays, which also goes by the acronym DPI (dots per inch). Believe it or not, but the display makers don't stand still. While the 96 DPI holds the market majority now (like CRT monitors did in late 90s), the 120 DPI is just around the mass market corner. Is Swing ready? Unfortunately, the answer is "mostly no".

At Desktop Matters, Chet has held a mini-survey on possible additions to Swing in the next Java version. There were about 50 people in the room, and about 40 topics to vote on (one could vote on each one of them). If i remember correctly, the support for high-DPI displays raised about 10 hands, and when OS X was removed from the question, there were only 2 hands left (myself included). Even more interesting was Chet's phrasing of the question - it went something like "In a hundred years, when everybody has 9000 DPI monitors, how important is it for Java to support this?" Well, it won't be in a hundred years, it's already here and will only get bigger, and without immediate action Java once again will be left behind (providing it in another eight years circa Java SE 10).

Karsten has pioneered the support for high-DPI desktop settings in the Looks look and feel last year. He provides a layer that queries the desktop settings and tries to match the font family and the font size so that a Swing application running under this LAF doesn't make the user squint (if the texts are too small) or grunt (if the texts use inconsistent font). This is not all - as he mentions on the project dev mailing list, you have to scale everything else, including borders, separators, insets, margins etc.

In the next version of Substance i have decided to reuse Karsten's code (thanks for choosing a liberal BSD license :) and to take it one step forward - scale the entire UI. Currently, Looks doesn't scale the other UI elements, such as checkboxes, radio buttons, scroll arrows etc (see screenshots below). This may result in inconsistent user experience, especially for DPI settings higher than 120 (such as 144 or even 192). Here is the complete list of UI elements that is scaled with the DPI in the next Substance release (release candidate scheduled on April 2nd with the release scheduled on April 16th):

• Check marks of check boxes and radio buttons (regular and as menu items), dimension and stroke width
• Arrow icons of combo boxes, scroll bars, split pane dividers, submenus, tabbed panes (dimension and stroke width)
• Title pane icons (for decorated mode) of frames, dialogs, internal frames and desktop icons + proper centering of the title pane buttons in Y direction
• Scroll bar width
• Slider icon and slider track size
• Tab close buttons (dimension and stroke width)
• Spinner button width
• Password field mark size (diameter and gap)
• Progress bar width / height
• Double arrow icon stroke size
• Minimum width and height of buttons with text
• Insets of combo boxes, text field and spinners
• Stroke width of focus ring

Now for the screenshots. Here is a screenshot of an internal frame under 96 DPI (normal settings):

Here is the same internal frame under 120 DPI (125% normal size). Note how all the UI elements (fonts, scroll icons, title pane icons, button size, check marks and radio buttons, combobox insets) scale:

Here is the same internal frame under 144 DPI (150% normal size):

For comparison, here are the screenshots of the same internal frame under Looks Plastic XP. Under 96 DPI:

Here is the same internal frame under 120 DPI. Note that the scroll bar remains with the same width, the title pane buttons (inherited from Metal) remain the same size and uncentered and the combobox button doesn't retain its proportions (and the arrow icon itself has the same size):

Here is the same internal frame under 144 DPI (with the same visual inconsistencies):

What about other core and third party look and feels? As far as i know, the other third-party look and feels do not respect the DPI settings (and i'd be very happy to learn otherwise). The default (for now) Metal / Ocean doesn't scale at all, but the Windows look and feel shows some promise. How does it do so? Well, since it uses the native APIs to paint the controls, it also respects the dimensions computed by these APIs. In most cases (see the table below), Windows look and feel does indeed match the native dimensions of the controls and UI elements. Here is the same internal frame under Windows LAF in 96 DPI:

Here is the same internal frame under 120 DPI. Pretty much everything scaled, including fonts, checkbox and radio button marks, scroll bar, tabs and combo:

Here is the same internal frame under 144 DPI. Now, while fonts, scroll bar and tabs scaled, the checkbox / radio button icons and arrow icons on scroll bar / combobox remain the same size:

The reason for the later is simple - the native implementation is still not perfect. Here is the table to illustrate this:

The first three columns show size of different UI elements under Vista in 96, 120 and 144 DPI. The last two columns show size of the same UI elements under Windows look and feel in Java 6.0. As you can see, while the native elements scale well from 96 to 120 (with the exception of combobox arrow size), the transition from 120 to 144 is not so smooth. The checkbox / radio button checkmarks have the same size, and the arrows on scrollbars, spinners and comboboxes are the same.

What about Windows look and feel? While it picks most of the native settings, it still has some issues. Look at the spinners, tab heights, progress bar height (not picked at all), slider thumb, menu arrow and combobox. There's still work to do here.

Which brings me to my main point. The current implementation of high DPI support in Vista (and in XP) is not perfect, especially for 144+ range. In addition, changing DPI setting requires OS restart (and once Explorer crashed under 144 DPI). Java must seize this opportunity now and provide the best implementation for the mass-market. Java shouldn't wait until the market demands this support (especially with releases coming out every 18 months and with the relatively slow adoption rate of the latest releases in production environments on the large scale). Be ahead of the curve, don't lag behind.

Comments

• Hi Kirill... Support for HiDPI displays in Swing is something that has been on our radar for a couple years now. We didn't address the issue in JDK 6 mainly because a) there were hardly any HiDPI desktop displays on the market (unless you count the $7000 ones, and even today there are none aimed at the desktop market, although certainly this will change in the next couple years), b) we did start to see HiDPI laptop displays (> 120 dpi), but not enough yet (and not at a high enough density to be truly problematic for usability, although certainly this is changing too) to truly justify the resource investment.

  I too was surprised at the small show of hands at Desktop Matters. I suspect that this is one of those areas that most developers don't realize will have a big impact on their user interfaces in the near future (kind of like software developers in 1987 couldn't care less about Y2K). Part of it is again the lack of actual displays on the market. For the developers who have heard about it, it's been this theoretical thing looming in the distant future. At WWDC 2006, Apple was predicting the first HiDPI displays to hit the consumer market sometime in 2008. Apple has been doing a pretty good job of educating developers about the impact of HiDPI displays, and are continuing to provide nice software support (more so in Leopard) for developers who want to be prepared for the future.

  BTW, you should know by now that Chet likes to be humorous. His comments that you quoted are not a serious reflection of our interest in this topic. HiDPI support in Swing is on our list for JDK 7. It is not clear yet what form this will take. We will try our best to provide an auto-scaling mode so that existing applications will look decent enough on HiDPI displays without modification, although developers need to be aware that some application changes (especially for apps with custom components/painting) are almost inevitable to make sure everything looks right. Given that most of our APIs in Swing and AWT are integer based, it will be difficult to provide an ideal solution, but what we can provide will likely be good enough to avoid pixel cracks and other artifacts assoicated with HiDPI. (If only AWT was designed from the start with floats, as Cocoa was! Well, at least Java 2D is in good shape.)

  Also, I've been experimenting with some multi-res image/icon classes so that developers/designers can provide images with multiple levels of detail (possibly vector artwork) so that icons look good at any resolution. Finally, it's important that we make our standard L&Fs look good in a HiDPI environment. We're trying to make Nimbus completely scalable from the get go. Our Windows L&F will require some tweaking, as you've already discovered, but it's a solvable problem. The GTK L&F is another story, however, as I'm not sure anyone in the GTK community is really looking into making it scale on HiDPI screens (and most existing engines/themes aren't really prepared for it, so it may be a long battle on that side). The Metal/Ocean L&Fs cut a lot of corners (this was back in 1997 when they thought it was safe to assume identity transforms, etc), so it would be a chunk of work to make them HiDPI ready. I guess it will depend on what we do with Nimbus to know how much work to invest in bringing Metal/Ocean up to speed.

  Anyway, I'm glad to see that you are interested in this topic and are educating other developers about what's on the horizon. Also, it's really nice to see that your painting code in Substance is scalable; the screenshots look good. I'll try to blog more about HiDPI once we have a better idea of what we can accomplish on our side in the JDK 7 timeframe (it's always tough to plan these things given all the other projects we have on our plate, but we should know more soon).

  Thanks, Chris.

  P.S. Geez, this was a long comment. Sorry about that.

  Posted by: campbell on March 17, 2007 at 04:26 PM

• Chris, many thanks for the lengthy reply. I know this about Chet, but his comment might be perceived as dismissive, and as such might make some developers think that this issue is not worth worrying about. Looking forward to seeing more information on this subject in JDK 7.0.

  Posted by: kirillcool on March 17, 2007 at 04:42 PM

• Chris, a thought occurred to me. Since now Windows LAF uses the native APIs, it's kind of limited by them as well. Say, Vista doesn't scale as well from 120 to 144. Would you decide to make the fixes yourself (like you do in JOGL for drivers bugs) or hold on to Microsoft to fix it?

  Posted by: kirillcool on March 17, 2007 at 08:12 PM

• From what I've seen, HiDPI support in Vista is a work in progress. If we run across problems, we'll definitely try to work with Microsoft to fix it at the source, if at all possible (similar to how we work with Nvidia and ATI on driver fixes, and only apply workarounds when all else fails). We'll see how it goes. Thanks, Chris.
  

  Posted by: campbell on March 17, 2007 at 09:52 PM

• It seems to me the major concern, as chris mentioned, is integers for co-ordinates and sizes. I cant see how that problem is going to be solved.. =(

  Posted by: benloud on March 18, 2007 at 03:44 PM

• My current top of the range 21,3" NEC LCD set me back $2500 when I bought it. This christmas we bought a 20" Samsung LCD widescreen. It cost $500.
  Big ass displays are here. Now.
  Also notice Apple TV, Xbox 360 HD and OS X Leopard - Computer + TV.
  

  Posted by: janerik on March 18, 2007 at 10:25 PM

• Janerik, "Big ass displays" != hiDPI

  One problem that also needs to be solved is the LayoutManager issue. If the LM is good enough we can get around the problem of integers at the component level. MiG layout will take over the responsibility of minimum/preferred/maximum size for the components (if desired) and can use cm, mm, inch, dots, pixels, percent or DPI compendated pixels (and more). It's everything you need. The fact that is is also a replacement for every single LM in the JDK (including most parts of GroupLayout) is another advantage. The fact that you can override the DPI when the OS reports the wrong thing is also good I guess.

  Of course I'm biased, since I created it, but if you are serious about resolution independence and solving the layout issue at the same time I'd suggest you take a look at it. It's even implemented in two IDEs as we speak.

  Cheers,

  Posted by: mikaelgrev on March 19, 2007 at 05:07 AM

• 
  Kirill: As Chris pointed out, my phrasing of the question was more in jest than reality. I believe the people in the room understood that, although taken out of context as a quote in your blog that's not as obvious. Clearly, Hi DPI will be very important at some stage, and we should be ready to meet it when it's there.

  
  I think the interest in the feature was more indicative of the type of feature that this represents; it's a problem that's not yet apparent, so people are not chomping at the bit to fix it. Yet. Meanwhile, there are other glaring issues or features that people can see in their daily development lives, so obviously they'd like those addressed first before this other theoretical one. If everyone in the room had hi-dpi displays, I think the response would have been different.

  
  I had a laptop (actually, it was more like a mainframe in a laptop casing; I finally got rid of it because it just wasn't portable) that had about 150 DPI. And that was over 3 years ago; displays have progressed since then. While apps on this display (Swing apps included) were quite usable, it was easy to see that increasing pixel density would impact the useability.

  
  Display technology is not progressing as rapidly as some predicted, but it's obviously coming along, and there will be affordable Hi DPI displays in a reasonably near future. I'd like to make sure that we do what's necessary to look good on these displays as well.

  Posted by: chet on March 19, 2007 at 07:10 AM

• Most people (guessing) who care about DPI bought a large LCD monitor and had a hard time reading the text. Then went into Windows XP settings and upped the DPI to the next number on the list.

  I am not spending $7000 on something that needs a $1000 BluRay player, HDMI (I think I have seven types of video cables now), $50 movies and a butler just to show a movie.

  I am talking about Joe Sixpack, not Bill Gates :)
  

  Posted by: janerik on March 19, 2007 at 08:18 AM

• Chet - apologies for the misunderstanding and thumbs up for looking into this issue for the Java 7 timeframe.

  Posted by: kirillcool on March 19, 2007 at 09:26 AM

• 
  Hi.

  What can be done by the community to help this along?

  Ciao; thanks.
  Steev Coco.

  Posted by: steevcoco on March 19, 2007 at 09:29 AM

• Microsoft has an article called "How to write High-DPI Applications". I was written in March 2001.

  As I understand it Vista and Leopard are the first operating systems that (supposedly) support high DPI out of the box. XP and Tiger only has partial support.
  

  Posted by: janerik on March 19, 2007 at 10:25 AM

• Just for clarity as some seem to think that HiDPI means big screens. Big size and high DPI are in fact not related.

  Higher DPI means that X pixels takes up less inches (or centimeters for that matter) on the screen and thus a 800x600 dialog box will get physically smaller (as measured with a ruler) when shown. HiDPI is common in smaller screens, like mobile phones and laptop screens (one of ours has 1920x1200 on 15 inches, which is a pretty high DPI). DPI is thus the physical size of the pixel.

  I know of no high DPI large screens other than the IBM made 22 inch beast sporting 3840x2400 pixels. Costs about 12.000$ and is made for medical imaging applications.

  Large screens have another, though similar, problem which is maybe more pressing here and now. Dialogs are today made in a certain number of pixels. People then buy large screens (with the same DPI as their former screens!) and but get dissapointed when the dialogs aren't using the real estate available.

  Take for instance the update center dialog in netBeans 5.5. On my 30 inch Apple screen it still only shows about 7 updates in a "small" dialog. I always have to enlarge it to get a good view and nice usability. It would be better if the size would be like 30% of the screen width or something. (Did I mention what layout manager that exists today and is BSD that can do this kind of thing? ;)

  The layout manager together with DPI-proportional fonts and icons, is the way to salvation here. Not only one of them.

  Cheers,

  Posted by: mikaelgrev on March 19, 2007 at 10:41 AM

• (humor) A high DPI value in hardware means everything is smaller. A high DPI value in software means everything is bigger ;)

  Posted by: janerik on March 19, 2007 at 01:01 PM

• Anyone thinking about refactoring Swing?

  I think lack of support for a real number based coordinate system in Swing is a major issue. As Chris had pointed out, on the other hand, Java2D is in good shape.

  Since Java is going open source. I think, it wouldn't be too hard, to refactor Swing using an IDE such as NetBeans or Eclipse. Maybe one could throw in support for generics as well, plus do some smaller and bigger tweaks here and there.

  Alas, such a revved up Swing would break backwards compatibiity, and wouldn't be much fun without GUI builder support. It still would be an interesting project.

  
  Cheers,
  Werner

  Posted by: wrandelshofer on March 19, 2007 at 01:09 PM

• > What's the next big thing in desktop typography and general layout?
  > High resolution displays, which also goes by the acronym DPI (dots per inch).

  The next big thing? I am sorry, but this was "the thing" from the day one. If I remember correctly, Charles Petzold's book has a special chapter dedicated to physical and logical resolutions, as well as to aspect ratios (not screen aspect ratios but pixel aspect ratios, because - surprise! - pixels don't have to be square). Every self-respecting programmer has been developing proper scalable UI from the early days of Windows (I cannot say about other systems). Do dialog units ring a bell? Delphi makes building scalable forms a quite simple task. It is true though, that many apps were and still are horrible, without any attention being paid to scaling at all, and their vendors were/are able to get away with them, because regular users change logical resolution very rarely.

  Calling HiDPI a "new big thing" is like calling police sweeps for street drug dealing a "new big thing" instead of saying bluntly that the cops just started to do what they were expected to do (this was an abstract example of some abstract city). Someone even created an acronym for something that always was part of the job, as if it is a new technology. Right, when the name is established, vendors can charge for it.

  > In the next version of Substance i have decided to reuse Karsten's code
  > (thanks for choosing a liberal BSD license :) and to take it one step
  > forward - scale the entire UI.

  Again, for normal Windows apps this is not a step forward, this is how the apps should be developed. It is a pity that Swing was so much behind, that bringing it to normal user expectations is considered a step forward. From time to time I can hear preaching "Why there are so few Swing apps? How Swing is worse than WinForms?" Well, here you are.

  I hope that my rant will be understood properly: I totally support what Kirill and other Swing masters are doing, this work is long overdue, but calling it a "next big thing" and a "step forward" is possible only in relation to Swing. Normal native apps have the scaling feature for at least 15 years. The fact that some Microsoft's own apps do not respect screen's logical resolution does not make things prettier for Swing.

  Posted by campbell
  > there were hardly any HiDPI desktop displays on the market

  Come on, I switched to 120dpi in my Windows95 system after I upgraded from VGA to SVGA. It was... er... well, quite long ago. I am not claiming that I used Win3.1 in 120dpi, I just don't remember. I don't think that I had a monitor large enough.

  Posted by campbell
  > HiDPI support in Swing is on our list for JDK 7

  What about building UI from resources? If not old-fashioned resource files, then maybe something like XAML? Creating a Swing UI from scratch in the code is a pain! I prefer not to develop in Swing at all, and this is despite my experience in building Windows apps with plain WinAPI. For great UI ideas take a look at Delphi.

  Posted by mikaelgrev
  > Dialogs are today made in a certain number of pixels.

  I wonder have Swing designers programmed for any other GUI, or they all were university professors with neat ideas of object-oriented interface? I can only repeat: dialog units, this is what Windows have been using for a long time with great success.

  Posted by: michael_jouravlev on March 19, 2007 at 04:45 PM

• I can only repeat: dialog units, this is what Windows have been using for a long time with great success.

  Oh reeeally..

  Is that why most Windows applications (except may be for Avalon's ) are screwed up on high dpi desktops?

  Just setting "large fonts" screws most of them up..

  "Dialog points" can only get you so far. How about artwork?

  Dmitri
  

  Posted by: trembovetski on March 21, 2007 at 12:59 PM

• Forgive me for being thick - but if you provided real number methods to developers (and keep the integer methods as shells that then call the real number versions) then you would have backwards comptability and and improved version for those that wanted to refactor their code. Or am I missing something?

  Posted by: bellbux on March 31, 2007 at 06:33 AM

• Definitely needed. My new laptop has roughly 130 dpi, and with my aging eyes, it's definitely a strain to use certain Java applications. There's a huge difference (to me) between the previous laptop's ~96 dpi and the new one; I may need to get a new set of bifocals to run Netbeans....

  Posted by: meltsner on April 30, 2007 at 01:23 PM

This work is licensed under a Creative Commons License.