Chet Haase's Blog

Get the [arti]facts: Browser image scaling

I've been playing around with various web and desktop technologies lately, and noticed a funny thing when passing images back and forth between the browser and an applet: IE7 thinks my image is a different size than it actually is.

Here is the scenario: I have an image on my web page (a URL in an <img> tag) that I'd like to hand off to an applet.  I want the applet to be sized appropriately, so I set the applet width/height according to the size of the image.  Then I display the image in the applet ... and the image is smaller than the applet.

Let's try that again.  I have an image that is 100x100, I create an applet of size 100x100, I use a simple drawImage(img, 0, 0, null) call in the applet, and I have a significant border to the right and bottom of the image; the image is not taking up the entire size of the applet, even though I specifically created the applet to be the same size as the image.  Also, the image displayed in the applet is clearly smaller than the image displayed through the <img> tag in HTML.

Then I run it in Firefox and it works the way I thought it should to begin with.

I pounded my head against this for a while, and finally debugged the problem to my "DPI setting" on my Windows XP box.

Apparently, IE7 (and possibly IE6, although I no longer have it available on my system) auto-scales images in the browser according to your current DPI setting.  It considers 96 DPI the default, where one physical pixel on the screen equals one pixel in the image (so an image won't be scaled).  But any other DPI setting will cause auto-scaling.  On my system, a laptop with a relatively dense 1400x1050 screen, the setting is 120 DPI, or 125% of the default.  The net effect is that images will be scaled to 125% of their physical size when displayed in IE7 on my laptop.

Here is a screenshot that show the problem.  Ideally I would use PNG, a nice lossless image format, but IE's support for PNGs is somewhat broken; hopefully this JPEG will do the trick (if none of the images look right, save the image locally and open it up in a different image viewer):

The image I used is a simple black-line grid on a white background, where the black lines and white pixels are spaced very regularly away from each other (always one pixel of separation).  With this simple image, it is easy to see the scaling artifacts that are bugging me.

What I wanted was that the images on the first line would all look good. If I simply load an image with no width/height specified, it should create a space of exactly the right size.  Similarly, if I specify a width/height that matches the size of the image (I tried this with both "100" and "100px" to see if it mattered), then I should get an image area of the right size. 

What I got was an image area that was clearly not the right size; the distortion you see in these top three images shows an irregular grouping of black/white pixels, which shows that the image is not being mapped one-to-one to the onscreen pixels.

To test the theory further, I loaded the images in the second row with width/height specifications of "80" and "80px".  Magically, these images look perfect on my system.  I pre-scaled the images to account for the 125% scaling that I now knew IE7 was going to do.

Now, if we look at what happens on Firefox, we see the reverse (again, if you're not seeing what I'm describing, save the image and view it in a different application):

Firefox shows the first three images all looking good, and the bottom two images distorted. Firefox is clearly not auto-scaling the images, and when I pre-scale the images, it ends up causing distortion for Firefox where it fixed the problem for IE7.

Why Should We Care?

For the most part, images are being viewed in the browser and these kinds of scaling differences are probably irrelevant in that context; you want your photos to look nice, the browser does a decent job of scaling things, and the photos look fine. 

So what's the problem?

The problem is when we need to actually access the pixel information of these images, we're going to need finer control.  And if the browser is hiding that information from us, the browser makes a poor UI engine.

For example, if I need to edit a specific area of pixel data on an image, I'm going to need to find out exactly where those pixels are.  Or if I want to work on individual pixels, I don't want to be working on some pre-scaled version of the image, but rather the raw image data.

But here's a better example, and one that you might have already hit: Google Maps.

I've been using Google Maps off and on since it was introduced; it's so nice to have a dynamic mapping interface instead of the old batch retrieval system of the original mapping sites.  But most of the time, the "Directions" results of Google were just ridiculous.  That is, the directions themselves were correct, but the path drawn on the map was way off.  I don't mean that it had me taking the wrong highway to work, but that it had me driving through the middle of the bay (where there ain't no bridge). 

Maybe Google Maps just had it in for me, but I don't think so.  When I realized what was going on with my images, something clicked in the cobwebs of my memory and I thought I'd see if Google was failing for me for similar reasons.

First, I needed a nice test.  The roads around here (the SF bay area) are way too non-linear; it's hard to do easy comparisons.  So I did a search around Northfield, Minnesota.  If there's one place in the world that knows its cardinal directions, it's the American midwest.  (Aside: My Iowan grandfather once gave directions like so: "If you're coming from the North, then it'll be to the East".  And if we came from the South?).

I ran directions test from highway 280 to Cedar Avenue in Northfield, and ended up with the following map:

You can see my difficulty (I hope).  The blue line should be drawn from the start (the top black blob) to the end (the bottom black blob).  (Ignore the blob graphics; this appears to be IE's continued hate-hate relationship with PNG images).  Instead, the path starts to the left of where it should and ends up far short of where it should, having taken an odd detour through some corn fields. If you look at the actual path it should have drawn, you'll see that it's basically correct, but that the path is drawn at a different scale than the map image.  And if you pop up a screen magnifier with pixel coordinates on it, you can calculate that the path on the map image is exactly 125% of the size of the drawn path. Voila! It's the Google Maps bug that's been dogging me all along.

Just to test this into the ground and then some, I reset my DPI to 75% (instead of 125%), or 72 DPI (instead of 120 DPI).  Again, the path was drawn incorrectly, but this time it overshoots the mapping area and the map path is 75% of the drawn path.

It's interesting (to me, at least), that in all of my tests the path appeared to be correctly placed in the top-left-most position.  So, for example, in the image above the first turn taken to the South (as my grandfather would say, the first South turn if you're coming from the East) is actually in the correct place.  I don't know enough (well, okay: anything) about the Google Maps rendering algorithms, so I don't know why this would be.  But after that one correct spot, it all falls apart and you find yourself off in the weeds.  Or the cornfields. Or the bay.

But what if we don't have the problem on our system?

Granted, not everyone has this problem.  You may have never noticed it before I pointed it out, even.  You would need a combination of a high-density display and a non-default DPI setting (which was the case on my laptop out of the box).  But with the increased availability of high-resolution displays, both for desktop and laptops, the problem will only get worse.  It is expected that resolutions of 200 DPI and up will be commonplace in the next few years.  DPI settings would reflect this change and more people will suffer.

What do we do about it?

Frankly, I'm not sure what we do about it; if the browser is telling us it's doing one thing (sizing things according to pixel dimensions) and then doing another (sizing them according to a scale factor applied to the pixel dimensions), where do we go from there?

In a desktop application, this would not be a hard problem to solve; there are APIs to do everything from querying the DPI settings to checking the size of the image on the physical screen.  But in a browser application, there is no such API (that I'm aware of).  We are reduced to try to divine the information from side effects in the environment.  Like trying to predict the weather by how your knee joint feels. Or trying to see whether a plant is poisonous by eating it.

But it's not clear to me now how to do this reverse-engineering; everything I've seen from IE7 makes it seem like my images are the size I think they are; it is only in measuring the screen pixels that I see the error of their ways.

For example, Dmitri pointed me to this interesting blog on the topic.  One of the comments claims that you can determine the DPI settings by creating a div of a particular size and then asking JavaScript how big it is.  So I tried this, creating a div:

        <DIV id="dpiDetector" style="WIDTH: 1in; HEIGHT: 1in;"/>

and calculating the width/height of the div in JavaScript like this:

    function detectDPI() {
        var IE = (navigator.appName.indexOf("Explorer") != -1);
        if (IE) {
            dpiDetectorElement = document.all.dpiDetector;
        } else {
            dpiDetectorElement = document.getElementById('dpiDetector');
        }
        offsetW = dpiDetectorElement.offsetWidth;
        offsetH = dpiDetectorElement.offsetHeight;
    }

The result?  IE7 reported the offsetW/offsetH as 96x96, and Firefox reported them as 120x120.

Strike out.

Got any other thoughts on finding out this info?  Post it here; I'll try it out.  I'm thinking that there are no obvious, good solutions here yet, or else we wouldn't be seeing the problems in such a widely used application as Google Maps; it's just that people have not yet hit the problem widely.  But they will...

One solution, I feel compelled to point out, would be to use a toolkit more suited to the purpose.  For example, an applet, even if sized incorrectly by the browser, could adjust its size according to the physical size of the image.  In fact, you could even use an applet just to do the DPI detection step (have it detect the size the image it is, the size the browser thinks it is, and then have it adjust the image width/height appropriately).

Comments

• Chet, the point about DPI settings and high res displays is interesting and I wonder how it effects Java programs. Does the Windows DPI setting effect Java2D in any way? It's my understanding that the default transform is defined as 72 user space units equals "approximately" 1 inch. Emphasis on the "approximately", as the default transform (as far as I know) actually just maps one user space unit to one pixel, so on a standard Windows 96 DPI setup, it would actually take 96 user space units to be considered an "inch", atleast to agree with inches as they appear in say, a Microsoft Office ruler. And as of Mustang (thanks to a bug I reported) you can use the Normalizing transform to correct for this. Anyway, point is, does the desktop DPI setting effect how the default transform maps units to pixels? and how do Swing apps look on high res screens? (hopefully nice and sharp, rather than being tiny)

  Posted by: benloud on October 02, 2006 at 08:58 PM

• Very interesting. Actually, I think there is a general warning in css styling that you shouldn't really be using pixels to specify lengths. If you stick to em and ex, the page will look the same all the time as it is relative to the font.

  Posted by: tobega on October 03, 2006 at 02:09 AM

• 
  benloud: I believe our fonts (and therefore most of our GUI stuff) react to the DPI setting by choosing font sizes that reflect the default native font settings. In changing the DPI for the desktop, you affect the font sizes the desktop chooses, and therefore the fonts that Java 2D and Swing use as defaults. Outside of fonts, Swing is pixel-based right now (physical pixels, that is, not these funky browser pixels that I've just butted heads with ). This needs to change in the future and we're looking into this for the next release.

  
  tobega: Interesting point; I haven't used em/ex before, and they seem useful for general web page sizing/styling. However, they don't (unless I'm misunderstanding something) impact the image-sizing problem I ran into here. If I ask the browser to display an image with no size metrics, it will still scale the image automatically in the situation I outlined. And if I provide sizing information, those sizes will still be scaled by the browser. If I'm missing something, please post some sample code and I'll try it out.
  

  Posted by: chet on October 03, 2006 at 07:55 AM

• I tried for for a long time to use 120 dpi, but eventually gave up. Just too much is broken if you use anything other than 96dpi (on Windows). Major web sites (such as the BBC) have made a deliberate decision not to support proper scaling.

  Posted by: mthornton on October 03, 2006 at 09:28 AM

• mthornton: Punting on 120 might be a reasonable workaround for these issues for now, although the stuff on my laptop starts getting pretty darn small at 96 DPI. But I worry about the systems of the future, where we'll have 150 DPI (which I actually had on a monster laptop a couple of years ago) or 200 DPI. At those high resolutions, the 96 DPI solution starts to get unworkable.

  Posted by: chet on October 03, 2006 at 10:06 AM

• Windows (or Mac or Gnome or ...) should be responsible for DPI handling in a way that is transparent to apps. Firefox and IE shouldn't even have to know that it changed, in my opinion. Tell Microsoft to fix it. (I realize that scaling like 130% or whatever can be ugly, but with enough OS or window manager cleverness, it should be doable okay in a majority of cases.)

  Posted by: tompalmer on October 03, 2006 at 04:23 PM

• You can query the screen object in Internet explorer to determine the actual and logical DPI setting the browser is working to.
  See MSDN for more details.

  Posted by: ben_meadowcroft on October 13, 2006 at 09:31 AM

• ben_meadowcroft: Thanks, I hadn't seen this approach before (apparently the folks at Google missed it as well). It seemed to work well in a quick test. I'll try to update the info above with details when I have time...

  Posted by: chet on November 30, 2006 at 09:25 AM

This work is licensed under a Creative Commons License.