joconner's Blog

Changes in Java 7 Locale

I admit that I haven't been particularly active in the i18n or Java standards bodies in the last few years. Pardon me, but I had a wild rumpus in a startup for almost 3 years, then joined Yahoo for a couple years, and BAM! ... during that time, the Unicode Consortium added emoji, and the Oracle Java folks overhauled the once relatively simple java.util.Locale class among other things. That'll teach me to turn my back on those folks.

You can almost forgive the Unicode people for their addition of emoji... BUT more interesting to me right now...have you seen java.util.Locale lately? What the heck happened there? Java 7 introduces Locale.Builder, Locale.Category, and a Locale.forLanguageTag method. With Java 7's support of BCP 47 (the best practices for language coding), the Locale got a beefy new Builder inner class to help with...well, to help with building BCP 47 compliant locales. More on this in a different post. With the BCP 47 support, I suppose you might need the forLanguageTag method too. Now here's what stumps me though -- that Category class!

As far as I can tell, this category is to help you differentiate a locale instance for two different purposes: user interface language and locale-sensitive data formats. The new categories are the following:

• DISPLAY
• FORMAT

Do we really need a separate Category class to make that explicit?

Locale has another new method: setDefault(Locale.Category, Locale newLocale). I can't wait to play with this a bit more. I suspect that this will set the default resource bundles (DISPLAY) for UI language and the default FORMAT for things like DateFormat, etc. However, I can't understand why these two categories are actually needed in the platform libraries. Convenience maybe? You could always use two different Locale instances for this in the past, but I suppose this makes that easier? I'll have to play around with these to find out more.

You can see this blog and followups on my own joconner.com site.

Encoding URLs for non-ASCII query params

NOTE: I've updated this blog to avoid "mojibake" -- garbled characters. For some reason, the word TA in TANAKA in the name query key in examples was garbled. Originally all occurences of the word 中 were prefixed with the character for TA...for the common family name TANAKA. After removing the mojibake, I think you can still understand the purpose of the blog. But it does make me ask....just what is wrong with displaying TA under this system.

Are you a web service API developer? A good one? Even great? Wherever you are on the greatness spectrum, I have a tip today that is going to make you better. But first, I have something to say. Yes, it is obvious really, but it's worth repeating. The web truly is a world-wide web. Unfortunately, a great number of globally unaware developers are on the global web. This creates an odd situation in which web services are globally accessible but only locally or regionally aware.

There are a few important things to remember when creating a global web service. Let's just cover ONE today: non-ASCII query parameters are valid, useful, and often necessary for a decent, global web service.

It seems so obvious to me, and it probably does to you. Sometimes a service needs to exchange or process non-ASCII data. The world is a big place, and although English is an important part of the global web, more people speak a different language. English is a big percent, but lots of people use Chinese or an Indic language too. Let's make sure your web service can process all those non-ASCII characters in English or any other language!

Let's look at some examples of non-ASCII query params:

• http://example.com?name=中&city=東京
• http://example.com?名前=中&市=東京

In these examples, you must perform two steps to get the query params (both keys and values) into the correct form:

1. Convert the keys and their values to UTF-8 if they are not already.
2. Perform the "percent encoding" on each UTF-8 code unit

To do #1, you'll need to use whatever character conversion utility you have: iconv, the Java charset encoding converters, whatever.

The #2 step is the important one for this blog. For each hexadecimal code unit in the UTF-8 query portion, you must "percent encode" the code unit. Let's look at the first example query params:

name=中＆city=東京

The JavaScript function encodeURI actually does a good job of doing this for us:

encodeURI("name=中&city=東京") produces the string:

name=%E4%B8%AD&city=%E6%9D%B1%E4%BA%AC

Notice that you should also include this encoding for the keys in the param list. In the next example, I've used Japanese values for both keys and values.

encodeURI("名前=中&市=東京") produces this string:

%E5%90%8D%E5%89%8D=%E4%B8%AD&%E5%B8%82=%E6%9D%B1%E4%BA%AC"

Note that both the keys and vaues have been "percent encoded".

On the server side, your server will understand how to decode these values into their correct UTF-8 string values if you have configured it correctly. Correct configuration of a server usually involves a charset conversion filter for a servlet container and sometimes just a config setting for Apache.

More on this at a later time.

Writing CSV files as UTF-8 for Excel

Yesterday a coworker complained that Excel wasn't displaying a CSV (comma separated values) file correctly. Our application allows the user to send a report via email. The application provides the report as a CSV file. Because the report can contain multilingual text, we've decided to encode it in UTF-8. Unfortunately, when users click on the file to display it, usually in Excel, all of the multi-byte UTF-8 characters display incorrectly.

The problem was immediately clear to me...Excel was opening the UTF-8 encoded files, but it was incorrectly identifying them as Latin-1 encoded files. In the absence of any charset identification, Excel must guess about a file's content encoding. In our environment, many host PCs use en_US locales with Latin-1 as the typical charset. Excel uses that default to read and display CSV files.

My solution to the problem was to use the byte-order marker (BOM) to identify the CSV file as a Unicode file. I instructed my colleague to prepend the FEFF character to the file. The Java application that writes the file uses a FileWriter that encodes to UTF-8 to create the CSV file. It was simple to just output the BOM as the first character in the file.

Now when our customers double-click on these files, Excel opens the file, notices the BOM, and automatically selects UTF-8 as the file's charset encoding. Now Excel displays the previously mangled characters correctly. And I was able to help resolve a problem with an easy solution.

Maybe you can give your applications a hint about plain text files as well. Writing the BOM to your file can help Unicode-enabled applications know how to decode your Unicode files.
 

Speculations regarding Google Chrome OS

Today's announcement of Google's Chrome OS is exciting in a few ways. I think it has implications for Java developers. With hindsight, I now think that Larry Ellison was hinting about Google's Chrome OS when he expressed some of his desires for JavaFX on small netbook-like devices.

So, without any real knowledge and armed with nothing more than a vivid imagination, I provide some of my predictions/speculations for the upcoming Google Chrome OS and the devices it will power:

1. Google Chrome OS will be a slightly more beefy Android OS. More beefy because it will have additional hardware driver support you might find in a netbook. However, its essence will be Android OS.
2. The Chrome browser (or a slimmed down cousin) will be the primary application on that OS. It's already integrated into Android via Webkit
3. The developer API will be very similar to what Android G1 developers already use. Android G1 apps are essentially Java apps written to a Java-like API. Same Java language on top of the most important, core packages of the Java SE platform. And, of course, Google won't be able to call it a "Java" platform because it will be stripped down to what Google engineers consider only the core, "good" parts of Java SE APIs + Google's own Android APIs of course.
4. Google Chrome OS will be attractive to Java engineers because it looks and feels so much like the the JVM...except it's really the Dalvik VM. Many simple applications that run on Java SE will be able to run on the Dalvik VM after a recompile. Or maybe you'll just have to run your class files through a simple converter to target the Dalvik VM. At any rate, Java developers will feel right at home.
5. Google Chrome OS devices will need to get onto the network easily, seamlessly, regardless of Wi-Fi availability. Google really does believe that "the network is the computer". Without the internet, these devices will be severely hampered. Expect these devices to have multiple network access technologies built in. Wifi hardware will obviously be on board. But you can imagine it also having a cellular transmitter/receiver built-in too.
6. Remember all that cellular radio spectrum that Google was interested in only one or two years back? Wouldn't it be just an awesome thing if Google purchased a huge portion of that and used it to make their Google Chrome OS devices be able to instantly jump onto that for network access? You buy the device, punch in a pre-purchased code for access, and your notebook is on the net in 5 minutes! It will be incredibly, insanely easy to get on the network with your Google Chrome OS-powered device.
7. Hey, what's that Google Voice project anyway. Only one of the coolest telephony projects around! Maybe Google will leverage this service? Here's a scenario for you: you buy a Google Chrome OS device, open it up, agree to the terms of a Google voice membership, get a Google voice number and Google account (if you don't already have one), and the device then connects to the network using the built-in cellular hardware to connect to some of that cellular spectrum that Google will or has already purchased.
8. After all of this, or perhaps even before this, we all start to feel a little uneasy about just how pervasive Google really is. And despite Google's mistrust and derision of Microsoft, they begin to look a little bit like Microsoft too...really, really big and really, really powerful and located at every digital turn. But this time, instead of controlling your PC, they control your network. Ooh, there's a suspenseful novel in there somewhere.

Ok, some of that's just silly, crazy talk...or is it? We'll see over the next few months.

Oh, one last thing. I just cannot resist the urge to compare Google Chrome OS to Sun's Java OS. Do you remember that? I could hardly find any references to it, although I did find an old article called Inside the IBM JavaOS Project. At some point, Sun apparently enslisted IBM to help. At any rate, the Java OS project started (and ended) a long, long time ago. It's been a decade at least. Remember the Hot Java browser? I actually ran it and used it. I remember that one of our tests at Sun was to run the SwingSet demo on it. But now I'm just distracted. What was I saying? Oh yes, there are even more similarities. Java OS is to Google Chrome OS as the Hot Java browser is to the Chrome browser. Maybe Google Chrome OS will finally be the successful reincarnation of JavaOS?

It's all fun to think about, and as I suggested, pure speculation at this point.