Encoding URLs for non-ASCII query params
NOTE: I've updated this blog to avoid "mojibake" -- garbled characters. For some reason, the word TA in TANAKA in the name query key in examples was garbled. Originally all occurences of the word ? were prefixed with the character for TA...for the common family name TANAKA. After removing the mojibake, I think you can still understand the purpose of the blog. But it does make me ask....just what is wrong with displaying TA under this system.
Are you a web service API developer? A good one? Even great? Wherever you are on the greatness spectrum, I have a tip today that is going to make you better. But first, I have something to say. Yes, it is obvious really, but it's worth repeating. The web truly is a world-wide web. Unfortunately, a great number of globally unaware developers are on the global web. This creates an odd situation in which web services are globally accessible but only locally or regionally aware.
There are a few important things to remember when creating a global web service. Let's just cover ONE today: non-ASCII query parameters are valid, useful, and often necessary for a decent, global web service.
It seems so obvious to me, and it probably does to you. Sometimes a service needs to exchange or process non-ASCII data. The world is a big place, and although English is an important part of the global web, more people speak a different language. English is a big percent, but lots of people use Chinese or an Indic language too. Let's make sure your web service can process all those non-ASCII characters in English or any other language!
Let's look at some examples of non-ASCII query params:
In these examples, you must perform two steps to get the query params (both keys and values) into the correct form:
- Convert the keys and their values to UTF-8 if they are not already.
- Perform the "percent encoding" on each UTF-8 code unit
To do #1, you'll need to use whatever character conversion utility you have: iconv, the Java charset encoding converters, whatever.
The #2 step is the important one for this blog. For each hexadecimal code unit in the UTF-8 query portion, you must "percent encode" the code unit. Let's look at the first example query params:
encodeURI actually does a good job of doing this for us:
encodeURI("name=?&city=??")produces the string:
Notice that you should also include this encoding for the keys in the param list. In the next example, I've used Japanese values for both keys and values.
encodeURI("??=?&?=??") produces this string:
Note that both the keys and vaues have been "percent encoded".
On the server side, your server will understand how to decode these values into their correct UTF-8 string values if you have configured it correctly. Correct configuration of a server usually involves a charset conversion filter for a servlet container and sometimes just a config setting for Apache.
More on this at a later time.