Anyways, different languages have different character sets for names and text, so EQII uses a combination of UTF-8 encoding and Unicode to represent all text (names, places, quests, player communication, etc). However, a common signaling protocol used by voice chat applications (including Vivox's that we're using for EQII) is a case-insensitive, ASCII-only protocol known as SIP. This leaves us with a dilemma: how do we represent Unicode player names and channel names in the SIP protocol?
Originally we were going to use Base64 encoding, but this has a variety of problems for our application:
- It requires case sensitivity, but SIP is case insensitive
- Encoded text is not human readable
- It increases the size of the data
I've enjoyed discovering this neat encoding algorithm (and it's been a lifesaver for our Voice Chat implementation). If you need to efficiently represent Unicode in ASCII, Punycode might be the way to go.