Good-Bye T-Mobile (and Good Riddance)

I just received a $446 bill from T-Mobile. To say that I'm disenchanted with T-Mobile would be an understatement.

I suppose I should rewind a little bit. I was a T-Mobile customer for over five years. Back in 2003 I signed up with T-Mobile because I was moving to Austin, TX to work for the now-defunct Origin Systems and they had the best plan for me at the time. My wife and I mostly use our cell phones to call each other, so the cheapest no-frills family plan usually suffices for me.

Fast forward to 11/27/2006. I was sick of using the bottom-of-the-barrel free no-camera monochrome Nokia phone that I got for free when signing up in 2003. My original two-year contract had long since expired. Verizon was offering the inexplicably popular RAZR phone for some ridiculously low price. However, having been a loyal T-Mobile customer for over three years, I decided that I'd give them the chance to beat Verizon's deal and compete for my business. The "Customer Loyalty" representative that I talked to jumped at the opportunity. Not only did they give me better RAZRs than Verizon was offering at the same price (and the same two-year committment), he also changed my phone numbers and gave me some text messages for free, waived all of the upgrade fees AND put me on an "unlisted" family plan that was slightly cheaper. Wow! That's customer service!

And then followed four months of hell.

They sent me the wrong phones. They charged me full price. They wouldn't reimburse me to ship them back. Every representative that I talked to showed a new previously-unheard-of level of incompetence and screwed something else up, making it all the more difficult to explain my plight to the next representative. Untold hours on the phone forever lost arguing with people, explaining my situation and fighting to get my bill fixed and figured out. Pages and pages of notes with a myriad of representatives' and supervisors' names weaving a trail of regret and resentment. For four months.

Had it not been for that damned two-year contract I would've flushed them right then and there, but eventually the dust settled and things were worked out.

Two years later, my contract was nearing its end, though I didn't know the exact date it was up. I once again figured that I'd give T-Mobile every opportunity to keep my business. I went into their stores looking at various Blackberrys and the G1. I was interested in a phone that would give me access to my corporate email and the Internet. I was asking for a little bit better price on the phone and the plan. They wouldn't budge. I called "Customer Loyalty" again and they were resilient against my requests. I told them that I really wanted the iPhone and was willing to go to AT&T unless T-Mobile would give me a better deal but instead got stuck talking to some blatherskite who insisted on hammering me on the irrelevant topic that the iPhone was available through T-Mobile--in Germany. I feel that I gave T-Mobile every opportunity to keep my business. Suffice to say, now I'm with AT&T and proud to walk around with my Apple iPhone (I have an iPod Touch too, but adding a phone to it is just brilliant).

Now, about this $446 bill. You'll notice that I upgraded (and started a new two-year contract) on 11/27/2006. I switched to AT&T on 11/23/2008. Four days away from two years and T-Mobile slaps me with two $200 early terminations fees--one for each of my two lines. Four days! At the very least paying the whole amount for my last month would fulfill my obligation to the two-year contract.

Needless to say, I immediately got T-Mobile customer service on the phone. Despite the lividity coursing through my veins I managed to remain level-headed. The "Customer Loyalty" representative offered to cut the $200 per-line fee down to $50 per-line. "That is unacceptable," was my reply. Her next response was cut short as I quipped, "I'll need to speak to your supervisor." The merry Christmas tunes that filled my ear while I was waiting filled me with a sense of irony. Finally the woman came back and told me that her supervisor was busy on another call but she could finagle it so that I'd just finish paying for the full final month, dropping $384 from my bill and returning sanity to the evening. Another 30 minutes of my life wasted on the phone with T-Mobile to combat utter, inane stupidity.

To T-Mobile, if you happen to be listening:

1. Either allow "customer loyalty" more authority to please customers (especially on plan prices) or lose them.
   
2. Pro-rate the early termination fee or at the very least, change your system so customers who are within the billing cycle of their end-of-contract just pay out the rest of their month to fulfill their obligations.
3. Recognize the Charlie Foxtrot cases and assign a single person be your customer's advocate and to sort the problem out. Dealing with one competent person who understood my problem after upgrading in 2006 would've been very nice, but alas, it was not so.
   
4. You can do better than the G1. It doesn't even natively support Microsoft Exchange email.
   

To everyone else, I'd have to recommend against T-Mobile. By all means, use a different cell phone provider. If you're with T-Mobile now, get out as soon as you can. If you're not big on data, you can probably save a bundle by buying individual minutes and not having a plan.

Memory-Mapping Files for Fun and Profit

It's amazing that I still meet programmers who don't understand the benefits of memory-mapping files. If you're unaware of what that means, let me explain it a little bit. Traditional reading from a file involves several system calls: opening the file, seeking to the desired position and then reading or writing to the file. System calls can be slow as the operating system may have to switch to kernel mode and call into drivers to perform the IO operations. Memory-mapping a file usually has just one call, and then you are able to access the file as if the entire file had been read into a contiguous block of memory. But here is the greatest benefit (at least to me as a MMO server programmer): all processes accessing a file can use the same virtual memory.

Take the EverQuest II server for instance. The (binary) data file that describes nearly everything except collision geometry runs about 460MB. Every zone server uses that same file and we run several zone servers on one physical machine. With memory-mapping, the OS uses 460MB of virtual memory (assuming the entire file is used) just once and all zone server instances can use that same virtual memory without having to read the file into each individual instance.

Like anything, there are drawbacks. Most modern operating systems use demand paging, so if you try and access a page of memory representing a part of the file that hasn't loaded yet, a page fault will be triggered and the OS will have to perform kernel-mode IO to load that page from the file. Depending on how much virtual memory your application has access to (and how spread out your accesses are), this could potentially trigger a very high number of page faults.

Another drawback is address space. A 32-bit application will only have access to less than 4GB of address space (Windows has a 2GB address space by default). If you map entire large files into memory, you are consuming this address space. This has become more and more important on the EQII servers. As I mentioned, our data file is 460MB. Since shortly after the game launched, this file has always been fully mapped into memory. That was fine back then as it was much smaller than it is today. Now, after launching our fifth expansion, the data file is now nearly one quarter of our allotted address space (and not getting any smaller).

Fortunately, the APIs for Linux (mmap) and Windows (MapViewOfFile) allow an offset and a length to be specified. This opens the door to partial memory-mapped files. However, there are some caveats. The offset cannot be just any offset; it must be aligned. On Linux the offset must be a multiple of the page size (found by getpagesize), but on Windows the offset must be a multiple of the "allocation granularity" (found by calling GetSystemInfo). The length of the mapping usually does not need to be aligned, but if you're planning a generic partial mapping solution, it's probably best to use the same alignment. Another thing to consider is that mapping parts of files as you need them will spread them all over your address space; you cannot assume that page 1 and page 2 of a file will be placed next to each other in memory if you map them separately.

Here are some of the design goals for my partial memory-mapping implementation:

• Phase sections out as they are no longer used (start-up data is only needed at start-up)
  
• Reduce thrashing (phasing out a section and then mapping it back in)
• Similar performance to the current full-file implementation
  
• Reduce memory usage!
• Ability to permanently map/unmap sections (i.e. sections that don't phase out)
• Ability to automatically re-size sections on demand
  

Allow me to touch a bit on that last point. For my interface, given an offset and a length, I wanted to return a pointer to contiguous data rather than copy data into a provided buffer. Say some previous access forced you to load page 10 (but not page 11) and now you want to read some data that spans pages 10 and 11. This presents a problem for the contiguous data interface. I can't just map in page 11 and hope that it ends up next to page 10 (I will point out that there are 'hints' that can be passed to the OS APIs to attempt address space positioning, but these are discouraged). Do you leave page 10 mapped and create a new mapping that combines pages 10 and 11? Since my interface always required an offset and a length for every access, I could safely unmap the existing page 10 and map pages 10 and 11 together. Data requests for data on page 10 would now refer to the joint mapping. However, this did greatly expand the number of test cases I needed and complicated the logic.

This nifty little piece of technology is currently running on our live game servers and doing very well. Instead of 460MB, we now keep a pool of about 20-30MB mapped from the file and generally enjoy a 0.1% or lower miss rate. Best of all, we still have all the benefits of memory-mapped files.

(Country) Rock Band!

Oh my gosh:

Consisting of five tracks, it'll be out on December 18, and will feature tracks by Brad Paisley, Brooks & Dunn, Dierks Bentley, Miranda Lambert and the Dixie Chicks. Pricing is fairly standard, at $2/160MS for a song or $8.50/680MS for all five.

As I've stated before, I love Country Music. Maybe I'll play hooky on December 18th.

Ok, I probably won't, but I'm still excited. Harmonix, if you're listening: MOAR PLZ! KTHXBYE