Good-Bye T-Mobile (and Good Riddance)

I just received a $446 bill from T-Mobile. To say that I'm disenchanted with T-Mobile would be an understatement.

I suppose I should rewind a little bit. I was a T-Mobile customer for over five years. Back in 2003 I signed up with T-Mobile because I was moving to Austin, TX to work for the now-defunct Origin Systems and they had the best plan for me at the time. My wife and I mostly use our cell phones to call each other, so the cheapest no-frills family plan usually suffices for me.

Fast forward to 11/27/2006. I was sick of using the bottom-of-the-barrel free no-camera monochrome Nokia phone that I got for free when signing up in 2003. My original two-year contract had long since expired. Verizon was offering the inexplicably popular RAZR phone for some ridiculously low price. However, having been a loyal T-Mobile customer for over three years, I decided that I'd give them the chance to beat Verizon's deal and compete for my business. The "Customer Loyalty" representative that I talked to jumped at the opportunity. Not only did they give me better RAZRs than Verizon was offering at the same price (and the same two-year committment), he also changed my phone numbers and gave me some text messages for free, waived all of the upgrade fees AND put me on an "unlisted" family plan that was slightly cheaper. Wow! That's customer service!

And then followed four months of hell.

They sent me the wrong phones. They charged me full price. They wouldn't reimburse me to ship them back. Every representative that I talked to showed a new previously-unheard-of level of incompetence and screwed something else up, making it all the more difficult to explain my plight to the next representative. Untold hours on the phone forever lost arguing with people, explaining my situation and fighting to get my bill fixed and figured out. Pages and pages of notes with a myriad of representatives' and supervisors' names weaving a trail of regret and resentment. For four months.

Had it not been for that damned two-year contract I would've flushed them right then and there, but eventually the dust settled and things were worked out.

Two years later, my contract was nearing its end, though I didn't know the exact date it was up. I once again figured that I'd give T-Mobile every opportunity to keep my business. I went into their stores looking at various Blackberrys and the G1. I was interested in a phone that would give me access to my corporate email and the Internet. I was asking for a little bit better price on the phone and the plan. They wouldn't budge. I called "Customer Loyalty" again and they were resilient against my requests. I told them that I really wanted the iPhone and was willing to go to AT&T unless T-Mobile would give me a better deal but instead got stuck talking to some blatherskite who insisted on hammering me on the irrelevant topic that the iPhone was available through T-Mobile--in Germany. I feel that I gave T-Mobile every opportunity to keep my business. Suffice to say, now I'm with AT&T and proud to walk around with my Apple iPhone (I have an iPod Touch too, but adding a phone to it is just brilliant).

Now, about this $446 bill. You'll notice that I upgraded (and started a new two-year contract) on 11/27/2006. I switched to AT&T on 11/23/2008. Four days away from two years and T-Mobile slaps me with two $200 early terminations fees--one for each of my two lines. Four days! At the very least paying the whole amount for my last month would fulfill my obligation to the two-year contract.

Needless to say, I immediately got T-Mobile customer service on the phone. Despite the lividity coursing through my veins I managed to remain level-headed. The "Customer Loyalty" representative offered to cut the $200 per-line fee down to $50 per-line. "That is unacceptable," was my reply. Her next response was cut short as I quipped, "I'll need to speak to your supervisor." The merry Christmas tunes that filled my ear while I was waiting filled me with a sense of irony. Finally the woman came back and told me that her supervisor was busy on another call but she could finagle it so that I'd just finish paying for the full final month, dropping $384 from my bill and returning sanity to the evening. Another 30 minutes of my life wasted on the phone with T-Mobile to combat utter, inane stupidity.

To T-Mobile, if you happen to be listening:

1. Either allow "customer loyalty" more authority to please customers (especially on plan prices) or lose them.
   
2. Pro-rate the early termination fee or at the very least, change your system so customers who are within the billing cycle of their end-of-contract just pay out the rest of their month to fulfill their obligations.
3. Recognize the Charlie Foxtrot cases and assign a single person be your customer's advocate and to sort the problem out. Dealing with one competent person who understood my problem after upgrading in 2006 would've been very nice, but alas, it was not so.
   
4. You can do better than the G1. It doesn't even natively support Microsoft Exchange email.
   

To everyone else, I'd have to recommend against T-Mobile. By all means, use a different cell phone provider. If you're with T-Mobile now, get out as soon as you can. If you're not big on data, you can probably save a bundle by buying individual minutes and not having a plan.

Memory-Mapping Files for Fun and Profit

It's amazing that I still meet programmers who don't understand the benefits of memory-mapping files. If you're unaware of what that means, let me explain it a little bit. Traditional reading from a file involves several system calls: opening the file, seeking to the desired position and then reading or writing to the file. System calls can be slow as the operating system may have to switch to kernel mode and call into drivers to perform the IO operations. Memory-mapping a file usually has just one call, and then you are able to access the file as if the entire file had been read into a contiguous block of memory. But here is the greatest benefit (at least to me as a MMO server programmer): all processes accessing a file can use the same virtual memory.

Take the EverQuest II server for instance. The (binary) data file that describes nearly everything except collision geometry runs about 460MB. Every zone server uses that same file and we run several zone servers on one physical machine. With memory-mapping, the OS uses 460MB of virtual memory (assuming the entire file is used) just once and all zone server instances can use that same virtual memory without having to read the file into each individual instance.

Like anything, there are drawbacks. Most modern operating systems use demand paging, so if you try and access a page of memory representing a part of the file that hasn't loaded yet, a page fault will be triggered and the OS will have to perform kernel-mode IO to load that page from the file. Depending on how much virtual memory your application has access to (and how spread out your accesses are), this could potentially trigger a very high number of page faults.

Another drawback is address space. A 32-bit application will only have access to less than 4GB of address space (Windows has a 2GB address space by default). If you map entire large files into memory, you are consuming this address space. This has become more and more important on the EQII servers. As I mentioned, our data file is 460MB. Since shortly after the game launched, this file has always been fully mapped into memory. That was fine back then as it was much smaller than it is today. Now, after launching our fifth expansion, the data file is now nearly one quarter of our allotted address space (and not getting any smaller).

Fortunately, the APIs for Linux (mmap) and Windows (MapViewOfFile) allow an offset and a length to be specified. This opens the door to partial memory-mapped files. However, there are some caveats. The offset cannot be just any offset; it must be aligned. On Linux the offset must be a multiple of the page size (found by getpagesize), but on Windows the offset must be a multiple of the "allocation granularity" (found by calling GetSystemInfo). The length of the mapping usually does not need to be aligned, but if you're planning a generic partial mapping solution, it's probably best to use the same alignment. Another thing to consider is that mapping parts of files as you need them will spread them all over your address space; you cannot assume that page 1 and page 2 of a file will be placed next to each other in memory if you map them separately.

Here are some of the design goals for my partial memory-mapping implementation:

• Phase sections out as they are no longer used (start-up data is only needed at start-up)
  
• Reduce thrashing (phasing out a section and then mapping it back in)
• Similar performance to the current full-file implementation
  
• Reduce memory usage!
• Ability to permanently map/unmap sections (i.e. sections that don't phase out)
• Ability to automatically re-size sections on demand
  

Allow me to touch a bit on that last point. For my interface, given an offset and a length, I wanted to return a pointer to contiguous data rather than copy data into a provided buffer. Say some previous access forced you to load page 10 (but not page 11) and now you want to read some data that spans pages 10 and 11. This presents a problem for the contiguous data interface. I can't just map in page 11 and hope that it ends up next to page 10 (I will point out that there are 'hints' that can be passed to the OS APIs to attempt address space positioning, but these are discouraged). Do you leave page 10 mapped and create a new mapping that combines pages 10 and 11? Since my interface always required an offset and a length for every access, I could safely unmap the existing page 10 and map pages 10 and 11 together. Data requests for data on page 10 would now refer to the joint mapping. However, this did greatly expand the number of test cases I needed and complicated the logic.

This nifty little piece of technology is currently running on our live game servers and doing very well. Instead of 460MB, we now keep a pool of about 20-30MB mapped from the file and generally enjoy a 0.1% or lower miss rate. Best of all, we still have all the benefits of memory-mapped files.

(Country) Rock Band!

Oh my gosh:

Consisting of five tracks, it'll be out on December 18, and will feature tracks by Brad Paisley, Brooks & Dunn, Dierks Bentley, Miranda Lambert and the Dixie Chicks. Pricing is fairly standard, at $2/160MS for a song or $8.50/680MS for all five.

As I've stated before, I love Country Music. Maybe I'll play hooky on December 18th.

Ok, I probably won't, but I'm still excited. Harmonix, if you're listening: MOAR PLZ! KTHXBYE

/resurrect Xbox-360

I sat down on Saturday to enjoy some Lego Star Wars with my son, but discovered a different feeling welling up inside me instead: dread. My Xbox-360 had fallen victim to every Xbox owner's nightmare--The Red Rings of Death.

He opted to play Super Mario Galaxy on the much more reliable Wii instead, while I googled everything I could about the RROD.

It's interesting to note that Microsoft extended the warranty on the 360 to 3 years. However, I purchased my xbox used from a friend and don't have the receipt from the original purchase. Additionally, the date of manufacture shows as 11/03/2005, so warranty repair seemed like a non-option to me. Plus, I've had friends send their xbox in for warranty repair and it's taken them over a month to get them back. I'm impatient when it comes to gaming! But if you are reading this and have a xbox that is still under warranty, please consider the options carefully before breaking that silver warranty sticker! There is no going back!

I found a guide that provided an at-home remedy for repairing the RROD. A good theory about what causes the RROD is a design flaw leading to flexing the motherboard. This warping leads to microscopic breaks in the tiny surface-mount solder connections exacerbated by numerous heat cycles. The guide directs you to fix the problem by removing the x-shaped brackets holding the heat-sinks to the CPU and GPU and replacing the mounts with screws from any old hardware store.

I thought, "Eh, why not."

So I tore down my xbox:


Getting the heat-sinks off was a pain, but persistence pays off:


The guide I was using didn't tell me this, but you'll also need a 1/4" socket or nut driver to get the posts out of the heat-sinks.

The next step was to clean off the old thermal paste from the chips so that new thermal grease could be applied:


To do this, I used left-over ArctiClean from my last project of this nature. Using the ArctiClean produced a very high-quality mirror shine from the CPU, GPU and heat-sinks:


I staged my bolts according to the guide, put the washers on the motherboard and applied the directed amount of Arctic Silver 5 (about .5 - .75 the size of a grain of rice) to the GPU right before replacing the heat-sink. I then spread the thermal paste out over the chip with the edge of an old plastic gift (credit) card:


The GPU heatsink was then applied, careful to make sure that I had all screws started before allowing the heatsink to contact the surface of the GPU. I very carefully tightened the screws until they were just snug, but evenly snug. Repeat the whole ordeal for the CPU. Here's what the bottom of the motherboard looks like now without the x-brackets:


Note the eight hex cap screws. I recommend using normal phillips machine screws if you can find them. Unfortunately, my local Home Depot didn't. This was all I could find. Originally I had washers on both sides of the motherboard, but the taller hex cap screws had major clearance problems trying to fit back in the metal case. I had to (carefully) remove each screw, remove the washers on the bottom of the motherboard and then replace the screw. I was very glad that I had only snugged the screws earlier.

According to the guide, at this point you should place the motherboard in the case, hook up the necessities and fire it up without the fans to purposely overheat the system and cause the solder to re-melt back into place. However, when I booted it up at this point, there were no more red rings!


Apparently, between the new fastenings for the heatsinks and the better thermal grease on the chips, the problem was fixed! I finished re-assembling the chassis and decided it was time for a real test. I took it back downstairs and hooked it up to the TV, re-inserted the hard disk and memory card and grabbed a controller.

I played Braid for over a half hour with no problems, though I do wonder how permanent this solution will be. If it does break down again, I'll come back and update.

8/9/2010: After 21 months the RROD have returned. I haven't actually pulled the machine apart again to check it out, but it might be worth buying a refurbished base model at this point!

The Shadow Odyssey

Well, the launch was a success! As I promised, I tweeted (through my twitter account) the happenings of launch day as they occurred. Here's the full compilation:

• 11/17 2:19am: Final green-light from QA regarding EQII:TSO is in. Critical mass building... Later today we'll be staging the code to all the servers.
• 11/17 9:46am: At work on the day before EQII:TSO launches. Feels like i didn't get a weekend. Oh wait, that would be because I didn't. :)
• 11/17 10:16am: Wow, 15 minute leads meeting. I think that's the fastest it has ever been #shadowodyssey
• 11/17 2:26pm: Leaving the office now and returning at 3am. Requests to have the Ops dept bring servers down at 12am have been set. cya@3! #shadowodyssey
• 11/17 6:57pm: I better hop in game and finish the D.I.R.T.Y. Quests before midnight! #shadowodyssey
• 11/18 3:05am: *yawn* 3am. Servers came down at 12am and back up running TSO at about 12:45am. QA doing smoke tests now (will be a while) #shadowodyssey
• 11/18 3:09am: i'm patching up now. luckily, I sneakily pre-copied all the expansion assets from the patcher (yay for being internal) #shadowodyssey
• 11/18 3:28am: Unbeknownst to most, Vivox is using the opportunity of all of EQII being down to do some voice chat maintenance #shadowodyssey
• 11/18 5:24am: Smoke test on production servers is going great. Vivox maintenance is done. #shadowodyssey
• 11/18 6:17am: QA still testing strong. Platform QA is testing to make sure that production keys can be consumed properly. Getting closer! #shadowodyssey
• 11/18 7:29am: Breakfast brought in. QA is still testing strong. Funny to see everyone queued up on the login servers. Getting closer... #shadowodyssey
• 11/18 [8am]: And We're Live! Devs are on the server (incognito) playtesting and reporting any issues. #shadowodyssey is smooth so far! [twitter was down at 8am]
• 11/18 10:11am: I (and other EQII team members) are running around Moors of Ykesha on various servers doing quests. Enjoying the expansion! #shadowodyssey
• 11/18 11:37am: Reports of a few minor issues with the broker. We're looking into them. #shadowodyssey
• 11/18 1:55pm: Been here 11 hours (and counting). May head to the launch party. Some minor issues with Ykesha, but I made some tweaks #shadowodyssey
• 11/18 2:40pm: Almost 12 hours here. Heading to the launch party and then home for some Zzzz. /yawn. Great launch so far! #shadowodyssey
• 11/18 4:59pm: Enjoyed seeing @tenamdar and others at the #shadowodyssey launch party! It was a very smooth launch. Now: sleep!

We really enjoyed working on this expansion! Enjoy!

A Nifty Trick

When I made EQII support the /LARGEADDRESSAWARE linker flag, I needed a way to see if people had the /3GB boot.ini flag enabled. The reasoning behind this was to detect what the top of our usable address space was and enable some countermeasures if we started using too much of the address space.

Here's how:

LPVOID pHighest = ::VirtualAlloc( 0, 1, MEM_TOP_DOWN|MEM_RESERVE, PAGE_READWRITE );
::VirtualFree( pHighest );

pHighest now contains the highest address the Operating System will let you have. If it's a value over 0x7FFFFFFF then the /3GB flag is enabled (or your software is running on a 64-bit OS). In any case, you now have the power to make decisions about how much address space to use or if you're using too much.

Of course, all of this becomes moot when we start writing software exclusively for 64-bit OSes.

I can't wait.

EQII Dev Blog: On Your Mark, Get Set...

I wrote a new blog post for the EQII Dev Blog:

We on the EverQuest II team are excited! We’re less than a week away from launching our Fifth expansion, The Shadow Odyssey! For some on the team, this will be the first title that they’re credited on. For others, it’s another box on the shelf that joins a vast array of excellent titles bearing their name. Everyone on the team shares feelings of joy, triumph and pride as we prepare for next Tuesday. Very soon, you will be able to experience the fruits of our labor over the past year.

Braid

If you have an Xbox-360 and you haven't tried out Braid, you're missing a very fun game. Get on Xbox Live Arcade and at least try out the free trial, and then go buy the game.

Braid (by Jonathan Blow) is a platform-based puzzle game (with many throwbacks to the original Super Mario game) that uses various methods of controlling time to solve puzzles. It challenges how you think about time by providing a different mechanic throughout each of five levels. The gameplay is unique, the music is captivating and the art is vibrant and rich. The end of the game is mind-blowing (I'll admit that I didn't "get it" the first time I played through the last level).

I can't say enough great things about this awesome game. Go check it out for yourself.

Americans: Go Vote!

Today is Election Day.

Regardless of what you believe or who you're supporting, this election will make history.

Be a part of it. Go out and vote!

Sure, it's not easy sometimes. The lines can be long. Voting machines can malfunction and cause delays. Weather can be an issue.

But it's a right and a responsibility. We The People of the United States of America have this privilege to choose those that govern over us. Use it.

Go Vote.

unique_vector

Occasionally it's fun to share a programming experience from work. Last week I found myself having to make changes to a tool that builds a particular client data file for EverQuest II. I discovered that this tool would build unique arrays by searching linearly for the content and adding it if it didn't exist. Some of these arrays would get very large (on the order of 30,000 items or more) and the number of checks even larger (200,000+). There are much faster ways to search for a unique item, but we had the constraint of needing to assign a permanent index for the item (i.e. its position in an array).

I wrote out a quick nifty STL-like helper class to accomplish the same behavior, but much faster. The class definition looks like this (note that I used unsigned instead of defining my own size_type for brevity):

template < typename T >
class unique_vector
{
    typedef std::vector< const T* > Tvector;
    typedef std::map< T, unsigned > Tmap;

            Tmap    m_map;
    mutable Tvector m_vector;
    mutable bool    m_dirty;

    void m_build() const;

public:
    unique_vector();

    unsigned size() const;
    unsigned add( const T& );
    const T& operator [] ( unsigned ) const;
};

This stitches together a map (to achieve uniqueness and provide faster lookup time) and a vector to provide index-based lookups. This is mainly achieved through two functions. Let's take a look at the add() function:

template < typename T >
inline unsigned unique_vector<T>::add( const T& t )
{
    unsigned index = m_map.size();
    std::pair< Tmap::iterator, bool > result =
        m_map.insert( std::make_pair( t, index ) );
    m_dirty = true;
    return (*result.first).second;
}

This function relies on the fact that std::map::insert() fails if the key (t in this case) is already present in the map, but returns an iterator to the existing item. This function basically tries to add the item to the map with the next index (using m_map.size() as the next index so the first item will have index 0, next will have index 1 and so on). If it's not there, insert() will succeed and return an iterator to the new record (using the provided index). If it's already there, insert() fails but returns the existing item. The function returns the index associated with the record, whether it's the new index from a freshly inserted item or the old index from a pre-existing item.

Notice that the m_dirty flag is set to true. This indicates that the vector needs to be rebuilt. The operator[]() calls m_build() if m_dirty is set to true:

template < typename T >
inline void unique_vector<T>::m_build() const
{
    m_vector.resize( m_map.size() );
    for ( Tmap::const_iterator iter( m_map.begin() );
          iter != m_map.end();
          ++iter )
    {
        m_vector[ (*iter).second ] = &(*iter).first;
    }
    m_dirty = false;
}

This function quickly builds the vector using the unique indexes stored in the map as array indexes and pointers to the objects used as map keys. The function is const (and data members mutable) to allow it to be called from the const operator[]() function.

So, how much of an improvement was it? The tool uses complex structures as T for this template, but I ran a simple test using 10,000 pre-allocated std::string objects and 100,000 attempted adds. On my AMD 64 4000+ with 2GB RAM, the linear version took 5,077.160 ms (5 seconds). The unique_vector ran the same test in 49.972 ms. Algorithm changes almost always trump when optimizing.

P.S. In case you were wondering, here are the remaining very simple functions:

template < typename T >
inline unique_vector<T>::unique_vector() :
    m_dirty( false )
{}

template < typename T >
inline unsigned unique_vector<T>::size() const
{
    return m_map.size();
}

template < typename T >
inline const T& unique_vector<T>::operator [] ( unsigned u ) const
{
    if ( m_dirty ) m_build();
    return *m_vector[ u ];
}

EQII Getting Multicore Support... Wait, what?!

An interesting note showed up in some recent EQII Test Patch Notes. Yes, it's true. We're working on "Multicore support" for EQII and plan to release the first iteration with Game Update 49. "Multicore Support" is a fan-friendly way of supporting multiple CPUs by way of multiple threads of execution in the client.

I thought I'd take some time to explain what led up to this and offer some explanation of the difficulties of threading. EQII was released in late 2004, but development actually started about five years earlier. In 2004, multiple-CPU machines were generally very expensive and reserved for servers and high-end workstations; the mainstream machines were single-CPU and relied upon increasing clock speeds to keep getting faster. Late in the development of EQII, Intel was starting to introduce hyper-threaded processors. Even though some Operating Systems would detect a hyper-threaded processor as two CPUs, it was really just one CPU. The game targeted computers more advanced for the day (in some ways) and launched largely with the expectation that CPUs would just get faster and faster.

However, clock speed increases started decelerating. Now we find ourselves getting machines that have smaller clock speed increases but more CPUs. Dual- and Quad-core machines are becoming quite commonplace and AMD has even introduced a Triple-core chip.

When an MMO is under development, the core engine design is towards the front of the project so that the tool chains and art pipelines can be created, allowing more people to be more productive. It is then refined over the course of the project, but single-threaded vs. multi-threaded (multicore) is something that should be decided early in the engine design. The reason for this is simple: multi-threaded programming can be significantly different than single-threaded programming and, unless you've done it every day for 20 years, decidedly NOT easier than network code.

Let me explain how single- and multi-threaded programming are different with a simple example. Single-threaded programming is like driving down the single lane of a highway. You're going fast and don't really have to stop for anything. Multi-threaded programming is like two (or more, but we'll use two for my example) highways: one running north-south and the other running east-west. At some point they have intersections. What happens if there are no traffic lights? Disaster. It's the same with multi-threaded programming. You have to find all of the intersections (programmers call them critical sections) and put traffic lights (synchronization primitives) on them. Intersections on a highway are easy to see; unfortunately they're not as easy in code. The other thing to consider is that you have to stop at the intersection and wait for your turn. It's the same with threads. That's also one of the reasons why adding a second CPU doesn't give you 100% more performance.

The first attempt at having additional cores help out was using a technology very similar to OpenMP. I took loops that iterated over lots of things (particles, vertex transformations, degenerate triangles in shadow meshes, etc) and parallelized them. The way this works is like this: Say you have 1000 numbers and you just have to add one to each of them. A single thread would step through all 1000 numbers adding one to each of them. OpenMP allows you to take those 1000 numbers and divide them up evenly among all processors. Unfortunately, this didn't give us any sort of meaningful performance gain as any gains were outdone by the time it takes to synchronize and hand off the data to other threads.

The second attempt became to take the specific system where we're spending most of our CPU time (animation and vertex transformation) and dividing it up in large chunks easily handed off to another thread. This turned out to work very well and netted at 10-15% frame rate gain in populous places. Another bonus was that it was mostly easy to integrate into the single-threaded engine. The way it works is by trying to do animations before the main game thread would. The first time something is animated in the main thread it gets added to a list of items that need animation. The next frame, the animation thread sees these items and can animate them before the main thread needs them. In some cases, the main thread needs one before the animation thread has gotten around to it. No worries! If something hasn't been animated yet, the main thread can take care of it. If the animation thread comes across something that the main thread is using, it just skips it and goes on to the next item. With how useful this system is, the drawbacks are that we only do animation in a separate thread and only one extra thread can really only make use of one extra processor (so Quad-core doesn't have a great advantage over Dual-core [yet], at least as far as EQII is concerned).

So what about the future? There's plenty of systems that we can offload in a similar manner to other threads: shadows, particles, maybe even scenegraph traversals or collision! However, just creating theads for each subtask isn't really the best way to help out. What I'd rather do is create individual tasks for each item: Animate, ExecuteParticleSystem, ComputeShadow, etc. that can be doled out to worker threads similar to Intel's Threading Building Blocks. This would give us the best fit to each processor type and support even higher processor counts in the future! Now that the proof-of-concept has been shown to work and the fan response so far is very favorable, all we need is time.

I hope this post has given you a little bit of an idea why multicore support is not simple, especially on a game engine that was not particularly designed with threading in mind.

TV-B-Gone!

A few days ago I read an article about (and subsequently ordered) a kit to build Mitch Altman's TV-B-Gone. I haven't had the opportunity to build anything electronic in quite a while, so I jumped at the chance.

It came yesterday and I enlisted the help of my 6-year-old son to put it together today.

It's a fairly simple kit with an 8-pin microcontroller, 4 transistors, 2 capacitors, 5 resistors and 5 LEDs (oh, and a push button).

Assembly was a little slow as my son wanted to know what each part did and was worried about his personal safety considering that he equates all electricity with being shocked. I enjoyed explaining it to him and asking him to show me where the parts should go.

I whipped out the old soldering iron and tools and went to town. Putting together electronic kits reminded me of my childhood days assembling HeathKits and tinkering with my 200-in-1 electronics lab (and later, real breadboards and etching my own circuit boards). Ah, the good old days. I'm not exactly sure when I decided to switch from hardware to software, and while I love programming, it's always nice to smell the solder once in a while.

Here's a short video of the initial test of the product:


Needless to say, eating out should be a little more enjoyable now!

STL: A Love-and-Hate Relationship

I'm a big fan of the STL and specifically the containers. Usually. The interface generally makes sense, code is written around them fairly quickly and they have good interoperability. However, there are some problems that really start to become apparent on larger projects. Consider these search results on EQII's codebase:

• "std::map" - 2460
• "std::set" - 1574
• "std::list" - 393

The EQII codebase has nearly 3 million lines of just C/C++/C# (client, server, tools, not counting server lua script and data files) and uses STL containers liberally. If I had to do it all over again, I believe I'd forgo using STL containers for a few reasons:

• Memory Usage - STL allocators typically hold on to memory forever and reuse it as necessary. However, EQII has a very good memory manager with a constant-time free-able small block allocator (written by yours truly), reporting, leak detection, etc. Writing custom allocators for STL is a major pain (especially on an established codebase) and I'd really rather not hack STLport or the gnu implementation to use our memory manager.
• Compile Times - The more templated code you have the longer the compile and link times. I've talked previously about how we replaced std::vector with our own class to save compile times. Also, have you looked at STL implementation code? It's bloated and ugly, which brings me to my next point...
• Debugging - Trying to dig through a container in a debugger watch window (or GDB) is a pain. Sometimes it's even a matter of obscure memory offsets and you have to calculate addresses to see values in a map.
• Additional Functionality - If you have a vector that you're not using anymore and you want to free the memory for it, clear() doesn't cut it. The code to do it is neither trivial nor intuitive: std::vector< T >().swap( myVec ). Adding functions isn't easy as the STL containers are purposely non-virtual.

I'll probably look for a different solution for containers for my next project. Home-grown containers can emulate the STL containers and provide all that I desire above. They can even be compatible with STL functors and algorithms too.

Fan Faire!

Well, Fan Faire is over and we had LOTS of fun. I'm sorry to everyone who wanted to come but didn't make it.

My Facebook friends will already know this, but I have pictures from Fan Faire and Star Trek: The Experience (curiously involving many Fan Faire friends) up at Facebook.

There were many highlights. Here's a list of some of my favorites (in no particular order):

• The Guild Hall Panel
• Alan's various shenanigans with fans up on stage
• Closing down Quark's and the other bar with friends and fans
• The costume contest
• Transmitting the Community Address using in-game voice chat for the first time ever.
• Seeing good friends like Brasse, Kendricke and many more!
• Watching Greg lose some money at Blackjack.
• The Karaoke Party
• and many more!

twitter

I finally made a twitter account. That's so, like, a year ago, but hey, so I'm a little behind the times.

You can follow me here: http://www.twitter.com/autenil

See you at Fan Faire!

In about 7 hours I have to be at work to leave for Las Vegas and SOE's Fan Faire.

It's always a blast to talk to players and sit on panels where we answer questions about EverQuest II.

We're going to do something interesting this year for those who can't make it to Fan Faire. For the Community Address where Smed and various SOE producers show off cool new things, we're going to be providing an in-game Voice Chat channel to listen to. I worked heavily on EQII's Voice Chat and for one am very excited about this.

A little sad is that the Star Trek Experience is closing in less than a month. Fortunately, Fan Faire is at the Hilton this year, so I'll be sure to experience it and get some good pictures. Anyone who wants to buy me a Warp Core Breach won't be refused :)

What should my next computer be?

So it's time to upgrade my computer.

Should I build or buy? Suggest components or a manufacturer.
Should I chose a slightly slower quad-core or a slightly faster dual-core?
Intel or AMD?
nVidia or ATI/AMD?

OS will probably have to be 32-bit to use VPN at work but I'm not opposed to running a 64-bit OS as a dual boot. However, if you suggest Vista, anything you say will be held against you. You have been warned. :)

We'll say that my budget is around $1500 and including a 1920x1200 flat panel.

Comment away!

Lacy, Gently Wafting Curtains

Oh Em Gee.

If you haven't yet, you MUST check out Dr. Horrible's Sing-Along Blog. Funniest thing I've seen in a long time. As a bonus, you can watch it now for free at Hulu.com. However, if you love it as much as I do, you might possibly want to purchase it from iTunes and benefit Joss Whedon and friends. It's only $4!

In other news, I had a bunch of fun at SOE's Block Party. However, the same day as the Block Party I found out that my awesome friend Patrick "Leurocian" Malott from Mythic was in town for a UO Town Hall, so I headed down there to attend the Town Hall and then brought Patrick up to the Block Party to check out the SOE New Hotness. We had a playable demo of DC Universe Online that people were loving, a new dungeon on display for EverQuest II, other games, a live band, food and more.

Speaking of EverQuest II, Game Update 47 is set to launch tomorrow. With support for (finally!) the Voice Chat system I've been working on for the past few months. Yay!

Going to AGDC?

I am headed out to Austin in September for the Austin Game Developers' Conference (woohoo!)

Who else is going? Definitely need to see some old friends while I'm out there...

Interesting Quiz

144 As a 1930s husband, I am Very Superior Take the test!

No doubt my wife would (will?) take issue with that score ;)

Voice Chatting Across Continents

EverQuest II is product that is translated into several different languages and runs in several different regions around the globe. We want our voice chat to be game-wide, so that means making it work in those different regions. (Having a Universal Translator would be nice too, but I think that's a few years away).

Anyways, different languages have different character sets for names and text, so EQII uses a combination of UTF-8 encoding and Unicode to represent all text (names, places, quests, player communication, etc). However, a common signaling protocol used by voice chat applications (including Vivox's that we're using for EQII) is a case-insensitive, ASCII-only protocol known as SIP. This leaves us with a dilemma: how do we represent Unicode player names and channel names in the SIP protocol?

Originally we were going to use Base64 encoding, but this has a variety of problems for our application:

• It requires case sensitivity, but SIP is case insensitive
• Encoded text is not human readable
• It increases the size of the data

In the end, we found and decided to use an interesting encoding scheme called Punycode. Consider the Russian word "свободными". UTF-8 encoding requires 20 bytes. Our encoding must fit within the limit of about 60 characters. Standard URL percent encoding requires 60 characters. Base64 encoding would probably require around 26+ characters. Punycode requires only 13 characters! What's more is that Punycode is human readable for ASCII strings ("Autenil" encoded in Punycode becomes "Autenil-", but "свободными" becomes "90abhqtfebx0i").

I've enjoyed discovering this neat encoding algorithm (and it's been a lifesaver for our Voice Chat implementation). If you need to efficiently represent Unicode in ASCII, Punycode might be the way to go.

More EQII/nVidia news

If you're still having trouble with EverQuest II and your nVidia 8800, rest assured that we haven't forgotten about you.

In my talking points on the subject in the past, you'll note that we added a feature called "Reuse Vertex Buffers" that seems to alleviate the problem for some folks. However, it doesn't completely fix the issues that EverQuest II (and other games) are having.

We're currently working with nVidia specifically on this problem and are currently testing out some super-top-secret-internal-only-penalty-of-death drivers in our compatibility lab with high hopes. I'll keep you posted when I hear more.

Geek Heaven

/. turned me onto the Interactive Linux Kernel map. This thing is ridiculously cool (well, for a geek like me anyways). Browse through subsystems of the Linux kernel from low level (hardware, drivers) to high level (system interfaces) and links take you directly to source code!

I don't imagine most people to geek out over this as much as someone like myself, but hey, let's face it: I eat and breathe programming and used to read Linux kernel code for fun.

Yeah.

EQII Voice Chat is on Test!

We've successfully pushed the new voice chat feature out to the Test server for users to play around with and provide feedback.

Everything went really well, except that we noticed while the client was being pushed to the patcher that the help files describing the new voice chat feature were missing (quickly remedied).

Zoltaroth, Rothgar and myself all hopped onto the Test server and invited people into a custom channel. Players had very few issues getting set up and chatting and we enjoyed talking to them rather than typing! Users really seemed to enjoy it and had great things to say about it (while providing good feedback too).

Feel free to hop on the Test server and give it a go!

EQII Voice Chat - almost here!

A while back I posted an intro about SOE's new relationship with Vivox in order to bring Voice Chat to EverQuest II and the rest of the SOE lineup of games.

As I write this I'm at work (yes, on a Sunday) to fix up a few outstanding issues in order to try and get Voice Chat working on our Test Server early this week.

Myself and Rothgar have been slaving away to bring this feature to fruition for EverQuest II. Recently we've been recruiting people inside SOE to try out the integrated Voice Chat in our awesome brand-spanking-new Usability Lab. It's been enlightening and caused us to rethink a few things about the UI before going to the Test server (plus it's rather humbling when watching someone's facial expressions when dealing with your code).

I also wrote up a Featurette on the EverQuest II forums to build some buzz (with screenshots!). Apparently Massively liked my writeup too, so that was cool.

We've been working to make the integration of our voice chat seamless with features like lowering the game volume when you or other people are talking in voice chat, speaking indicators on the Raid and Group windows, ability to invite into Guild, Raid and Group channels, and more! I can't wait for you to try it out! :)

The Stork is comin' back around!

We're pregnant! And super excited about it!

This will be our third munchkin and the baby is due around January (which, curiously, is when my and my wife's birthdays are, as well as several of our friends). Maybe the baby will be born in December and I can claim the tax exemption for the whole year :D *crosses fingers*

Anyways, this new addition is sure to cause several changes. We need a bigger vehicle (which I lament due to fuel costs) and will probably end up re-arranging who is in what room of the house. My wife is already in nesting mode and I have about nine bags of clothes to donate after she cleaned out a few closets.

So here's to another very-large-quantity-of-money dedicated to a sure-to-be love of my life! Money well spent! (allocated?)

Random Ramblings on the price of Gas (and other such annoyances)

Early disclamer: This is one of my rare but sometimes provocative opinion posts. Brace yourself.

Well, the average price of gas right now is over $4.21/gal. After some random looking on the web for ways to get higher MPG, I came across a story about Wayne Gerdes. It's a really interesting story and I highly suggest you check it out. I totally admire the guy's motivation: he wants to do his part to reduce American oil money funding Middle-Eastern terrorists. My core reason however is probably a little more self-preserving: I just want to save money.

I've really been trying to modify my driving style these past few weeks, though not to the extent Wayne does. Turning your car off and drafting semis doesn't seem like a great idea with two kids in the backseat. The people who know me as "speedman" and, well, anyone who's ever ridden in a car with me probably can't believe this, but it's true: I barely hit the speed limit on the highway, I'm coasting as much as I can and I'm watching the MPG meter in my wife's car (mine doesn't have one... yet) like it's a video game score.

Fortunately, there's one thing that allows me to travel the 6.5 miles to work on a whopping zero gallons of gas: my bicycle. Other bonuses of biking to work include exercise, fresh air, more energy and the occasional wolf whistle. Not all cities are great for biking, but fortunately California has paid special attention to cyclists and it's not uncommon to see them on the roads. Unfortunately, I can't sell my wife's beloved turbocharged premium-guzzling Audi and get her a bike to take the kids all over town. At least, not if I expect to have a wife much longer.

I also hear cycling is also pretty good for the environment. While that's all fine and good, things I do for the environment are usually to save me money, so you won't catch me buying carbon offsets (or onsets for that matter) or funding silly counter-productive ventures like Earthrace. No, I don't buy into the global warming nonsense. I do however recycle what I can and I paid a gardener to plant a whole bunch of flowers and vegetation in my back yard.

Which brings me to my second rambling point: Compact Florescent Lamps. I saw the money-saving promises on the box and was about ready to jump on the bandwagon when I thought I should dig a little deeper. Certain things are obvious: they're more expensive and should last longer than incandescent bulbs, but instead of burning out in one giant flash, they have some rather undesirable end-of-life characteristics. The light they give off usually isn't the most pleasing, and if you can hear high-pitched sounds, the noise they make can be irritating. The kicker for me is that they contain harmful (to people and the environment) mercury and require special consideration for breakage and must be safely recycled or disposed of following hazardous waste guidelines. Also, check out this quote:

Although mercury in these bulbs is a health hazard, special handling upon breakage is currently not printed on the packaging of household CFL bulbs in many countries. It is important to note that the amount of mercury released by one bulb can exceed U.S. federal guidelines for chronic exposure.

No wonder the packaging looks enticing! You don't find out how dangerous it is by reading the box.

Now, all that said, I did use a few CFLs, but I'm not converting all of my lights to them. I only used CFLs for the lights that I tend to leave on for long periods of time, like ceiling family and living room lights and the lights I leave on overnight. Again, trying to save money. But I don't want to imagine the horror of my kids breaking one or my wife or I having to follow the HAZ-MAT cleanup requirements. There are reasons we don't have mercury thermometers anymore.

Ok, done ranting for now. I'll be the guy on the bicycle laughing at the SUVs with the pedal to the metal speeding past on $4.21/gal gas.

Iron Man

Two Thumbs Up.

I was never a big fan of comic books, so I went into the movie with near-zero expectations. The humor elements were great and the visual effects were awesome.

For the geeks, here are some articles you might be interested in.

Well worth seeing more than once in my opinion.

Oh, and don't forget to sit through the credits ESPECIALLY if you are a fan of the comics. There's a scene at the end that you definitely don't want to miss.

Latest news on the nVidia 8800 problem

We just did an update to our Test Server that has a new option for players suffering from the nVidia 8800 stuttering issue. The new option turns on a vertex buffer cache that will hold onto and reuse vertex and index buffers without trying to free them all the time.

We plan to hotfix this code live soon.

Incidentally, this Test update also has a fix for the Veeshan's Peak particle leak.

Leaking memory by the kilo[byte]

I finally got around to looking into reports of a memory leak in Veeshan's Peak mostly dealing with lava bombs on a bridge.

Sure enough, there's a leak. After a 30 minute meeting my test client had grown half a gig. Just while watching lava bombs bash a bridge. (Well, this is a debug-build client which uses more memory and doesn't have any optimizations, so that number is a little inflated)

Knowing that you have a leak is just the beginning. Sometimes, finding it (or fixing it) can be a pain.

We have some pretty decent tools built into the EQ2 framework for dealing with memory issues. I'm pretty proud of them. Given that, it didn't take long to find out that we were leaking particles, hundreds of them per frame.

EQ2 uses a lot of reference-counted objects to reduce memory leaks. Ironically, they can also contribute to memory leaks when you have a reference cycle (or circular reference). A reference cycle is basically when two objects hold references to each other. Unless something breaks the cycle, the reference counts won't decrease allowing both objects to be destroyed. Finding reference cycles can be difficult too since it's easy to determine that something is holding a reference to an object, but it's not always so easy to know which object is holding the reference.
... Until I instrumented our smart-pointer class to do that without having to change the many thousands of places where smart-pointers are used. The concept is pretty simple: The smart-pointer remembers the line of code (code address) where it increments the reference. The base object (with the reference counting support) remembers which smart-pointers are pointing to it. This allows me to know which line of code caused the excess reference counts. In this case, the smart-pointer is no longer a simple class that adds and removes references; it gets a little heavier. The good news is that you only need that extra weight when you're trying to find memory leaks. I'll leave this as an exercise for the reader.

So, case in point, we definitely had a reference cycle with some particles. Basically particles would reference scenegraph nodes that owned them. The fix is essentially to not hold a reference if it would cause a cycle, but we still need to hold a reference if there wouldn't be a cycle.

Therefore, without further ado, the fix:

Now, any excuse to use placement new gets me a little giddy. Plus I want to keep the Particle object size small because we always have a lot of them.

union
{
    SceneGraphNode* pRawNodePointer;
    char space[ sizeof( SmartPointer< SceneGraphNode > ) ];
} m_nodePointer;

Since these two members are bound by a union, I can only use one at a time. If I want to use a smart pointer, I have to construct it using placement new:

new (m_nodePointer.space) SmartPointer< SceneGraphNode >( pNode );

If I want to switch and use the raw pointer, I have to destruct the smart pointer:

reinterpret_cast< SmartPointer< SceneGraphNode >* >( m_nodePointer.space )->~SmartPointer< SceneGraphNode >();
m_nodePointer.pRawNodePointer = pNode;

Voíla! All the particles get cleaned up now. This fix should be making its way onto live servers pretty soon.

Update 04/17/08: This fix is now on live servers!

nVidia 8800 Update

If you're following the current EQ2/nVidia 8800 continuing saga, here's an update:

nVidia finally contacted us (after seeing my previous blog post i might add). We're working with a technology evangelist there to get the problem resolved.

Since some people mentioned not having issues with Vanguard: Saga of Heroes, I started chatting with the VG team about ideas. Unfortunately the ideas that they've given me so far haven't panned out, but there's more to try.

Between both of these, we're definitely working on a solution. I'll keep you posted.

Update 4/16/2008: See latest update

Interview up on Stratics

My good friend Ransom of BiC is working for Stratics now and asked me if she could do an interview of me.

Head on over to Stratics and check it out! (And the forum thread)

April Fools' Memories

Every April Fools' day it reminds me of my first 4/1 in the game industry. I had been working on Ultima Online for about six months and it was still run by Origin Systems in Austin, TX. The players had rose-colored glasses of the era of UO that came with the first expansion (The Second Age) just prior to the second expansion (Renaissance). There was always a post on the forums calling for a pre-Renaissance server.

So on April Fools' I decided to give them just that. I tweaked some flags on the test server to turn on the much more difficult "Siege Perilous" ruleset and renamed it "UO Pre-Ren".

I wish I still had a link to the post on the forums with the player responses after that.

In retrospect, while it may seem a little mean, it was well within the spirit of the holiday. And you've just got to expect stuff like that on April Fools' day.

And, in case you're wondering what we did in EQ2, Kendricke found it by surprise.


ALSO: I just checked the Rock Band song list to see if there were any new songs worth downloading. At first I thought the new song was just an April Fools' joke, but it's for real! I'm jammin' away on it right now! Best of all, it's free!

nVidia 8800 Frustrations

We've had several EQ2 players complaining about stuttering issues with their nVidia GeForce 8800s. Enough that I requested a machine with the card in it from our Compatibility Lab so that we could diagnose the problem. We were able to reproduce the problem right away. The programmer who looked into the problem used MS PIX to isolate the cause and discovered that some Release() calls were taking over 100ms! So, it appears that sometimes when lots of dynamic allocation (and freeing) is happening, returning resources to Direct3D can take WAY too long.

It seems EQ2 isn't the only game affected. UT2003 and UT2004 and Team Fortress 2 as well as Test Drive Unlimited have people reporting problems. Even a Mac user reports issues using applications. Curiously, however, some games seem to be largely unaffected but I don't know why.

With so many games affected but comparable- and lower-end cards not having issues, I suspect the card and/or the drivers. The data we collected from PIX regarding Release() supports this theory. Furthermore, we've been able to adapt one of the Direct3D sample apps to demonstrate this issue. The fact that newer cards seem to work better and some people flat out don't have the issue suggest that it might be hardware, but I'm just speculating. nVidia is usually hailed as one of the better hardware/driver providers, but recently reports are surfacing that paint a different picture (pun only slightly intended).

So, assuming nVidia doesn't (or can't) fix this problem, where do we go from here? If the problem is specifically limited to Release() calls, we could cache textures and vertex and index buffers instead of releasing them, but this has its own set of problems. We've tried to open contact with nVidia, but so far have been unable to get a response (update).

If you're using a GeForce 8800-series card with EQ2 and have stuttering problems, my apologies. We're working on the problem.

Update 4/4/2008: See post
Update 4/16/2008: See post
Update 7/1/2008: See post
Posts tagged "nVidia"

Infinite Loops Are Bad

When working on an MMO server, one thing that you're never going to get rid of are crashes. We generally don't like to admit it, but it's true. Crashes can be caused by many things including bad software, bad hardware, bad programmers, bad state, bad anything. Linux (and Windows) MMO server developers typically have crash recovery down to a science: dump a core file (or minidump under Windows), mail it off to the programmers and restart the process.

Worse than crashes (and hopefully less frequent) are a little problem known as Infinite Loops. But if you have a server process lock up, how do you get useful information in order to fix it? We want to treat Infinite Loops like crashes and get a core file or minidump that shows where the lockup occurred and hopefully why.

The concept behind detecting an infinite loop is trivial: Start the main game loop as a separate thread and increment a value over time (say, every main loop iteration). The initial thread then looks for this value to be changed over time. If the value goes long enough without having changed, your process can be considered to have locked up. I'll leave it as an exercise for the reader.

So how does our infinite loop detection thread get the other thread to drop useful information?

In Linux (and most other flavors of *nix) it's really easy. Just do a pthread_kill() to the locked-up game thread with a signal that drops a core (SIGABRT usually works nicely) and then _exit().

For Windows, there's a little more code to write and you have to have dbghelp.dll. Basically, just write a minidump file. The normal minidump file should have information on all threads, so even though you're writing it from the main thread doing the infinite loop detection, it will include all the necessary info about your locked-up thread.

Here's a very simple sample of a Linux program for making a thread drop a core:

#include <pthread.h>
#include <stdio.h>
#include <signal.h>
#include <unistd.h>

#include <sys/types.h>
#include <sys/stat.h>

void* threadMain( void* )
{
        while ( 1 )
                ;
        return 0;
}


int main( int argc, char** argv )
{
        pthread_t id;
        if ( 0 != pthread_create( &id, 0, threadMain, 0 ) )
        {
                perror( "pthread_create failed" );
                return 1;
        }

        sleep( 1 );

        pthread_kill( id, SIGABRT );
        _exit( 0xabcd );
}

Why am I writing about this? Previously EQ2's infinite loop detection would essentially do this:

assert( 0 && "INFINITE LOOP DETECTED" );

Which would crash the process and cause it to be restarted, but doesn't give us any useful information about where or why it locked up. In a past life when working on UO we didn't even have any infinite loop detection, so a locked-up server would just blackhole people and prevent the shard from doing synchronized backups. On EQ2 we don't typically see a lot of infinite loops at this point, but it's recently been changed to give us more information. And that's not something that you're likely to see in the patch notes ;)

Edit 3/29/08 10:23 PM:
Response based on a comment by KC (see comment below). Unfortunately, blogger doesn't support <pre> or <code> tags :P
Infinite Loops in EQ2 are usually caused by programming errors. Our script system (lua) has an instruction limit that the EQ2 team added so Designers can't cause infinite loops. An unending dialog cycle (if I'm interpreting you correctly) wouldn't be an infinite loop since it requires user interaction (the server is waiting on the client before serving up the next dialog). The simplest infinite loop would be something like this:

while(true){/*do nothing*/}

This just "does nothing" forever without giving the server the chance to do anything else. We actually have a dev-only command that does this for testing the infinite loop handler :)

The last time we actually had a legitimate infinite loop issue, the code looked something like this:

for ( unsigned i = 0; i != kiCount; /*increment in loop*/ )
{
    if ( shouldSkip( i ) )
    {
        // oops, forgot to increment i!
        continue;
    }
    // real logic
    ++i;
}

EQ2 Devs playing on Test

Greg "Rothgar" Spence started a thread on the EQ2 forums asking for classes that the devs should create characters and play with.

I started a Dwarf Ranger named "Autenil" in Greater Faydark on Test server.

Feel free to message me if you see me in-game.

Voice chat is coming to EverQuest II

A few days ago, SOE and Vivox announced an agreement to bring voice chat and more to SOE's lineup of games.

Personally, I'm very excited about this. And it's not just because I'm the one working on it for EverQuest II. Some of the features mentioned in the press release will be interesting, such as the voice font feature (which allows you to change your voice real-time) and the ability to call into guild chat with a regular phone.

Memory Usage and the EverQuest II Client

3D graphics tend to use a good amount of memory which increases with their level of detail, texture size, complexity of geometry, etc. Combined with sound, scenegraphs and other things that make up a game client and you can use a lot of memory.

The EQ2 client is pretty memory hungry, but we're making strides in changing how that works a little bit. Currently, we have several reports of issues with Vista. While it's not technically a supported OS for EQ2, we still want to ensure that people can play the game. Unfortunately, either Vista's DX9 compatibility or Vista drivers seem to be using a LOT more memory then they did on Windows XP.

To mitigate this problem, there's a few things we're doing.

• /3gb switch and 64-bit support - Most Windows applications are not aware of large addresses (i.e. memory addresses over 2GB). There is a linker switch that tells Windows that your application can use addresses over 2GB. This has a dual benefit: for users on 64-bit Windows, the EQ2 client will have access to nearly the entire 4GB address space. For users on 32-bit Windows, there is a switch that can be added to your boot.ini file that tells Windows to allow up to 3GB user address space. While this change may prevent out-of-memory crashes, they don't solve the original problem that the client uses too much memory.
• Texture Downsampling - This system picks textures that are far enough away and unloads the highest quality mip level. This can lead to significant memory savings as well as saving texture memory on the video card, without significantly degrading the quality of the graphics.
• Non-drawn geometry unloading - Since the game loads assets basically within a radius of the camera, sometimes things get loaded but are occluded and used rarely. Things that haven't been drawn in a long time don't need to be kept in memory and can be unloaded. This saves both system memory and video memory since we don't need to hold onto textures and vertex/index buffers that aren't being used. Generally, the performance cost is negligible. Loading things back in has some cost, but assets have a better chance of being in the disk cache greatly reducing the load time.

The systems mentioned above also won't come into play until you hit a certain amount of memory in use. There are other things that we're looking at too, but the changes mentioned above appear to have the most bang for the buck.

/shakefist Time-Warner

I turned on my HD cable box/DVR to find that Time-Warner had automatically downloaded new software to it that, among other things, has a new UI.

I haven't messed around with any options yet, but my biggest complaint is...
... it erased all of my scheduled recordings for the DVR!

Well, all of the ones that won't be on in the two-week program guide. Unfortunately, this includes most of the shows that were prematurely canceled/postponed this season (i.e. The Office, Heroes). It also somehow transformed my recording of Star Trek: The Next Generation (which was taken off of Spike a while ago) into Star Trek: Voyager. Sub par.

I have to say, this is quite inconvenient and I'm a little disappointed. Even the new menu fading is really annoying.

On the plus side, they got rid of those nasty gray filler bands when watching standard programming on an HDTV. That's about the only thing that I've seen that I like. Next time, Time-Warner, at the very least warn me. Preferably, give me an option.

All work and no play...

... Fortunately, that's not me. I've been playing plenty when the time allows it. Here's what I'm currently playing:

• Rock Band
Yeah, I'm a sucker for rhythm games. I'm best at the guitar having completed the solo career on Expert mode. My (lack of) drum skills keep me constrained to the Medium difficulty currently, but I have finished the vocal career on the Hard difficulty. Unfortunately, the drums tend to keep the family awake at night when I normally play, so it may be a while before I can play them very well. Besides, mine are nowhere as cool as Greg's setup.

When I get the chance (not very often) I play plastic guitar for two bands, Juicy Torrid Crane and Ugly Lampshade. Unfortunately I haven't gotten many achievements yet in Band World Tour mode.

• Guitar Hero 3
I was playing this pretty heavily until I got Rock Band, but it pales in comparison. There are definitely some great song selections. I've beat the solo career on Hard, but am having trouble with one of the songs on Expert, Raining Blood (not my favorite song by any stretch).

• EverQuest II
Besides working on it, I do enjoy playing EQ2. I two-box an Ogre Shadowknight and a Wood Elf Fury and am currently around level 72. I generally play with some other people from SOE but also love a good pick-up group. I've moved servers recently and currently play on Unrest.

So that's what I'm playing these days. Oh, and I'd love to finish Mass Effect and Lego Star Wars: The Complete Saga. I generally don't get a whole lot of play time since I play late at night after everyone else has gone to bed. Family comes first!

Hey look! People making fun of me!

Apparently my LEGO Millennium Falcon Video ended up posted at I-Am-Bored. The comments range from nice to brutal. It's fun to LOL at peoples' speculations. Check 'em out and see if you agree ;)

Eee Queue Too

If you've come here looking for EverQuest II info, you might be interested to know that one of my programmers, Greg Spence, has a blog over here. Well worth checking out!

Haxx0ring 4tw

When I heard that Hoglund and McGraw were writing a book about Exploiting Online Games, I just had to have a copy. It's the best book purchase I've made this year, and I think it's a great book for my crowd, the MMO game developers.

I've written on this topic before, and I feel very strongly that the game development industry doesn't give enough thought about black hats who will be on your system.

That being said, not all hackers are malicious. When I first started hacking Ultima Online in 2002, it was mostly out of curiosity. I drank up as much information about UO as I could find. My hacks were never malicious in nature; I never tried to crash the server or cheat other players or anything. I was just a programmer with a love of the game and a penchant for the low level workings of machines. And I was getting bored trying to tame a bajillion animals to get my skill up.

I discovered for myself most of the techniques that Hoglund and McGraw talk about in their book. My first tool was a keyboard/mouse macro program. By then I was curious about what kind of stuff was going over the network, so I set out trying to get at the data. As expected, it was encrypted. Instead of trying to crack the encryption, I wanted to find something easier: how do I make the client tell me the decrypted info? I eventually produced a DLL that I could inject into the client which would re-write some machine code and forward me the messages. Voila! The tools started getting really interesting now. In no time, I had a packet logger, apps that pointed me to hidden objects on the ground, and a simple macro program making the aforementioned taming much easier.

After reading through the book I dug out my old code and sat down with a blanket by the fire for a good read. The code wasn't pretty but the things it was doing were pretty amazing.

But now I work on the other side of the fence. I have to think about why people want to hack and how it affects everything. It all comes down to money. That and an impatient fast-paced culture. Simple economics teach us about supply and demand. If there were no demand for gold, then gold farmers would move on to doing something else. You can do all you want trying to detect, ban, or threaten farmers until you're blue in the face; that's where the money is, so that's what they'll be doing. Plus, there will always be more of them then there are of you. These are points that you learn in the industry, but Hoglund and McGraw spell out explicitly.

There's some stuff that you can do to mitigate the circumstances: don't rely on the client for anything and watch for irregularities on the server (economy logs, inhuman playing time, repetitive behaviors, etc). But I think there's another point that not many people discuss. I think you can make a difference in farming in your game design. If you design your game so that there is no player trading, you've effectively cut off the farming. But not being able to sell anything to other players has a problem: it isn't fun. The trick is striking a balance, but we haven't seen it yet.

If you are interested in how MMOs work and think about hacking software, this is the book for you. If you are a programmer in this industry, this book is definitely for you. Go buy it!

New MP3 player (part deux)!

Last August I found myself in the possession of a Zune after my iPod caused me great suffering. This post should alternatively be called "Don't buy a Zune." Any glee that I had about replacing my old iPod quickly melted away the first time I tried to buy songs online for the Zune. Is it just me or does Microsoft actually want to prevent you from buying songs online? I never did figure out how to do it through their unintelligible software. But of course, I did give up pretty easily. This article has a pretty decent shortlist of things wrong with the Zune and Microsoft's campaign (i.e. marketing hari kari).

My loving wife sensed my frustration (either that or got sick of looking at the ugly thing) and gave me a 16GB iPod Touch for our anniversary. Like with the original iPod and the iPhone, Apple continues to innovate and blow the socks off of the competition. Whatever issues were present in my older iPod that caused songs to cut off seems to have been fixed with the brand new software in the Touch. The screen is brilliant and the touch sensitive UI is so amazingly intuitive. The video playback quality is quite nice as well. There's no FM tuner, but that was never a feature that I found myself using on the Zune.

The downsides that I have encountered have been few, but important. First, the lack of memory due to using flash rather than a miniature hard disk. Even my old iPod had 30GB, whereas now I'm trying to make do with 16GB. Furthermore, the iPod no longer functions as a USB storage device. Third party utilities can be installed on a jailbroken iPod Touch, but it's much more difficult for the average user. Also, there's no firewire support for the Touch. That, combined with inefficiencies in iTunes (change the name of a video and you have to download the whole video) make for some long sync times. Oh, and in full Apple fashion, it's not cheap.

My opinion is that if you have the cash, go for it. Stay away from the Zune and grab an iPod Touch.

Welp, I guess it's public

According to this article, I'm now (publicly) the Technical Director for EverQuest II:

Zam: With Don Neufeld also leaving, who is the new Technical Director going to be?

Bruce: Joshua Kriegshauser (Autenil) has been the Assistant Technical Director for years, and is stepping up to the plate to handle everything that Don was. I worked with Joshua quite a bit before I left for the team, and can’t think of anyone better suited for it. He’s a real star, and I’m delighted to have him here.

Thanks, Bruce. It's a pleasure to have you back on EQ2 and I'm glad to be part of your leadership team.

Holy Holiday Whirlwind, Batman

We just got back from nearly three weeks in beautiful Kansas City. Where to begin?!

We left on Tuesday 18 Dec 2007 and man, traveling with 2 kids and the all-out ridiculous amount of luggage that we had was insane. 5 checked bags, 4 carry on items, 3 (and a half) hour flight, 2 kids and a partridge in a pear tree. We brought the Christmas presents for the kids so that they'd have plenty to open on Christmas day.

On Friday 21 Dec 2007 we were supposed to hop in the car for 5 hours and visit family in Iowa. However, that plan went down the toilet when we learned that my younger sister had gone into the hospital in Emporia KS for complications arising from Crohn's Disease. This wasn't her first hospital visit, but little did we know how bad of shape she was actually in. She was flown to Kansas City and underwent emergency surgery. I was in the surgery waiting room when the surgeon came out and told us flat out that she might not survive. That's something that you never want to hear and I gotta tell you, I wasn't ready to hear that. There were probably 30-40 other people from church who showed up to support us while sis was in surgery. Everyone started praying and we even started a Facebook group to get people praying for her (if you don't have a Facebook account, go make one). That group also has status updates and most of the story. Long story short, she's still in the hospital and probably will be for another few weeks. The doctors and nurses have been astounded at the miraculous recovery that she is making, so we'll just keep on praying :)

We stayed with my folks and spent a lot of time there and visiting family (my wife has 7 siblings and most are nearby, so there are lots of people to see). We also were introduced to my brother's fiancé and are looking forward to their wedding coming up in May.

I also got to have lunch with an old friend from my high-school/early college days when I worked at CompUSA. I had a lot of fun doing that and talking about old times, game development, cycling, etc.

With Jessica in the hospital, we did all of our present opening on Christmas Eve. Once again, we completely overbought for the kids. My wife walked away with lots of nice jewelry (including a diamond bracelet and a new watch) and I received wireless headphones, clothes, and an iPod Touch. The watch and iPod were technically anniversary presents. With the new iPod I was able to give my Zune to Jessica and she has been getting some great use out of it while she is in the hospital.

To my wife's chagrin, I took my XBox 360 with Guitar Hero 2/3 and they were a real hit. My brother got hooked on GH (picked it up for the Wii). Some of my siblings-in-law really enjoy playing it and with their help I managed to unlock most of the 2-player career achievements.

I also got a chance to play Super Mario Galaxy and must say, it's a very entertaining game! My brother picked it up for his Wii and my son instantly fell in love with it (he loves Mario anyways, so anything new is just icing on the cake). I'm definitely going to have to pick it up myself.

Earlier today, we packed up our utterly insane amount of luggage (6 checked bags this time around!) and headed back to the airport. Everything went as smoothly as it can with 2 kids and more carry-on than you can carry on.

It's good to be home.