UOForums Interview

UOForums asked me for an interview a while back. I finally obliged:

What has been your most memorable accomplishment in your career?

Making Technical Director before 30? :) Getting hired by Origin is probably the most memorable accomplishment as it was my first gig in the game industry. It was an amazing experience: I got to meet and work with the people behind the names on the forum that I idolized. The whole experience was surreal.

Haxx0ring 4tw

When I heard that Hoglund and McGraw were writing a book about Exploiting Online Games, I just had to have a copy. It's the best book purchase I've made this year, and I think it's a great book for my crowd, the MMO game developers.

I've written on this topic before, and I feel very strongly that the game development industry doesn't give enough thought about black hats who will be on your system.

That being said, not all hackers are malicious. When I first started hacking Ultima Online in 2002, it was mostly out of curiosity. I drank up as much information about UO as I could find. My hacks were never malicious in nature; I never tried to crash the server or cheat other players or anything. I was just a programmer with a love of the game and a penchant for the low level workings of machines. And I was getting bored trying to tame a bajillion animals to get my skill up.

I discovered for myself most of the techniques that Hoglund and McGraw talk about in their book. My first tool was a keyboard/mouse macro program. By then I was curious about what kind of stuff was going over the network, so I set out trying to get at the data. As expected, it was encrypted. Instead of trying to crack the encryption, I wanted to find something easier: how do I make the client tell me the decrypted info? I eventually produced a DLL that I could inject into the client which would re-write some machine code and forward me the messages. Voila! The tools started getting really interesting now. In no time, I had a packet logger, apps that pointed me to hidden objects on the ground, and a simple macro program making the aforementioned taming much easier.

After reading through the book I dug out my old code and sat down with a blanket by the fire for a good read. The code wasn't pretty but the things it was doing were pretty amazing.

But now I work on the other side of the fence. I have to think about why people want to hack and how it affects everything. It all comes down to money. That and an impatient fast-paced culture. Simple economics teach us about supply and demand. If there were no demand for gold, then gold farmers would move on to doing something else. You can do all you want trying to detect, ban, or threaten farmers until you're blue in the face; that's where the money is, so that's what they'll be doing. Plus, there will always be more of them then there are of you. These are points that you learn in the industry, but Hoglund and McGraw spell out explicitly.

There's some stuff that you can do to mitigate the circumstances: don't rely on the client for anything and watch for irregularities on the server (economy logs, inhuman playing time, repetitive behaviors, etc). But I think there's another point that not many people discuss. I think you can make a difference in farming in your game design. If you design your game so that there is no player trading, you've effectively cut off the farming. But not being able to sell anything to other players has a problem: it isn't fun. The trick is striking a balance, but we haven't seen it yet.

If you are interested in how MMOs work and think about hacking software, this is the book for you. If you are a programmer in this industry, this book is definitely for you. Go buy it!

A Decade of Ultima Online, Part 2

Since I have UO on the brain now, I think I'll list off some of my best memories both as a player and as a developer.

Player

• My first character: a smithy/miner. This was back near launch in Oct 1997. Mining was tough because of player killers (PKs) and you always had more ore than you could carry, so you would have to set it on the ground and move it a tile at a time. Pack animals didn't exist yet so your precious ore was always in danger. Players actually paid me to fix their armor.
• The first time I (successfully) went from a city to another city. When the game was very new the only map you had was a little mini-map in the game and the cloth map that came with it. Tools like UO Auto-Map didn't exist yet, so getting lost was the nature of exploring.
• Dying to forest creatures. Rabbits and deer were merciless to new characters.
• Sailing around the world. When I came back after a few years away, the first thing I did was buy a boat and sail all over. It provided hours of enjoyment exploring the world, with nothing to show except to be able to say I had been there.
• Joining a guild. While I was wandering in the dangerous non-consentual-PVP world of Felucca, I met a nice guy named Alydar. We chatted for hours while killing creatures and discovered that we were both programmers. I had started writing some tools for UO and he was interested in learning the same things that I was discovering. He invited me to join his guild, the Defenders of Virtue. He became my mentor and later sent me on a quest to discover the Virtues in order to become a Knight within the guild.
• Animal Taming. I tamed white wolves for hours to get my Animal Taming skill up, but it was all worth it the first time I tamed a dragon. Stealing the not-so-original name Trogdor, my dragon and I could solo Ancient Wyrms and Balrons.
• Age of Shadows. With the help of my pal Alydar, I was able to obtain a house plot inside the brand new city of Luna in the new continent of Malas. I still own that house to this day and use it as a vendor location, allowing other players to set up vendors at my house. This expansion also launched with a critical bandwidth usage problem that eventually got me a job working on UO.

Developer

• Fixing the bandwidth problem mentioned above in less than two weeks on the job. It was actually a very simple problem. The Age of Shadows expansion introduced real item tooltips to UO for the first time. Tooltips were always the same for everyone and had a global 'version'. Whenever something would change the tooltip was invalidated and the new version number would be sent out to everyone in range. This would cause all of the clients to request the new version. However, the problem was that tooltips were being invalidated whenever anyone new would come into range of the item. Instead of sending the current version to just the new in-range player, it would spam everyone in range when someone new came close. In busy areas the net traffic got ridiculous.
• The Day the Servers (almost) Died. UO's player backup format is very prone to bloating the backup file with empty space. Someone made a change to the way player characters were stored that broke the compression utility that removed said empty space. Right before Christmas. Everyone was gone right before Christmas break started, and player backups on production servers were quickly approaching their 4GB limit, above which Bad Things^TM happen. I had to make a very timely change to the compression utility and get the fix out to all the servers without dropping them by myself after having been on the job for about a month.
  
• Fixing the 'insurance bug'. Targeting in UO is request/response. The server tells the client that it needs a target. The client then changes the cursor to a targeting cursor and sends a response back to the server when the player picks something. The server then fires a callback on the item that requested the target. The problem was that the item was determined from information in the target response, so the server was stupidly trusting the client in this respect. In most cases, the scripts that controlled targeting had their own "security" but it was up to the developer to script it in every case. In the case of insurance, the 'item' the callback fired on was the player requesting to insure/uninsure something and there was no security. Now the security of all request/response formats is ensured by the server at a global level.
• Character Transfer system. I was pulled in to the character transfer team to wrap it up with my good friends Deathwish, AviStetto and MrTact. It was the last major feature that we did before Origin was dismantled and it was a lot of fun.
  
• Moving to Redwood City. Although I miss Origin, moving to CA with EA has been one of the best things to happen to me, even though I really didn't enjoy working at EA nearly as much as Origin.
• Fixing the 'chunk egg crash'. Each 8x8 tile section ("chunk") of the UO world has an invisible item called a 'chunk egg'. These are immovable (even by GMs) items that store all of the resources for the chunk (wood, metal, etc) that can be harvested by players. However, periodically these chunk eggs could end up inexplicably inside a moving crate in someone's house. The server would crash when the players would try to re-organize their house. The workaround was for a developer to go and remove everything from the moving crate, lock it down in their house and then delete the moving crate. Oddly and seemingly unrelated, people were also complaining about items disappearing from their houses. It turns out that the two problems were related. Object IDs in UO are 32-bit numbers that can be reused and the reference to the moving crate wasn't cleared when a player cleaned it out. This would leave a 'dangling reference' which was normally invalid. Sometimes the ID referenced would be reused to be a container in someone else's house or a chunk egg. Summoning the moving crate would pull that container from someone else's house or crash the server due to attempting to summon a chunk egg.
• Samurai Empire expansion. I did most of the coding for getting in the new wearables. I also did the a lot of the scripting side of the new Bushido and Ninjitsu skills. I did a proof-of-concept for the roof tile system which was later turned over to an intern. I also did a proof-of-concept for a Unreal Tournament-style PvP battlefield system that was rejected (and continues to be a bit of sore spot with the point of view of management).
• Unannounced game revamp. This isn't a 'great memory' so I don't know why I'm mentioning it. Yes, we were working on UO2 for the third time. We had a 3D client and were doing awesome things with breaking the skills into specialization and quest-based learning. The server technology was mostly staying the same, but we had finer-granularity movement actually working that allowed you to move within a tile. The scope of the project grew huge and we didn't have time to finish it nor wherewithal to cut part of it. Everything was scrapped and the idea eventually became Kingdom Reborn.
  
• Mondain's Legacy expansion. Mostly support work here. I supported the rest of the engineering team and picked up a lot of the "smaller" tasks like the world collections (museum, zoo, etc) and parrots. I also worked on an asset replacement system that was supposed to help our artists be able to test out new content in the game engine faster. I also helped MrTact and EvilMantis design the guts of the Quest system. I left EA before this expansion went live.
• Wombat. For some reason, I had great love for this proprietary scripting language. I took it under my wing and made some major improvements to it:
• An integrated debugger that allowed breakpoints, stepping and printing out variables.
• The ability to do negative numbers. Yeah. Really.
• Floating point support.
• Compiled binary support. Greatly reduced server start up times.
• A simple optimizer.
• Support for array and dictionary container types. Previously the only type was linked-list.
• Global variable support (saved on memory usage)
• Reference-counted containers. Saved time when large amounts of data were thrown around. Previously everything was copied. A lot.

Good times!

A Decade of Ultima Online

Recently, Ultima Online celebrated its Tenth Anniversary. UO will always hold a special place in my heart because it was my first MMO, both as a player and as a developer.

As part of the celebration, IGN has an article and speedman (my dev name on UO) got a mention.

I can't explain why, but I always look at my memories in UO through rose-colored glasses. I remember playing back in 1997 and getting PK'd all the time as I mined ore (which was always gone due to the now-defunct resource bank system). I took some time off for a few years but eventually found my way back. A lot had changed in my time away, but I found a good group of friends to help me through the transition.

UO was a sandbox in many ways. Origin Systems provided the world but the players were the content. There were no predefined quests, no raid encounters and no leveling. There were only a handful of skills and spells and a wide-open world. When I wanted to gain a higher rank in the guild (which wasn't defined in the game mechanics), my mentor sent me on a quest (again, not defined in the game mechanics) which I had to document. It was the most unique experience I've ever had in an MMO and I passed it down to other guildmates when I became their mentors. As a developer, I also had the opportunity to chronicle it for the players. The story of my guild quest is now available as two in-game books written by Autenil.

UO was also my break into the game industry. I was a bit of a hacker at the time and wrote a few utilities for UO. I never distributed them to anyone but my closest friends; it was mostly a learning experience. The crown of my collection was a DLL that would attach itself to the UO client and re-write part of the software to pass network messages to my applications after the client had decrypted them. All of my tools were centered around this hook DLL. I had message loggers, information sniffers, and a few useful tools. Yes, I even had tools that would make it easier to gain skills by automating some tedious tasks (my animal taming tool would keep retrying until something was tamed, rename it so that you wouldn't tame it again, and release it).

Anyways, the company I worked for at the time sent me on a business trip to Austin, TX which also just happened to be the city of Origin Systems, the very headquarters of UO. Turns out they were looking for a server programmer and I had passed the tech test with flying colors. Since I was going to be in Austin anyway, I arranged a meeting with the UO dev team. Over three hours at Cheesecake Factory they grilled me about programming and MMOs (some of my old friends might remember that night). They were having some bandwidth problems with their latest expansion and I (due to my tools) knew what the main problem was. I told them that I would fix the bandwidth problem within two weeks if hired. They hired me and I fixed the problem within the first few days. Within two weeks, it was live and players were rejoicing.

The development environment that UO required was challenging. The servers would only build and run on Linux and we used GDB for our debugging. All programmers and designers had and maintained their own Linux boxes. Getting a server up and running required a spider web of filesystem links and a delicate matrix of configuration files and scripts (of several different formats). Still, I loved the proprietary script language known as Wombat and the badly-written C++ code.

I don't work on or really even play UO anymore, and I don't always (ever?) agree with what Electronic Arts/Mythic does with the game, but I still think about the game (and the friends I made working on it and playing it) often.

Congratulations, UO. Here's to another decade. At least.

Hackalicious

The more I beta-test/play/develop MMOs, the more I realize that developers still cut corners when it comes to security. If you're working on an MMO, especially before launch, it's very important to take a look at your game through the eyes of a hacker.

I've worked on hardening some games to the hacker types (and let's admit it: I was one myself once), so here's my short primer on MMOs and hacking:

Don't ship your game with the .PDB file! Obvious? Maybe, but not to some.

Turn off RTTI. Why are you using dynamic_cast anyways? It supports crazy things like casting horizontally in multiple inheritance. Not to mention that it's slow. Hackers love RTTI though, as it causes the executable to include mangled names of pretty much everything. If you need a dynamic typesafe casting solution, search the web or get a book. Or use virtual functions, though it's ugly as sin.

Take a look at your client in a hex editor, especially strings. You might be surprised at what you find. Those assert lines can actually contain a ton of useful information, like source file names and line numbers, not to mention C++ code itself. Or you might just find that you're including the class names of every message type in the game. Yes, it's been done.

Beware large areas of [un]initialized memory. Hackers love it when they already have space allocated in your program that they are free to write their code into. UO had a bunch of these before I cleaned them up, the largest of which being a 640x480 graphics buffer that was never fully used. Use the heap and make it lazy allocate. If possible, delete it as soon as you're done with it.

Change message IDs frequently. If you can do it right (and unpredictably), this can work well to foil those who like to read the network traffic.

Log stuff to the server, but don't take action on it right away. You don't want to do anything that lets a hacker know that you're onto them. Check with your legal department to find out what you can report, but it generally shouldn't be bad to do something like CRC the client in memory and send back a pass/fail response to the server. Or report if they're using a debugger, though this can be challenging. Unfortunately, uploading their modifications for review is anathema.

Watch your encryption keys. Yes, EQ2 actually held the encryption key in plaintext in a static buffer after negotiating it. It's fixed now, but hackers rejoiced!

Verify user data. UO has an immediate request/response targeting system. The server sends a request to the client with an ID which the client returns unmolested to the server. But what happens when a hacker figures out how to make use of changing that ID? You get the insurance bug.

Beware of unreleased/undocumented content. Again with UO, there were quite a few "gumps" that were in half-implemented features that still got pushed to live servers. A hacker figured out how to make use of one of these and ended up teleporting half the players on the server to a single X,Y location. Use source control branches or some configuration system to make sure this untested stuff can't be used. Period.

Don't make assumptions. You've heard the clichés so i'll spare you. Just because a normal user can't see something (like a UI window or network message format) doesn't mean they can't exploit it.

Movement. A notoriously bad topic for MMOs, especially 3D ones, there should really be more seriousness here, at least in the way of logging. Servers can at least sanity check movement without having to run simulations of every client.

Buffer overruns/remote code execution. Yes, it can happen to you. Watch how you're using the string functions, especially the varargs ones (sprintf, sscanf, etc) and wide functions. The discussion on this one topic alone could be (and has been) a lecture in itself, so I'll just say: be aware of it.

You can never fully stop people from hacking on your game, but you can make it difficult (and you should). Then there's the other side of the coin: banning the people you catch hacking. Usually this is financially advantageous: if a hacker is turning people off of your game, it's probably not just one.

Unfortunately, maintaining hack resistance and detection can be a full-time job, especially if your game is well established and there are many hack programs out for it. In such cases, it might be better to look at using something like PunkBuster.

This is definitely an area of MMOs that should be collaborated on, since all MMO developers don't want the hackers :)