Dr. Laura: Dr Jekyll and Mr. Video Gamer

Dr. Laura has a video blog about a man who plays video games for his down time:


Now, I actually enjoy reading most Dr. Laura articles. Her common-sense approach to most family psychology issues provides both entertainment and self-reflection.

However, sometimes she's just a little too old-fashioned for me. She seems to come off in this video as entirely anti-video-game. I'll grant that the guy mentioned in the video is acting like a complete idiot, but not all video games are bad. Consoles like the Wii consistently encourage family gameplay.

Maybe it's just because I'm in the video game industry, but if a guy wants to play video games for his downtime, that seems fine. Of course, you have to look at your priorities. Personally, family comes first. Always. I will usually sit down to play games after the rest of my family has turned in for the night. If the rest of the family is napping, occasionally my son and I will enjoy LEGO Star Wars or some other innocuous game.

Everything in moderation, but please don't demonize all video games because some idiot guy can't put down the controller and spend time with his family.

Keep Designer Data in Files (not Databases)

Whoa! Did I just fall off the deep end?! Designer Data in ... files?! Data... doesn't that belong in a database?! There are those who say that it does, but I disagree, at least when talking about development.

In MMO terms, Designer Data is data that our Design team creates: Quests, Spells, NPCs, Paths, Items, Recipes, Locations, etc. Tons and tons of data that, together with art, is really the lifeblood of the game.

Designer Data is to Designers as Source Code is to Programmers. It's constantly changing and growing. Different people own different parts of it. Everyone on the dev team needs all of it and as up-to-date as possible to do their current tasks. Servers need it in a highly-optimized form to run the game. There are many parallels between Designer Data and Source Code.

Which brings me to my first major point. As files, Designer Data can be stored in Source Control (we use Perforce, and I love it). This gives an amazing amount of incredibly important functionality and basically for free:

• Revision history - As text (EQII uses XML) files, anyone on the team can go back and see (through the same tool that everyone must use) all previous changes to a file. Anyone can see exactly what changed between each version or between multiple versions and who made the changes. Also, you can go back in time and grab previous file revisions.
  
• Changelists - Most source control systems group together file changes. Perforce calls these Changelists. Grouping files together is an effective tool for seeing relative changes based on concept. One changelist might be labeled "Zone 1 population" and contain NPCs, Quests that those NPCs give out, Paths that those NPCs follow, etc. If a Changelist breaks the game, it can be rolled back wholesale (and you can see who made it and go pound them).
  
• Integration - Changes are made in branches and we have branches for each distribution of the game. As a group of changes become ready to go Live, they are integrated into the next distribution (Main goes to QA, QA goes to Staged, Staged goes to Live). With Code and Data both in the same system, integration to the next distribution becomes a one-step process.
  
• Code/Design in sync - Since Source Code and Design Data are both stored in the same system, they are always in sync. If a Programmer makes changes that affect both (which often happens), he can rest assured that everyone will easily get the latest in both.
• Sandboxing - Designers are only affecting their own local version of design data until they check it in. They can do testing and make tweaks before checking in something that could potentially break everyone else.
• Blame - It's so easy to see who made a change (and confront them if necessary).
  

If you were doing Design development in a database, you would have to develop your own solution around each of these (or use database tools that not everyone would have access to). And since the database is always up-to-date, and everyone is generally working out of the same database, problems could arise from changing formats or unexpected data.

Furthermore, Designer Data in files are free to have complex formats. For instance, in EQII the Designer Data is object oriented. Data definitions specify that a Character inherits from Entity and Entity inherits from a Base type. This also allows us to do things like have a base Predicate type and more complicated types like CharacterHasQuestPredicate that inherits from Predicate but has additional data members that only make sense for CharacterHasQuestPredicate (like the Quest name). In a database, you might have a Predicate table with a type column that is a number. You would have to look up what that number means from somewhere or write special tools that understand the relationship. Additionally, that Predicate table would have to include all of the options that any types could have in a very generic fashion. So you might have columns called "StringParam1" and "NumParam1". In the database then, you might have a row with type set to 12 and StringParam1 set to a quest name (or more likely NumParam1 set to a unique ID for a particular Quest). If you were looking at raw data, what would you rather see:

<object name="CharacterHasQuestPredicate">
  <string field="sQuest">quests/heritage/dwarven_work_boots</string>
</object>

Or:

Type     IntParam1    IntParam2
12       1243         <null>

The EQII Designer Data tool is very generic as it knows how to read the game's data definition. To actually add a whole new data type (Achievement for instance) actually takes zero changes to the Designer Data tool. It just takes writing a small text file that describes the Data Definition such as this one for our aforementioned CharacterHasQuestPredicate:

<objectdef name="CharacterHasQuestPredicate" inherits="Predicate">
  <fields>
    <fielddef name="sQuest" type="String" require_dir="quests/" default="" />
  </fields>
</objectdef>

However, reading XML files isn't the fastest thing to do. Server startup is fairly slow when reading from XML files, but EQII has a utility that converts the myriad XML files into a single file that contains all of the data in a highly optimized binary format (I've actually written about this file before). This file is mapped into virtual memory allowing all server processes running on the same physical machine to share one copy of the file in memory. For development purposes, developers can run their own servers against the XML data (and both internal development servers and rapidly changed external servers such as a Beta server could run against XML as well for fast turn around). Plus, there are enough options for converting XML data to databases that you could still use XML files for development and run production servers against a database.

The only major shortcoming of keeping Designer Data in files as opposed to a database is for searching and updating large amounts of data--things that a database is designed to do.

For searching data, EQII actually has a very workable solution. The EQII team has developed a system called VooDLe (a name playing off of Google and VDL--the internal name for EQII's data library). This is a very simple web solution that syncs to the latest data and indexes it for searching. It also detects file references and generates links. However, it doesn't help if you'd want to find all of the shields that have a block value higher than 300. For this, you can search VooDLe for the field that declares shield block value and quickly scan through the results. (Rarely is searching for a numeric value so cut and dried in EQII though with level scalars and combat scalars that affect everything.)

Updating large amounts of data is actually something that (shouldn't) happen very often. Furthermore, updating large amounts of data is error-prone and should be limited. You might even call this an inherent benefit to having designer data in files. When this is necessary, it's possible with a simple Perl script. If something does go wrong (even later) the Changelist can easily be reverted.

Based on all this, I feel it's more beneficial to a development team to use files for Designer Data during development rather than a database.

EverQuesting

It has been a busy time for me. For the past few weeks I've been splitting my time at SOE between three different game projects.

Two of them are new projects that I can't talk about yet. I'm very excited about both of them (and think know you will be too!) and look forward to being able to talk more freely about them. One was a temporary post while waiting for the second to ramp up. Now that it has I will be transitioning to it full time.

The third is the project that I've been working on for over four years: EverQuest II. However, with the new project my involvement on EverQuest II is shrinking. My official title has moved away from Technical Director on EverQuest II to being the Technical Director on one of the new projects. Don't worry though: the EverQuest II programming team is in the hands of the very capable Greg "Rothgar" Spence. Plus I fully expect to be available to the EverQuest II team for questions and technical direction for a while to come.

I'm currently finishing up my last task with EverQuest II. This last hurrah is designed to do something very important--increase exposure to EverQuest II. I've been personally involved with nearly every change to EQII's trial program over the past four years, but this is the one I'm the most excited about.

EverQuest II is a very large game. Overwhelming content from the get-go followed by five (Soon™ to be six) expansions have conspired to create a client footprint of about 10GB. Yikes! That takes hours upon hours to download even on pretty fast connections. As my last task, I'm working on reducing the initial download down to about 60MB (which is barely a blip with broadband) and streaming the rest as the client needs it.

A few things make this possible. First, the fetching technology that we're using was first pioneered by the incredible talent on the Free Realms team. This made it much faster for us to get on-demand fetching up and running since they had already done the legwork and experimentation. Second, the fetches are nothing more than HTTP GETs which allows us to distribute all of EverQuest II's 500,000+ individual assets over a very fast CDN. Using a CDN means that we don't have to develop, test, distribute and manage our own file serving solution which obviously would take more time. Third, EverQuest II already has a fairly robust asynchronous resource-loading system that was fairly easy to extend with remote file fetching.

Getting asset fetching in place was really only the beginning of the story. My time since then has been focused on heuristics for prefetching the next assets the client will need and reducing some dependencies on synchronous IO (which would appear to "lock up" the client until the needed assets were fetched). These challenges are far more complicated, but the challenge is welcome!

It's been a great four+ years on EverQuest II, and I look forward to getting people playing after downloading a mere 60MB.

Interview of Yours Truly

I was recently interviewed (a month ago, heh) on SOE Podcast #58. The interview starts at about 1:19:44. Whew, long podcast!

SOE does a podcast about every two weeks and feature interviews, goofy ads and exciting news about current and upcoming SOE games. You can also follow SOE_Podcast on Twitter for news and notification.

Lost in Translation

EverQuest II currently runs in five languages: English, German, French, Japanese and Russian. However, all of the programmers and designers actually build the game in English. To accomplish the translation, we use a proprietary SOE library: the Text Translation and Templating Tool, or T4 for short. T4 is a very powerful tool that allows us to do translation through anything from dictionary lookups to macro expansion. Our Internationalization department (i18n for short) is responsible for maintaining the T4 library and the various language modules for it.

Unfortunately, sometimes we write things in a way that isn't easily language-agnostic. Take the following example for instance:

"Rothgar's Touch of Pestilence critically double attacks Autenil for 38 disease damage."

It's a common bit of text that appears in the combat chat when you're fighting in EverQuest II.

Let's break it down a bit.

First of all, there's the actor performing the action. This can be you or someone else; Rothgar in this example. There's also the target which can be you or someone else; Autenil in this example. There's an optional spell name (Touch of Pestilence) and how much of which kind of damage. Oh, and the action -- double attack in this case -- and actions may or may not be critical.

With the way T4 works with insertion macros, we use a base string like this:
"^1 $2 critically $3 $4 for +5 $6 damage"

But this really doesn't work, at least not for anything but English. That pesky verb needs context in other languages to be conjugated properly. We currently put in either "double attack" or "double attacks" based on who the source and target actors are. It works fine in English but comes out in other languages like gibberish.

The way to fix this is to build base strings that include the verbs, but this can explode quickly. Say there are five different verbs that could be used in that case. To make base strings that include the verbs with all of the other possibilities you end up with about 640 strings, and that's assuming that there is only one damage type (EQII allows multiple per attack).

In the end, I went with using a generic verb for non-English languages ("hit") and put special attacks or critical status parenthetically on the end:
Rothgar's Touch of Pestilence hits Autenil for 38 disease damage. (critical double attack)

This will be much easier on our translators and will make sense for our players. It's been a long time coming.

LOGIN 2009

I'm going to be speaking at the LOGIN 2009 conference in Seattle this year:

Unfortunately, client crashes are bound to occur during the course of a live game. While prevention is definitely preferable, how quickly they are detected and resolved helps to form a positive opinion of the development team in the players’ minds. This session is a technical discussion surrounding generation, retrieval and processing of client crash logging. Though many concepts presented are fairly generic and may apply to other platforms and languages, examples will be given in C++ on the Windows platform.

Additionally, one of my programmers is also speaking:

This lecture will be a technical discussion about the Bayesian spam filtering solution used by various SOE MMOs to block over half a million spams per day per game. The lecture will discuss SOE spam filter implementation including what can be done beyond pure Bayesian filtering to help MMOs increase their filters accuracy and reduce feedback to the spammers. We will cover the general theory behind the filter, as well as the mathematical theory behind it and real-world examples of how it is working today.

Be sure to stop by and visit our lectures if you're going to LOGIN 2009 this year!

Meme'd

Several people have tagged me with the "25 things" meme, so I figured I'd satisfy the adoring masses:

1. I’m a science fiction geek. My favorite Star Trek series is The Next Generation and I’ve seen every episode at least twice. I also like Star Wars and Return of the Jedi is finishing up on my TV as I write this. I’ve even gone as far as to draw up deck plans for starships… Snow Crash by Neal Stephenson is one of my favorite books.
2. My favorite genre of music is Country, much to the chagrin of my family and my co-workers. I do also like Classic Rock and some Oldies, so I can usually find something mutually pleasing to the people around me.
3. Tests show that I’m allergic to peanuts, but I’ve never had a problem eating them. My kids, however...
4. My family is very involved politically, and I have some strong political convictions, though I’m not very vocal about them... usually.
5. I would play Rock Band until my arms fell off if I was ever uninterrupted for a long enough period. I never am though (darnit).
6. I used to play first-chair alto saxophone in the high school band… while in elementary school. I’ve done solos and performances. I still have my sax and could probably play it, but not very well anymore.
7. I’ve driven 157mph. In a car. On the road. In real life.
8. I’ve lived completely on my own since I was 19, except for a few months living with an aunt and uncle while looking for a place after moving to CO.
9. I’ve slept in the bottom of the Grand Canyon and on 14,000 ft mountain peaks in Colorado.
10. I almost never dream while sleeping.
11. Unlike my dad, mom and brother, I don't wear glasses or contacts. My vision is 20/15.
12. I started programming computers in BASIC in the third or fourth grade. I got my first C compiler before graduating elementary school. I would design simple games with my brother and then write them. One early Christmas I painstakingly programmed several Christmas carols to play over the 3-voice music synthesizer on a TI-99/4a. My whole extended family was present for the concert.
13. I love cycling and enjoy commuting to work by bicycle when I have the time.
14. A friend and I nearly died climbing down cliffs while lost in the Gila Wilderness of New Mexico. My Ham Radio was useless in the mountains. We had to abandon our equipment and wade through a torrent of snow melt runoff. After finding our way to a highway, we had to hitchhike for 17 miles to get back to my car, which had a flat tire.
15. I’ve worked at McDonalds, CompUSA, an indoor minigolf place, and as a self-employed consultant.
16. My dad, mom, sister and brother all have a college or trade degree. I don’t, though not for lack of trying. I just kept getting hired away from school...
17. I’ve been to Mexico several times and Canada, but I’d really love to see parts of every other continent.
18. I’ve been skydiving and loved it. If I were independently wealthy and didn’t have kids, I’d make a hobby out of it.
19. My love of electronics goes hand-in-hand with my love of computers. In elementary school, I walked around with a 40-pin IC in my pocket. I etched my own circuit boards and built an adjustable power supply in junior high. I built a turbo timer and air/fuel ratio indicator for my Supra. I was the only kid I knew with an o-scope.
20. I’ve logged a few hours towards my private pilot’s license. I’d love to complete it despite the impracticality.
21. My faith in Jesus Christ has affected nearly every part of my life in some way.
22. When I proposed to my (now) wife, I arrived 3 hours before I was supposed to, snuck up behind her and surprised her while holding the open ring box. When she whirled around she knocked the ring out of my hand. I had to crawl around on the floor for a few minutes to find it. We’ve been married eight years as I write this.
23. I had lunch with Curt Schilling of the Boston Red Sox when he was interested in having me join the team at his game studio, 38 Studios. Good guy, but I respectfully declined.
    
24. I have a pencil stab wound in my hand that is still visible after 15 years.
    
25. I collect Ultimate Collector Series LEGOs. The models in my office at work total over 15,000 bricks alone.

UOForums Interview

UOForums asked me for an interview a while back. I finally obliged:

What has been your most memorable accomplishment in your career?

Making Technical Director before 30? :) Getting hired by Origin is probably the most memorable accomplishment as it was my first gig in the game industry. It was an amazing experience: I got to meet and work with the people behind the names on the forum that I idolized. The whole experience was surreal.

Tsk, tsk, Microsoft

This is awesome.

Thousands of Zune portable media players made by Microsoft Corp. suddenly froze up early Wednesday, Dec. 31, labeling Internet wits to label the phenomenon "Z2K."

It goes without saying that I'm quite happy with my iPod Touch and iPhone, neither of which locked up at 12:01am yesterday.