A Decade of Ultima Online, Part 2

Since I have UO on the brain now, I think I'll list off some of my best memories both as a player and as a developer.

Player

• My first character: a smithy/miner. This was back near launch in Oct 1997. Mining was tough because of player killers (PKs) and you always had more ore than you could carry, so you would have to set it on the ground and move it a tile at a time. Pack animals didn't exist yet so your precious ore was always in danger. Players actually paid me to fix their armor.
• The first time I (successfully) went from a city to another city. When the game was very new the only map you had was a little mini-map in the game and the cloth map that came with it. Tools like UO Auto-Map didn't exist yet, so getting lost was the nature of exploring.
• Dying to forest creatures. Rabbits and deer were merciless to new characters.
• Sailing around the world. When I came back after a few years away, the first thing I did was buy a boat and sail all over. It provided hours of enjoyment exploring the world, with nothing to show except to be able to say I had been there.
• Joining a guild. While I was wandering in the dangerous non-consentual-PVP world of Felucca, I met a nice guy named Alydar. We chatted for hours while killing creatures and discovered that we were both programmers. I had started writing some tools for UO and he was interested in learning the same things that I was discovering. He invited me to join his guild, the Defenders of Virtue. He became my mentor and later sent me on a quest to discover the Virtues in order to become a Knight within the guild.
• Animal Taming. I tamed white wolves for hours to get my Animal Taming skill up, but it was all worth it the first time I tamed a dragon. Stealing the not-so-original name Trogdor, my dragon and I could solo Ancient Wyrms and Balrons.
• Age of Shadows. With the help of my pal Alydar, I was able to obtain a house plot inside the brand new city of Luna in the new continent of Malas. I still own that house to this day and use it as a vendor location, allowing other players to set up vendors at my house. This expansion also launched with a critical bandwidth usage problem that eventually got me a job working on UO.

Developer

• Fixing the bandwidth problem mentioned above in less than two weeks on the job. It was actually a very simple problem. The Age of Shadows expansion introduced real item tooltips to UO for the first time. Tooltips were always the same for everyone and had a global 'version'. Whenever something would change the tooltip was invalidated and the new version number would be sent out to everyone in range. This would cause all of the clients to request the new version. However, the problem was that tooltips were being invalidated whenever anyone new would come into range of the item. Instead of sending the current version to just the new in-range player, it would spam everyone in range when someone new came close. In busy areas the net traffic got ridiculous.
• The Day the Servers (almost) Died. UO's player backup format is very prone to bloating the backup file with empty space. Someone made a change to the way player characters were stored that broke the compression utility that removed said empty space. Right before Christmas. Everyone was gone right before Christmas break started, and player backups on production servers were quickly approaching their 4GB limit, above which Bad Things^TM happen. I had to make a very timely change to the compression utility and get the fix out to all the servers without dropping them by myself after having been on the job for about a month.
  
• Fixing the 'insurance bug'. Targeting in UO is request/response. The server tells the client that it needs a target. The client then changes the cursor to a targeting cursor and sends a response back to the server when the player picks something. The server then fires a callback on the item that requested the target. The problem was that the item was determined from information in the target response, so the server was stupidly trusting the client in this respect. In most cases, the scripts that controlled targeting had their own "security" but it was up to the developer to script it in every case. In the case of insurance, the 'item' the callback fired on was the player requesting to insure/uninsure something and there was no security. Now the security of all request/response formats is ensured by the server at a global level.
• Character Transfer system. I was pulled in to the character transfer team to wrap it up with my good friends Deathwish, AviStetto and MrTact. It was the last major feature that we did before Origin was dismantled and it was a lot of fun.
  
• Moving to Redwood City. Although I miss Origin, moving to CA with EA has been one of the best things to happen to me, even though I really didn't enjoy working at EA nearly as much as Origin.
• Fixing the 'chunk egg crash'. Each 8x8 tile section ("chunk") of the UO world has an invisible item called a 'chunk egg'. These are immovable (even by GMs) items that store all of the resources for the chunk (wood, metal, etc) that can be harvested by players. However, periodically these chunk eggs could end up inexplicably inside a moving crate in someone's house. The server would crash when the players would try to re-organize their house. The workaround was for a developer to go and remove everything from the moving crate, lock it down in their house and then delete the moving crate. Oddly and seemingly unrelated, people were also complaining about items disappearing from their houses. It turns out that the two problems were related. Object IDs in UO are 32-bit numbers that can be reused and the reference to the moving crate wasn't cleared when a player cleaned it out. This would leave a 'dangling reference' which was normally invalid. Sometimes the ID referenced would be reused to be a container in someone else's house or a chunk egg. Summoning the moving crate would pull that container from someone else's house or crash the server due to attempting to summon a chunk egg.
• Samurai Empire expansion. I did most of the coding for getting in the new wearables. I also did the a lot of the scripting side of the new Bushido and Ninjitsu skills. I did a proof-of-concept for the roof tile system which was later turned over to an intern. I also did a proof-of-concept for a Unreal Tournament-style PvP battlefield system that was rejected (and continues to be a bit of sore spot with the point of view of management).
• Unannounced game revamp. This isn't a 'great memory' so I don't know why I'm mentioning it. Yes, we were working on UO2 for the third time. We had a 3D client and were doing awesome things with breaking the skills into specialization and quest-based learning. The server technology was mostly staying the same, but we had finer-granularity movement actually working that allowed you to move within a tile. The scope of the project grew huge and we didn't have time to finish it nor wherewithal to cut part of it. Everything was scrapped and the idea eventually became Kingdom Reborn.
  
• Mondain's Legacy expansion. Mostly support work here. I supported the rest of the engineering team and picked up a lot of the "smaller" tasks like the world collections (museum, zoo, etc) and parrots. I also worked on an asset replacement system that was supposed to help our artists be able to test out new content in the game engine faster. I also helped MrTact and EvilMantis design the guts of the Quest system. I left EA before this expansion went live.
• Wombat. For some reason, I had great love for this proprietary scripting language. I took it under my wing and made some major improvements to it:
• An integrated debugger that allowed breakpoints, stepping and printing out variables.
• The ability to do negative numbers. Yeah. Really.
• Floating point support.
• Compiled binary support. Greatly reduced server start up times.
• A simple optimizer.
• Support for array and dictionary container types. Previously the only type was linked-list.
• Global variable support (saved on memory usage)
• Reference-counted containers. Saved time when large amounts of data were thrown around. Previously everything was copied. A lot.

Good times!

A Decade of Ultima Online

Recently, Ultima Online celebrated its Tenth Anniversary. UO will always hold a special place in my heart because it was my first MMO, both as a player and as a developer.

As part of the celebration, IGN has an article and speedman (my dev name on UO) got a mention.

I can't explain why, but I always look at my memories in UO through rose-colored glasses. I remember playing back in 1997 and getting PK'd all the time as I mined ore (which was always gone due to the now-defunct resource bank system). I took some time off for a few years but eventually found my way back. A lot had changed in my time away, but I found a good group of friends to help me through the transition.

UO was a sandbox in many ways. Origin Systems provided the world but the players were the content. There were no predefined quests, no raid encounters and no leveling. There were only a handful of skills and spells and a wide-open world. When I wanted to gain a higher rank in the guild (which wasn't defined in the game mechanics), my mentor sent me on a quest (again, not defined in the game mechanics) which I had to document. It was the most unique experience I've ever had in an MMO and I passed it down to other guildmates when I became their mentors. As a developer, I also had the opportunity to chronicle it for the players. The story of my guild quest is now available as two in-game books written by Autenil.

UO was also my break into the game industry. I was a bit of a hacker at the time and wrote a few utilities for UO. I never distributed them to anyone but my closest friends; it was mostly a learning experience. The crown of my collection was a DLL that would attach itself to the UO client and re-write part of the software to pass network messages to my applications after the client had decrypted them. All of my tools were centered around this hook DLL. I had message loggers, information sniffers, and a few useful tools. Yes, I even had tools that would make it easier to gain skills by automating some tedious tasks (my animal taming tool would keep retrying until something was tamed, rename it so that you wouldn't tame it again, and release it).

Anyways, the company I worked for at the time sent me on a business trip to Austin, TX which also just happened to be the city of Origin Systems, the very headquarters of UO. Turns out they were looking for a server programmer and I had passed the tech test with flying colors. Since I was going to be in Austin anyway, I arranged a meeting with the UO dev team. Over three hours at Cheesecake Factory they grilled me about programming and MMOs (some of my old friends might remember that night). They were having some bandwidth problems with their latest expansion and I (due to my tools) knew what the main problem was. I told them that I would fix the bandwidth problem within two weeks if hired. They hired me and I fixed the problem within the first few days. Within two weeks, it was live and players were rejoicing.

The development environment that UO required was challenging. The servers would only build and run on Linux and we used GDB for our debugging. All programmers and designers had and maintained their own Linux boxes. Getting a server up and running required a spider web of filesystem links and a delicate matrix of configuration files and scripts (of several different formats). Still, I loved the proprietary script language known as Wombat and the badly-written C++ code.

I don't work on or really even play UO anymore, and I don't always (ever?) agree with what Electronic Arts/Mythic does with the game, but I still think about the game (and the friends I made working on it and playing it) often.

Congratulations, UO. Here's to another decade. At least.

Kunark Rising

I've really been enjoying the new expansion. As a developer, playing it while under development really isn't "fun": you're looking for bugs, only a few people are in the zones, you're trying to take notes and not really enjoying the content. That all changes after the expansion launches.

Some things that I'm really loving:

• Zone Art - I'm only level 69, so I've been hanging out in Kylong Plains. The art here is amazing. The spires at Teren's Grasp are amazing and ridiculously tall and the transition from the warmer lowlands to the frigid area is great. Plus, I can see my character's breath in the snow. The new environmental system that we put in is really making a great debut in the intra-zone transitions as well as POIs like Karnor's Castle.
• Increased Quest XP - I love quests more than level grinding. The new quests are fantastic and have great rewards. The Quest Designers are using the relatively new item-association system that I put in and it's helping in some very subtle but effective ways. Also, this is a great time for me to brag about my Quest Sharing system. I've been adventuring with some pick-up groups and sharing quests has REALLY helped out.
• Smooth Launch - I talked about this a bit in my previous post, but this launch had to have been one of the smoothest EQ2 launches we've had. The team was in great form and we really cranked on getting the last few issues wrapped up before launch.

Some programming-related things that I (and others on the team) have noticed and we're looking into:

• Memory Usage - Those big zones aren't cheap memory wise. I'm currently looking into new ways to save memory.
• Quest Sorting - If you're in one of the additional instances of the big zones your quest journal isn't showing quests for the current zone at the top.
• DB Issues - Some of our larger servers have been under additional stress from the popularity of Kunark. The team is working hard to resolve these issues.

Happy Kunarking

This! Is! Kunark!

We on the EQ2 team are launching our Rise of Kunark expansion today. I'll periodically edit this post with updates. If all goes well, this should be a very boring read. :)

8am - Gah, too early. They brought in breakfast which is greatly appreciated. We also just received our Rise of Kunark boxes and I registered my RoK keys for the live servers. Servers are receiving the RoK publish package and everyone is trying to patch.

10:30am - I finally got patched up. Servers are still locked (and will be for a while while QA continues to get patched up and runs tests). Besides some slowness with patching, there's no problems to report yet. Back to work trying to get the client to use less memory!

Noon - The community people have come by to take my picture a couple times. Maybe I'll be famous! Or not. They'll probably post a picture on eq2players.com or something. Due to overloaded patchers, we had to extend the downtime to 2 pm.

1:30pm - It's getting close. A quick check of the IRC channels show that the unlock is imminent. QA appears to have signed off and the operations group is preparing to let the players in. Now the real fun starts. We'll watch the performance and hope that all of our hard work payed off.

2:15pm - Everything is going well. Servers have been live and unlocked for 45 minutes now and everything looks great. I'm on-call for any emergencies that might crop up, but everything looks great for now!

Turns out they wanted pics for an interview. In French.

'Twas the Night before Kunark...

Most days I feel incredibly blessed to be in the job that I'm in. Programming, working on (and playing) games, all while makin' bucks has just been a lot of fun. SOE is a great company to work for, especially with how family-friendly they've been.

We're launching Rise of Kunark tomorrow and it's bound to be an interesting day. I'm part of the launch support team, so I'll be in there bright and early at 8am tomorrow morning. Hopefully tomorrow I'll post a play-by-play of how an EQ2 expansion launch looks through my eyes. In the meantime, here's some pics of where I'll be spending at least 8 hours tomorrow: