07 March 2008

Memory Usage and the EverQuest II Client

3D graphics tend to use a good amount of memory which increases with their level of detail, texture size, complexity of geometry, etc. Combined with sound, scenegraphs and other things that make up a game client and you can use a lot of memory.

The EQ2 client is pretty memory hungry, but we're making strides in changing how that works a little bit. Currently, we have several reports of issues with Vista. While it's not technically a supported OS for EQ2, we still want to ensure that people can play the game. Unfortunately, either Vista's DX9 compatibility or Vista drivers seem to be using a LOT more memory then they did on Windows XP.

To mitigate this problem, there's a few things we're doing.
  • /3gb switch and 64-bit support - Most Windows applications are not aware of large addresses (i.e. memory addresses over 2GB). There is a linker switch that tells Windows that your application can use addresses over 2GB. This has a dual benefit: for users on 64-bit Windows, the EQ2 client will have access to nearly the entire 4GB address space. For users on 32-bit Windows, there is a switch that can be added to your boot.ini file that tells Windows to allow up to 3GB user address space. While this change may prevent out-of-memory crashes, they don't solve the original problem that the client uses too much memory.
  • Texture Downsampling - This system picks textures that are far enough away and unloads the highest quality mip level. This can lead to significant memory savings as well as saving texture memory on the video card, without significantly degrading the quality of the graphics.
  • Non-drawn geometry unloading - Since the game loads assets basically within a radius of the camera, sometimes things get loaded but are occluded and used rarely. Things that haven't been drawn in a long time don't need to be kept in memory and can be unloaded. This saves both system memory and video memory since we don't need to hold onto textures and vertex/index buffers that aren't being used. Generally, the performance cost is negligible. Loading things back in has some cost, but assets have a better chance of being in the disk cache greatly reducing the load time.

The systems mentioned above also won't come into play until you hit a certain amount of memory in use. There are other things that we're looking at too, but the changes mentioned above appear to have the most bang for the buck.


MrTact said...

Vista mirrors your video memory into system memory, so video drivers can run in user space. So if you're running a 512 MB video card, your 2 GB system is now down to 1.5 GB.

Which will be fine, when 8GB is standard on shipping systems. Today, however, it is completely sucktastical.

Joshua said...

It's true that Vista uses system memory as a backup to video memory, but it appears to do it twice for older versions of DirectX. If you read KB940105, Microsoft attempts to address the problem by making non-lockable resources not use additional system memory. Unfortunately, this doesn't help much as most resources in games are in the 'managed' pool which generally means that they can be locked. Managed also means that DirectX takes care of determining what should be on the card and can handle overcommitments of video ram. Nonetheless, we check for and recommend that the patch referred to by this KB article is applied when running the EQ2 client.

In DX9, resources can also be put into the 'default' pool which is usually video ram or AGP memory only (and is required for things like render targets). There is no (or very little) system memory overhead for these items but there are a few caveats: You have to rebuild the items if the device is lost/reset and there is nothing stopping you from running out of video memory on lower-end cards. Apparently DX10 doesn't have this kind of restriction due to WDDM and automagically handling device lost/reset (and the ridiculous amounts of system ram that they use to keep track of everything).

If they are keeping multiple copies in system ram, that really just seems ridiculous and it's the reason we tend to be blowing our address space.

I haven't found anything yet on whether those caveats are cleared up for 'default' pool allocations on DX9 under Vista and I don't yet have a Vista development box to test on. If those caveats were removed under Vista, I'd invest the time to put things in the default pool.

We're also stuck on DX9 for the time being due to a bunch of version 1.1 asm pixel shader code and GeForce 3 support. Porting to DX10 would require rewriting ALL of our vertex and pixel shaders (alot) while maintaining backwards compatibility for the GF3. Plus, seeing as DX10 runs only on Vista, we'd have to maintain DX9 compatibility regardless.

Toldain said...

I hope this isn't abusing the privilege of posting on your blog, but I have a question.

When I right-click for a contextual popup menu, sometimes my frame updating will halt completely for several seconds until the menu is painted. Mostly it's a minor annoyance, but I've trained myself to avoid it during critical moments.

I've been guessing that this is due to a context switch kind of issue, e.g., memory. Does that seem right?

Mindeen said...

Have you tried doing a similar thing with textures? eg. Completely unloading textures that haven't been drawn in the last x seconds and reloading as needed? As textures are needed again you could load a poor quality 32x32 version and when I/O allowed load in the mipmaped version.

Illuminator said...

What ruins the visual experience for us isn't the memory usage, it's the frame rate hit from the software rendering. Sooner or later you guys just gotta run into that code with C-4 strapped to your chest, you can't hold out and hide forever.

It's unreasonable for anyone absolutely insist on DX10 for this, DX9 is fine.

Joshua said...

Toldain: The right-click contextual menu is probably pausing to load resources for the context menu. I've noticed this too and I'll add it to my list of things to look at.

Mindeen: Objects that aren't drawn in a while that get unloaded cause ALL resources (that aren't shared and in use by something that needs them) to be unloaded. So if there's a table that hasn't drawn in a while and it gets unloaded, we remove meshes, collision meshes, textures, shaders, even relevant sounds, etc. Usually loading things back in isn't an issue due to the OS's in-memory disk cache.

Illuminator: We make full use of the programmable pipeline, but some things (for legacy reasons) run on the CPU rather than the GPU. These include animations, particles and shadows. We're looking for a graphics programmer (and have been for some time) to look into some of these issues and add new features to our graphics engine.

Toldain said...

Thanks, Joshua.

Mindeen said...

I think nobody has applied to the job because they would have to rewrite the entire graphics client and all your shaders to get everything onto the gpu ;)

To be honest though I think you have the fastest and best looking graphics client out there running ver 1.1 shaders.

Are you doing the dual pass routine (depth pass) for your shadows? I haven't worked with graphics in a while but I heard nvidia had some shadow routines built onto their cards newer cards(NVshadow or something), seems like something that could be a quick fix for speed without hiring a new person to do it.

Joshua said...

Thanks for the praise :)

While we currently do run mostly in shader 1.1 I would like to add support for at least shader 2.0 and HLSL. Need that Graphics Programmer though :)

EQ2 uses shadow volumes, but the degenerate triangles have to be calculated on the CPU as geometry shaders aren't supported until DX10/shader 4. The NVshadow that you refer to might be some kind of OpenGL extension, but unfortunately we use DX9 instead of OpenGL.

I'd like to tackle some performance issues with flora, shadows and animation which is why I need a Graphics Programmer. :) We're making progress in that area though.