Twitter  YouTube  E-Mail  RSS
The One Man MMO Project
The story of a lone developer's quest to build an online world :: MMO programming, design, and industry commentary
That Should Take a Day
By Robert Basler on 2019-05-21 16:59:29
Homepage: email:one at onemanmmo dot com

There is no level editor for Miranda. The terrain is built in a tool called WorldMachine and then all the textures, plants, rocks and everything you see in the game are added procedurally based on a bunch of text files. So when I added all the vendors and monoliths to the game I randomly assigned them locations, but I didn't have a way to check that every single one was in a good location (not on the side of a cliff.)

Monday morning I decided I wanted to address that bug. I figured I'd need to add keyboard controls to visit all the discoverables, and some debug prints so I could figure out which discoverable to modify in the text files, then it would take a couple hours to move them all, then another hour to verify after the moves. That should take a day.

[A Vendor in her new home.]

I started teleporting to them, I'd wait for each one to load, then look at it, choose a new location by clicking there then copying the coordinates into the text files. Everything was fine, then I hit an Independent vendor. I was playing as Empire so I couldn't open the vendor, and the debug output when I clicked on it didn't tell me which vendor it was. I also realized that I wasn't starting at the 0th discoverable, so I fixed that bug. And I added some additional debug text to tell which vendor it was.

Of course I didn't have any fancy navigation so I had to go through every one of them again, waiting for loading, trying to get to where I left off.

[My new desktop background.]

I got about 20 into the list of 140 or so items when the game crashed with an out of memory error. Hmm. It shouldn't be doing that. So I checked a bunch of things, they were all working as they were supposed to. I thought maybe I was just teleporting too often and the terrain system wasn't able to discard old terrain as fast as it needed to. I went through slower, waiting periodically to let the terrain system catch up. I got to 20 or so again and crash.

[The worst placed monolith I found. I couldn't even see it when I teleported in.]

I realized that my fix for visiting the 0th vendor was not going to work, so I reworked that again.

I knew that the out of memory problem could eat a week and turn out not to actually be a problem, so I switched to a release build with debug prints turned on and continued to check vendors and monoliths, since release builds use much less memory.

You should probably know that when I move a vendor or monolith, I need to rebuild the game data and restart the servers for them to take their new positions. For vendors that only takes 2 minutes, but for monoliths it takes about 12 minutes. And sometimes I'd forget I have to delete one particular cache file for the update to work (bug) and I would have to build the data for 12 minutes again. I had to do this several times throughout the week to check my work.

[This Vendor probably doesn't get a lot of walk-ins.]

Because I was teleporting repeatedly, I noticed that the "You are here" popup stayed visible when the loading screen appeared. I spent 15 minutes figuring out what controls that and making it so that it would disappear if the loading screen came up.

The release build worked better as expected, it was much faster and again I was happily moving vendors and monoliths. I did about 60 when the game told me HEAP: Free Heap block 609E74A0 modified at 609E7570 after it was freed. Ugh. Those are the worst. By the point the error is discovered, the memory has been wiped and there's really no telling what's doing the damage, you just know that something's broken in a particularly bad way.

So I turned on full memory debugging which records every allocation the game makes so that you can match an error address to what code allocated the block, and if you're lucky, get a clue as to what's doing the damage.

I went through those 60 discoverables and their loading screens again. Full memory debug makes the game pretty slow, even the optimized build. I never encountered that HEAP error again. However the memory tracker stopped the game, complaining that 1.2 million memory blocks was just too many. Well yeah. And then the game hung. It turns out that putting 1.2 million lines through OutputDebugString is a bad idea. I reworked the memory reports not to do that.

I restarted and went through the discoverables again, this time when the game stopped the debug output worked and showed that there were 850,000 colliders being tracked by the collision system. Colliders are rocks, trees, and player's tanks and buildings. 850,000 of them sounded like a lot. To check, I added code to count each type of collider. I figured it was rocks from the start since the number was so huge, but I had to be sure. Back through the 60 discoverables again, slowly still because all the debug tools were running, sure enough, only about 1000 of the collders were trees, tanks and buildings. I spent an hour hunting through the collision code until I discovered a cut and paste error where I was removing the rock colliders for an area with its top left corner at row/row instead of row/column. By far the largest category of bugs I make are cut and paste errors. After that fix the number of colliders in the game hovers around 30,000.

[I just think this looks nice.]

Ok, back to checking vendors and monoliths. I changed the start index temporarily to 60 so I didn't have to wait through all those ones I'd already done yet again. I did a few more, then I opened an Imperial vendor and the game crashed. There were apparently fewer of something on the current loop iteration than on the previous one and that was caught by an assert. The vendor data was being changed while the game was putting together the information for the UI. UI requests in Miranda are all asynchronous, so I added a lock around all the data used by the UI handler. I rebuilt, tested the new code to make sure it worked and all tolled, another hour gone.

I set the start index to 90 and continued. Then I noticed that the game was getting slower and slower, too slow to continue actually. That usually means memory fragmentation. I didn't have any way to examine memory fragmentation for the game, so in the evenings while I was watching TV I modified my memory dump analysis tool to make a simple report of free and allocated memory blocks for the game. The memory dump file for that was 10.6GB, so it is a little unweildy. I spent a couple evenings fiddling with that and it turns out that the Visual C++ DEBUG allocator wastes a TON of memory for small allocations. But the tool didn't really show memory fragmentation to be the problem. That's as far as I have gotten with that particular problem so far.

In the name of progress I continued to increase the start index, going through smaller sets of discoverables at a go until I succeeded in making it through all of the discoverables in the game. Then I rebuilt all the game data with the final positions and went through them again to verify. Somehow I had missed moving 3 of them at the 65 mark so I fixed those and went through the whole process again.

[The worst vendor location I discovered.]

Friday, 6pm, I checked in the changes to the vendor and monolith positions plus all the bugfixes.

If you're playing and you notice that you can't see vendors or monoliths you had previously discovered, search nearby and you should find them again.

New Comment

Cookie Warning

We were unable to retrieve our cookie from your web browser. If pressing F5 once to reload this page does not get rid of this message, please read this to learn more.

You will not be able to post until you resolve this problem.

Comment (You can use HTML, but please double-check web link URLs and HTML tags!)
Your Name
Homepage (optional, don't include http://)
Email (optional, but automatically spam protected so please do)
What color is an orange? (What's this?)

  Admin Log In

[Home] [Blog] [Video] [Shop] [Press Kit] [About]
Terms Of Use & Privacy Policy