Post feature Report RSS Advanced Feature: Error Handling with C++ and Python

Ever since we started using python and outside C++ code to an extensive level in WiC, we've run into one problem we didn't foresee. Because the game engine is being pushed to an area where it was not intended during its original development, we've run into stability issues where game just flat out crashes from time to time.

Posted by on


Error Handling with C++ and Python


To stabilize the game application and prevent it from unnecessary code-related crashes as much as possible, I've been spending some time improving the python code that interacts with the game.

One of the features we were writing over the past weekend is called Exemptions Handling where a set of anticipated errors would be intercepted and raised as an "Exception Fault" to be handled before game could crash.

What kind of crashes?

Massive developers did a nice job in writing many Exemption Handlers in wic.exe, anticipating many potential crashes during their development time to stabilize the game and making it bug-free as possible.

However, when us crazy modders are doing all kinds of things that developers never thought about, it is inevitable that game engine is being pushed to its limit and someday will crash.

Dealing with Crash Conditions with Humor

To help stabilize the game, we started writing several basic exemption handling facilities in our PredictorFCS basic abstraction layer scripts written in python, which interfaces with external code.

When a condition is detected that a potential crash may occur, we raise an Exception Fault to be handled locally before the game engine ends up dealing with it. When exception fault is raised, a "handler" script is run to "handle" the exception. Handling the exception could come in variety of ways, the most common way is to write an error message to the screen or to the debug output log, or quit the user's game session back to main menu, or quit the game itself.

We've decided to handle known common exceptions with humorous fashion as an easter egg :)

Exception Handling Examples

Dual-Stage SAM Code

The new dual-stage SAM code will operate the Patriot and S-300 missile batteries. The Dual-Stage SAM spawns controller unit at an optimal altitude and location on the map to intercept the air player. But sometimes, the SAM code "over-leads" the moving aircraft and ends up directing the controller unit to a location outside of the map boundary, known as "game box."

Now, Massive developers actually did think about outside-of-game-box unit spawn errors so the game does not crash. But, when the controller unit ends up being spawned outside of the map where there is inconsistent terrain mesh, game will instantly crash and burn to the desktop.

So, when our new IADS SAM code is operating the Patriot/S-300 batteries and is engaging the air player, we run several checks.

We check the LOS rate of the pursuing interceptor SAM in transit and make sure that it is not over-leading the air player to the point of running out of map boundary and leave the game box. If we detect that the missile may run outside the boundary, we raise an Exception Fault.

When Exception Fault is raised, the support player will get the following messages:

IADS GCU Exception Fault! (48N6E2_BASKET.XZ) (SAM will over-run map boundary box!)
IADS GCU SWITCH_MODE PDFCSG_Method.NAV_PPN (Switching to pure-pursuit homing to prevent map over-run.)

And we change the SAM's guidance mode from lead-collision homing to "pure-pursuit" tail-chase navigation to prevent it from over-running the boundary of the map and potentially crash the game.

If for some unknown reason, the SAM is about to run outside of map boundaries, even after we switch to pure-pursuit guidance mode, we then raise a second Exception with a standard assert() handling facility:

IADS GCU ASSERT Failed! (48N6E2_BASKET.XZ) (SAM will over-run map boundary box!)
IADS GCU TERMINATE PROTECTED_MEMORY (Self-Destruct Missile to Prevent Game Crash.
)

And the pursuing SAM immediately self-destructs and terminates. If we did not terminate the SAM, it will continue flying, run outside of game box, and out of luck, may end up landing at an inconsistent terrain mesh then crash the game.

Now what are the odds of these errors happening? Very low in most cases. But sometimes, they DO happen and we would prefer the pursuing SAM to commit suicide than crash the game :D

Auto-Pilot for Tomahawk Cruise Missiles

The Tomahawk missile in our new system now uses inertial navigation system and guidance logic where the code automatically takes over control of player's cruise missile unit, then flies it to the target and blows it up at the right moment.

We run several arithmetic calculations during the cruise missile's flight to properly guide it to its target, such as leading ahead of target's moving position, and also scanning ahead for buildings and terrain obstructions for the cruise missile to fly around them instead of getting stuck while flying.

While the cruise missile is flying to the target and crunching out the numbers in the background, often times, your team will lose Line of Sight (LOS) vision on the target. When LOS is lost, several things could occur because we just lost the target and the code that runs the missile is now baffled as to what to do.

So when this event occurs, we raise an Exception to "handle" and deal with this situation. The player gets the following message:

TLAM GCU TGT LOST (Target lost, switching to INS backup.)

With that, the cruise missile's auto-pilot script then falls back to inertial navigation mode, and flies the missile to the last known position of where the target was last seen. It then explodes the missile once it arrives to the area, regardless of whether that target is visible to your team's LOS or not. If the target never moved, the missile will kill the target, but if the target moved out, then nothing would have been hit.

"Sunk by Windows NT"

A more serious condition could occur when the Tomahawk auto-pilot script is crunching out the numbers while guiding the missile to its target. Because we do a lot of division in our calculations, sometimes the numbers we end up feeding may end up being well.. zero. This could create a situation where the calculation program may end up dividing by zero.

Division by zero is a serious business. While Massive developers made error-handling facility to deal with division by zero in game's math logic, it apparently can still crash the game when dealing with an outside code, and it will definitely crash multiplayer.

So, whenever we detect a potential divide-by-zero condition while crunching out the numbers for Tomahawk missile guidance, we raise an immediate priority Exception to be handled right away before anything else. The player receives the following message:

TLAM GCU Exception Fault! DIV_ZERO (Floating Point Division by Zero)
TLAM GCU TERMINATE PROTECTED_MEMORY (Self-Destruct Missile to Prevent Game Crash.)

And the Tomahawk cruise missile unit immediately blows itself up while flying in mid-air, before division-by-zero math calculation could be performed.

Again, does this happen a lot to affect gameplay? Not at all. But sometimes, it can happen, especially when the player interferes with missile guidance by selecting and issuing Move orders to it (just let it fly on its own). Let's just hope that Tomahawk missile does not run into Divide by Zero error while it was flying over your friendly forces. Because, it will blow up right on top of them. Whoops, talk about epic friendly fire.

We wanted to make sure that we cover these most basic errors internally within our own scripts, preemptively, before they could crash the game. And we prefer to be humorous about it by destroying the flying missile under your command in mid-air, so it feels more natural even as errors are being generated.

We're hoping we've covered most Exception error cases to stabilize the game, but we'll see :)

Post comment Comments
aloha2436
aloha2436 - - 119 comments

Look at it this way, even military hardware breaks, and always when you least expect it! In fact, you may have just made the game more realistic :D!

Reply Good karma Bad karma+3 votes
roycewicz
roycewicz - - 14 comments

hahaha, divide by zero is very very seriouz biznezz indeed!

Reply Good karma Bad karma+4 votes
blahdy Author
blahdy - - 997 comments

lol, your damn Tomahawk div-zero'd on top of my infantry squad. team killer :P

Reply Good karma+3 votes
GriffinZ
GriffinZ - - 4,719 comments

I have a feeling that some maches may turn into "divide by zero error" friendly fire fests :p

Reply Good karma Bad karma+1 vote
pintur
pintur - - 67 comments

instead of the tomahawk self destructing could u make it just crash with out detonating the warhead?

Reply Good karma Bad karma+1 vote
blahdy Author
blahdy - - 997 comments

Nah, that's too much unnecessary engineering for little benefit.

The failures listed above do not happen often, they are emergency measures taken to prevent your game from crashing when an anticipated rare accident occurs with the code.

Reply Good karma+1 vote
Post a comment

Your comment will be anonymous unless you join the community. Or sign in with your social account: