Mod Tester's Handbook tutorial - Crispy

Writing a Bug Report: Bug Classification

A.k.a. severity
Separating bugs according to how severe they are is an important stage of the prioritisation process. At any stage of the project, the focus should always be on fixing the most severe bugs to maintain a playable, testable build for all developers to have access to for their work. Bug classes generally range from A to D, here's a brief guide to what these represent.

Aclass bugs are any issues that massively undermine either the end-user experience or some form of legal agreement or mandate. Any bug that disrupts user progression within the game, such as a hard lock that freezes or crashes the game and hardware, or a soft lock that prevents a user from controlling their avatar as normal, or any vital game text that is missing or indecipherable is an A class bug. Even if a crash can be avoided without cheats in order to progress, it is still considered an A class. This is where 'severity' (denoted by letters) differs from 'priority' (denoted by numbers): avoidable crashes and hangs will receive a lower fix priority, even if they are still classed as As. Legal issues should not be ignored, copyright laws apply as much for mods as they do for commercial game releases.

B class bugs are still considered 'must fix' in most scenarios. The difference between an A and a B is that a B class generally does not prevent progress and can be avoided easily. If a bug does cause massive disruption but it is extremely difficult to recreate under normal play conditions, it may be classed as a B instead of an A. B classes include major graphical and audio corruptions, missing key audio and graphics, holes in geometry that allow the user to fall easily out of the level, geometry that allows the user to get stuck and have to restart the level or load a save, any text that cannot be understood because it does not make sense or is incomplete.

Cclass bugs are perhaps the most difficult bugs to come to terms with. These are those minor issues that need to be sidelined late in development if the team is to hit a scheduled release date. Usually C class bugs are easy to fix and will be fixed eventually, but sometimes the As and Bs have to take priority and a line has to be drawn in the sand if there is ever to be a release. C class bugs should never stop you from releasing a mod. They are defined as bugs that are either far from the beaten track or otherwise hard to find, or they are so minor that many users will not even notice them during play. Examples include characters partially clipping into geometry, minor grammatical or typographical text issues, any bugs that require very specific timing or multiple intersecting conditions to reproduce, minor lighting and sound problems, and any issues that don't really hinder the user's ability to play the game but are still obviously erroneous. 'Tester bugs' are usually assigned a C class, simply because they can only be found through behaviour that specifically focuses on the unconventional.

D class bugs are essentially suggestions. If a bug summary roughly reads "X feature should be Y", it can be classed as a suggestion. D class bugs will be reviewed by the designer/lead and prioritised depending on how relevent they are deemed to the intended gameplay. If they are completely at odds with what the game is, or if they are simply unrealistic and unachievable, they will be waived and closed. Testers shouldn't really be putting forward any D class issues once a mod reaches RC (Release Candidate) status.

Here's an example of bug classification:
In the first level of a single-player game, there is tutorial section aimed at teaching a player to use the sword. Some text shows up on screen asking the player to destroy a pot with their sword.
A class - In addition to hitting it with their sword, the player can alternatively pick the pot up and drop it to smash it. If the player does this, the game crashes to the Windows desktop and the user has to reboot the game. The player can only progress if they use the sword to smash the pot.

B class - In the same scenario, if the player drops the pot instead of smashing it with their sword, the game will not crash but instead the exit door will not trigger. The player cannot go any further and must exit the room and re-enter to spawn a new pot. The player can only progress if they use the sword to smash the pot.

C class - In the same scenario, when the player hits the pot with the sword, it is broken into pieces which scatter over the ground, but the 'smash' sound effect is very quiet and can barely be heard over the ambient birdsong and the swing of the sword. Clearly the volume of the sound effect needs to be mastered to a more audible level against the other audio in this scenario, but the fact that it is too quiet does not hinder progression in the game, so it is not a major issue.

D class - In the same scenario, one tester believes there should be a reward for smashing the pot. "It would be nice if you got some gold or healing herbs for destroying the pot." This is a suggestion that may or may not be pertinent. If, in later stages, pots will contain items once smashed, it makes sense to explain this relationship in the tutorial phase of the game. But, on the other hand, if no pots contain items for the entire duration of the game, receiving gold here would create a contradiction.

Note: Some bug databases also have an option to set a bug's priority. As mentioned above this is not the same thing as severity. Only Project Leads, Department Leads and QA Leads should be setting the priority level for bugs. They will do so according to how noticable the bug's effect is on the game as a whole. A high class issue may only be found by 1% of people playing the game if it is hard to reproduce or far from the 'critical path' (the main route through the game). In this case it would get a lower priority than a crash experienced 50% of the time in an unavoidable section of the game.