Lead Production Tester for a AAA studio

Post tutorial Report RSS Mod Tester's Handbook

A reference guide for testing mods. Includes info on how to find bugs, how to narrow down the steps to reproduce and how to write good bug reports. Also includes a 'game development terminology' reference section.

Posted by on - Intermediate QA/Testing

Table of Contents

Introduction

  1. Why test?
  2. Who should read this tutorial?

The QA Process

  1. Lifecycle of a bug

Bug Hunter

  1. Be a loser
  2. Be fast
  3. Be curious
  4. Be destructive

Bug Detective

  1. Communicate
  2. Information gather
  3. Eliminate
  4. Reproduce

Writing a Bug Report: The Bug Summary

  1. Keywording
  2. Summarising

Writing a Bug Report: Bug Classification

  1. A class bugs
  2. B class bugs
  3. C class bugs
  4. D class bugs
  5. Examples for comparison

Writing a Bug Report: Communication

  1. Precision
  2. Relevance
  3. Neutrality

Writing a Bug Report: Steps to Reproduce

  1. Format
  2. Example

Writing a Bug Report: QA Terminology

  1. List of terms

Conclusion

  1. What testing can do for you
  2. About the Author
  3. Extra Reading

Introduction

Why test?

When you design and create a game you usually do so according to a vision. Your aim is to get the game to accurately convey this envisaged experience to the player. Since your vision is usually focused on making the game fun and enjoyable to play, you won't always consider areas of the game that could be manipulated or overlooked to result in confusion, frustration or boredom. In essence, the purpose of testing is to give developers a new perspective on their creation. On almost every occasion, testing will have a developer saying to themself: "Oh, I didn't think of that." or "I can't believe I forgot to do that." Testing is about anticipating player perceptions to the game and giving developers a second chance at improving their creations before they reach the end-user.

Who should read this tutorial?
This tutorial is aimed at everyone involved in the QA process, which is basically everyone on your mod team from testers to developers. Anyone who is writing or editing bug reports should be reading this tutorial.

For testers
I made the tutorial because it seems most people testing mods don't actually know how to really probe the game and push at its limits. Most mod testers (appropriately named 'playtesters') simply play the game as normal and note down anything they happen to see. They don't go looking for bugs, or creating bugs by being inventive, they simply stumble across them by chance. These people are playing the game, not really testing it, and that's why I made this tutorial. The tutorial has been written in such a way that it does not apply specifically to any platform or any particular genre, this is so that any tester for any mod can glean some information from the handbook and apply it to their project, whatever it may be.

For developers
I especially stress the need for developers to be familiar with good bugging practise. The amount of times I've been looking for a bug in a database and seen ambiguous, unhelpful bug summaries like:

"Jon, this looks broken"
"hole in geom"

This not only forces the reader to open the bug to try to find out what it's about, but it means that nobody else on the team is going to find the bug via a keyword search. It's an accumulation of bad communication like this that wastes time and pushes back development. Help your team by being as transparent as possible in your documentation at every stage of development.

It's amazing that this sort of lazy approach to bug reporting doesn't vary from studio to studio, even when you look at industry-leading developers you still get time-wasting bug summaries like these, often without much more explanation in the bug description. Part of the reason they do this is because they assume they're writing this bug for people in the office who can come up and ask them if they have any queries. Remember that in modding the majority of the time we don't have the luxury of face-to-face discussions, so make sure your bugs are as descriptive and clear as possible, whether you're a tester or a developer.

For QA Leads?
Of course, whoever is managing your testing process should read this and be familiar with good bugging practise, but if you're more interested in setting up a test schedule, deciding on testing strategies and managing bug databases, I am also writing a separate tutorial for mod QA leads that deals with the management side of things. Keep an eye on the tutorial section for this or just watch my profile to be notified automatically when it goes live.

Now, let's get on with the show!


The QA process

The lifecycle of a bug

This is a list of all the stages a bug may go through during its life. I've included it here because it contains many processes that many first-time testers are unaware of. It's easy to assume on the outside of development that bugs are found, reported and fixed in a flawless process that goes from A to B without hiccups. The truth is that errors are made and multiple checks are in place to make sure that developers don't waste their time trying to fix duplicate bugs that are already fixed, or that fixes the developer has made on their own version actually make it into the game and aren't left out by accident.

This is where a lot of double-checking and a hell of a lot of communication come into play. These areas are vital to quality assurance and are there to save time in the long run. Any bugs in your database should be going through most of the following steps:

  1. Discovery The bug is discovered by a developer, tester or perhaps a player.
  2. Communication The testers/developers are notified of the problem. A player will make a forum post or send an email, or perhaps tell someone in-game or in an IRC channel. This needs to be logged immediately and passed onto whoever handles QA. On the other hand a tester or developer will need to talk to their teammates and check the bug database. Perhaps the issue has already been found and a report exists? Perhaps you need to edit the report with new information?
  3. Investigation A developer or tester should investigate the bug. Perhaps it is a design feature that has been misinterpreted by the user. If it is a bug, the tester will need to document which conditions the bug can be reproduced under and how reliable these steps are.
  4. Documentation A bug report is logged in a bug database with information including a search-friendly summary, description, steps to reproduce, reproduction rate, priority, seriousness,etc.
  5. Verification The bug is checked by a senior member of QA for clarity, accuracy, significance and urgency.
  6. Assignation The bug is assigned to a particular member of staff to be fixed or discussed. This process can go back and forth between team members. Whenever a bug is assigned to someone a reason must be given, e.g. "NMI" (need more info), "NAB" (not a bug), "not an art issue - maybe an error in the code?", etc.
  7. Fix An attempt at fixing the bug is made. Some comments briefly describing the solution should be added to the notes so that expertise is shared across the board and similar future issues can be dealt with quickly.
  8. Regression The fix is committed to a build of the game and that build is tested. The result of the regression is logged in the bug entry, e.g. "Fixed" (plus the name of who reported it fixed), "Fail" (plus details of why it failed; the attempted fix may have created a different issue), "CNR" (cannot reproduce; plus the reasons for this).
  9. Closure The bug may be closed after regression or even before a fix has been attempted, but a reason for closure must be given, e.g. "Fixed" (regressed as 'Fixed'), "Closed - as designed" (not actually a bug), "Waived" (the bug has been deemed too low a priority for this release, "Duplicate" (the bug has already been reported; a link to the duplicated bug should be given).
  10. Re-opening If a bug has been waived for one version, it may be re-opened to be fixed for a later patch when priorities and focus are re-assessed.


Bug hunter

Tips for finding and producing bugs
Here are some methods you can adopt to find bugs in your game:

Be a loser
Fail challenges. Get yourself killed. Give up midway through a fight or a scripted sequence. Reload the game. Do you lose a life? What happens if you lose all your lives? Is the game saving properly? When you reload are you in the right place, with all the equipment you had before? Are all the NPCs where they should be? Is your mission progression saved?

Be fast
Find the quickest way to complete the game and keep pushing the boundaries. Can you get stuck in a closing door, or kill an enemy before they are ready to fight you? Can you trigger tutorial messages on top of eachother by completing tasks in record time? Can you get somewhere before the game has had time to draw it, maybe even fall out of the level? Use as many movement exploits as you can find to give you speed boosts, jump boosts and so on. In multiplayer a lot of the time by pushing the game to its limits you will find you can reach an area quicker than any other player or team and benefit from a considerable (and unfair) advantage.

Be curious
Bump into walls. Bump into enemies. Fall off cliffs. Trundle through rivers. Attack walls and lunge into the environment. Try to avoid savepoints, cutscenes, scripted sequences. Jump to supposedly inaccessible areas. Do you fall out of the level? Can you walk through enemies or scenery? Do you get stuck? Do the lighting and other graphics display correctly in all areas? Does the sound play correctly in all areas and on all surfaces?

Be destructive
Break the game mechanics and the game experience. Attack everyone and everything with everyone and everything you can get your hands on: friends, foes, flora, fauna, inanimate objects, breakables, collectibles... everything. Button spam. Spam the pause menu. Spam everything and anything. Trigger as many events as possible at the same time. If you can input text: spam as many characters as possible. Go round the the track the wrong way. Get in people's way. Do things out of sequence (both in-game and in menus). If it's multiplayer: piss people off. Spawncamp, spawnrush, spawnblock: be a hyperactive kid with ADHD.

Single-player versus multiplayer testing
Now, not all of these processes can be applied to multiplayer as equally as single-player. In fact, due to their largely scripted nature on only one machine, it's a lot easier to test singleplayer than multiplayer games that span multiple players on multiple machines all doing completely different things. But, in general, you can apply the same theories to multiplayer, the key difference is that you need to be able to communicate with eachother at all times because a lot of multiplayer bugs come from the fact that people are interacting in unforeseen ways and with unforeseen consequences. For example:

Be a loser
If you don't perform well (for whatever reason: intoxicated, tired, newbie, griefer), will it dramatically affect the enjoyment of the rest of the players? If you get off to a slow start, can you still complete with other players? If you join the game late, can you still compete with other players who have been playing the match from the start? If you die lots does the game get more difficult or are you able to able to improve as you play regardless of number of deaths?

Be fast
Try to discover ways to increase your movement speed. Can weapon knockback be used to gain an advantage? (e.g. rocket-jumping) Can common engine exploits be taken advantage of in your mod? (e.g. wallstrafing, bunnyhopping, strafejumping, scripts) What about running to the enemy as fast as possible; does both side arrive in the middle of the level at the same time, or is there a route that needs adjusting to balance things?

Be curious
Try changing settings during a match. Try quitting and re-joining games; could this be used to gain an advantage in any way? Try joining the other team. Try using abilities at distance; is it balanced? Do third-person animations play correctly and match what the other player is seeing first-hand on their screen? Can all victory conditions be met as expected? Are the victory conditions communicated clearly to beginner players? Can you use any materials (or even other players) to access anywhere you shouldn't be able to? Is all the information you are reading correct? (e.g. scoreboards, weapon/ability/class descriptions)

Be destructive
What happens if you quit the match at different times? What happens if everyone in the game performs an action at the same time? (shoot, micspam, connect to a server, quit the game, change team/class, etc.) Can you spawncamp? Can you obstruct your own teammates? Can you block any paths unfairly? Can you kill teammates? (if so, is there anything in place to deter you from this?). If you go AFK or deliberately hide away somewhere, does the gameplay suffer? Can you exploit game menus for any gain?

Bug detective

What to do if you find a bug

Communicate:

As a tester, the first thing you need to do when you find a bug is to tell everyone else about it to create a general sense of awareness. This will mean other testers are more focused so if they do find the same issue, they will be able to provide more information on how to attempt to reproduce it. Reproduction is the second course of action. Try to recreate it by doing exactly the same thing you did before. If this still produces the same effect, you need to try to narrow down your steps by changing one condition but keeping the rest the same.

Information gather:
The best first step is to try to note down as much information about the incident(s) as possible. When you write a bug you will need to include most of this information in any case, what you need to do next is eliminate any information that isn't specific to the bug. Remember that hardware and software settings are also 'conditions' for the bug. It could be that the game is asking your graphics card to do something it can't, it could be that you have on-screen text turned off and the game is trying to give you a tutorial message that conflicts with this option setting.

Eliminate:
The greatest detective of all time once said: "If you can eliminate all the other choices, the remaining choice, no matter how improbable, is the answer." Once I spent 4 hours trying to figure out what was producing a 100% reproducible crash in a game. When I tried it with another game disc it still occurred, but when I tried it on a different machine it didn't occur. The only conclusion left was that the hardware (the console) was unreliable. Usually we go into testing assuming that the tools we use are there to help us find problems, but as a tester you have to consider that sometimes they can be part of the problem themselves. Ever since that experience I always make sure that I check in the early stages that hardware isn't the cause of the issue, I'd advise you to do the same, especially when talking about PCs that have many different pieces of hardware to keep track of.

Explore every possibility.


Eliminate conditions beginning with the most general. One of the first things you should try if you can do so easily is to try to reproduce using the same conditions on a different machine. Other general conditions could be to see if the same bug can be reproduced on a different level, with a different weapon, with a different class, against a different enemy. If it's a graphics issue see if it occurs on a different resolution. Try to ascertain how exact the steps need to be for successful reproduction.

Reproduce:
Reproducing is really part of the elimination process, but even when you think you have the answer you really need to remember to try to get another tester to reproduce the problem using what you have written as a guide. Sometimes you might have missed an important bit of information that will be uncovered when the other tester fails to trigger the bug using your steps to reproduce. Sometimes this missing information is the key to what is causing the problem in the first place.

Only after completing these steps are you ready to begin writing your bug report

Writing a Bug Report: The Bug Summary

The bigger the project, the more important the summary. The purpose of the summary is not only to give a concise description of the bug, but also to provide keywords that will allow the same bug to be found in the database at a later date. The below example is a fairly professional method of doing it. Cherry-pick any useful information you find in this section, but it doesn't necessarily have to be replicated. If you have testers that won't be able to follow a single format, use a system that you know will be adopted readily and can work for your team.

A typical summary in a good professional bug database might look something like this:

[GFX] E1_L02: Emergence > Docks - Northernmost crane model has incorrect texture (black)

This summary contains a vast amount of information that tells me, in just one line:
- what area of development the bug affects
- its location within the game
- which part of the aforementioned location if affected
- exactly which asset is affected
- and the symptoms the issue is presenting

Here the issue is that an environment model (a crane) is showing up as completely black in-game for some reason. The tester may or may not think they know the reason for this, as far as the summary goes this is irrelevant, the focus needs to be on communicating the essential pieces of information in an objective, clear and searchable format.

[GFX] E1_L02: Emergence > Docks - Northernmost crane model has incorrect texture (black)

I have added some colours to the example summary to highlight the different key areas. When using a basic (usually web-based) bug database or forums, both of which rely heavily on text-based searches, the more information you can put into the summary line, the better. On a forum, the summary line will obviously be the title of the thread.

[ ] : > - (structure)
These symbols are used to give the summary a uniform structure and allow the eye to locate different pieces of information quickly. Every bug always includes certain items of info, and these items always appear in the same order. The symbols are there to separate these pieces of info and to guide the eye through the summary line. When creating your summary format you can use any symbols in any order, so long as they are easy to remember and do the job of making the summary clear.

GFX (category)
To help with searches and search filters, you can choose some important information to put at the beginning of the summary. The category of the bug (i.e. which area of functionality it affects)
is very useful, especially for developers trying to locate bugs that affect their own work. With this information at the beginning of all bugs, I can search via a keyword like "black", and then alphabetise all the results to bunch up all the bugs by category from 'AI' to 'SFX'. If I see this bug in a list of perhaps 20 other bugs, I immediately know that this one in particular is a graphics issue, so as a reader I am well situated to understand the rest of the summary that bit better.

E1_L02 / Emergence / Docks (location)
This information leads you through the game to the location of the bug. In this case you have the location of the bug within the game world itself: "E1_L02" means it is in the 2nd level of the first episode, "Emergence" is the title of the level/mission/quest, "Docks" is the actual physical location within the game world. It's important to have both "Emergence" and "E1_L02" in the title because while the name of the level is easier to remember, the label will be more useful when searching through folders or code and it also contains information about its location within the game's progression flow.

This part of the summary isn't only for locations within a level, though, it is used to give information about the location of the bug within the game.
A bug could be located within a particular feature: "General: Classes > Soldier -"
or a menu: "Front End: Options > Sound -"

Northernmost crane model has incorrect texture (black) (summary)
Although the summary is only 7 words long, it contains 4 adjectives to link the 2 nouns (the 'texture' and the 'model'). There is information about what generally is affected (a model of a crane), which specific crane model is affected (the one furthest north), and how specifically the model is affected (that the texture is somehow wrong and showing up black).

A less meaningful way of summarising the bug would be "crane in docks is black". This simply doesn't convey as much information; it doesn't tell an artist or level designer which of the multiple cranes in the docks area has an error, and it doesn't describe the error in much detail. Saying the
crane is black could mean that it is in shadow and there is a problem in fact with the lighting in the area.

Don't be afraid to add keywords in parentheses if you think they will help other team members find the bug in the database. You don't have to masterfully work them into the summary sentence, it's fine to add them in brackets so long as the meaning is clear. Finally, if you can economise the word count in the summary without compromising the perceived meaning, it helps to keep things short. Removing superfluous words like "the" and "is/are" and letting an adjective or past participle follow the noun is usually acceptable. You can also sometimes eliminate "which"/"that" by use of the gerund/present participle.

E.g. "The door is broken before the player activates the switch which stops progression" could be turned into:

"Door broken before switch activated, blocking progression"

Writing a Bug Report: Bug Classification

A.k.a. severity
Separating bugs according to how severe they are is an important stage of the prioritisation process. At any stage of the project, the focus should always be on fixing the most severe bugs to maintain a playable, testable build for all developers to have access to for their work. Bug classes generally range from A to D, here's a brief guide to what these represent.

Aclass bugs are any issues that massively undermine either the end-user experience or some form of legal agreement or mandate. Any bug that disrupts user progression within the game, such as a hard lock that freezes or crashes the game and hardware, or a soft lock that prevents a user from controlling their avatar as normal, or any vital game text that is missing or indecipherable is an A class bug. Even if a crash can be avoided without cheats in order to progress, it is still considered an A class. This is where 'severity' (denoted by letters) differs from 'priority' (denoted by numbers): avoidable crashes and hangs will receive a lower fix priority, even if they are still classed as As. Legal issues should not be ignored, copyright laws apply as much for mods as they do for commercial game releases.

B class bugs are still considered 'must fix' in most scenarios. The difference between an A and a B is that a B class generally does not prevent progress and can be avoided easily. If a bug does cause massive disruption but it is extremely difficult to recreate under normal play conditions, it may be classed as a B instead of an A. B classes include major graphical and audio corruptions, missing key audio and graphics, holes in geometry that allow the user to fall easily out of the level, geometry that allows the user to get stuck and have to restart the level or load a save, any text that cannot be understood because it does not make sense or is incomplete.

Cclass bugs are perhaps the most difficult bugs to come to terms with. These are those minor issues that need to be sidelined late in development if the team is to hit a scheduled release date. Usually C class bugs are easy to fix and will be fixed eventually, but sometimes the As and Bs have to take priority and a line has to be drawn in the sand if there is ever to be a release. C class bugs should never stop you from releasing a mod. They are defined as bugs that are either far from the beaten track or otherwise hard to find, or they are so minor that many users will not even notice them during play. Examples include characters partially clipping into geometry, minor grammatical or typographical text issues, any bugs that require very specific timing or multiple intersecting conditions to reproduce, minor lighting and sound problems, and any issues that don't really hinder the user's ability to play the game but are still obviously erroneous. 'Tester bugs' are usually assigned a C class, simply because they can only be found through behaviour that specifically focuses on the unconventional.

D class bugs are essentially suggestions. If a bug summary roughly reads "X feature should be Y", it can be classed as a suggestion. D class bugs will be reviewed by the designer/lead and prioritised depending on how relevent they are deemed to the intended gameplay. If they are completely at odds with what the game is, or if they are simply unrealistic and unachievable, they will be waived and closed. Testers shouldn't really be putting forward any D class issues once a mod reaches RC (Release Candidate) status.

Here's an example of bug classification:
In the first level of a single-player game, there is tutorial section aimed at teaching a player to use the sword. Some text shows up on screen asking the player to destroy a pot with their sword.
A class - In addition to hitting it with their sword, the player can alternatively pick the pot up and drop it to smash it. If the player does this, the game crashes to the Windows desktop and the user has to reboot the game. The player can only progress if they use the sword to smash the pot.

B class - In the same scenario, if the player drops the pot instead of smashing it with their sword, the game will not crash but instead the exit door will not trigger. The player cannot go any further and must exit the room and re-enter to spawn a new pot. The player can only progress if they use the sword to smash the pot.

C class - In the same scenario, when the player hits the pot with the sword, it is broken into pieces which scatter over the ground, but the 'smash' sound effect is very quiet and can barely be heard over the ambient birdsong and the swing of the sword. Clearly the volume of the sound effect needs to be mastered to a more audible level against the other audio in this scenario, but the fact that it is too quiet does not hinder progression in the game, so it is not a major issue.

D class - In the same scenario, one tester believes there should be a reward for smashing the pot. "It would be nice if you got some gold or healing herbs for destroying the pot." This is a suggestion that may or may not be pertinent. If, in later stages, pots will contain items once smashed, it makes sense to explain this relationship in the tutorial phase of the game. But, on the other hand, if no pots contain items for the entire duration of the game, receiving gold here would create a contradiction.

Note: Some bug databases also have an option to set a bug's priority. As mentioned above this is not the same thing as severity. Only Project Leads, Department Leads and QA Leads should be setting the priority level for bugs. They will do so according to how noticable the bug's effect is on the game as a whole. A high class issue may only be found by 1% of people playing the game if it is hard to reproduce or far from the 'critical path' (the main route through the game). In this case it would get a lower priority than a crash experienced 50% of the time in an unavoidable section of the game.

Writing a Bug Report: Communication

Exhaustive, clear, concise: a bug report is all these things

The most important quality to look for in a tester is an ability to communicate important information clearly to your developers in a professional manner. If they are really good at finding bugs but can't explain the bugs clearly to the devs they will end up wasting a lot of time. If they come across as very negative with their feedback they will create disruption within the team. There are a few things to keep in mind when writing a bug report: you need to be precise, you need to keep it relevant and you need to stay neutral.

Precision
Accuracy is key here. Don't make any claims in a bug description unless you are 100% sure about them, the last thing you want to do is give information that could be misleading. If you have a gut feeling about what is causing the issue, use phrases like 'it seems to be', 'this may be' or 'perhaps this could be'. Don't ever make assertions if you aren't completely certain, it only wastes time. If talking about reproduction rates, give actual numbers instead of percentages; 3 out of 15 is a much more illuminating figure than 1 out of 5, even though both could be described as a 20% reproduction rate. Precision also extends to terminology: when describing bugs and how to reproduce them, you should be using the exact names the designers have given to characters, weapon, moves and locations. If an area of the game is introduced as the 'Vineyard' in the game, do not refer to it as a 'Farm' or 'Field' or fellow team members will be less likely to find your bug in a database.

Relevance
Don't include any extraneous information. Your job as a tester isn't over until you have narrowed down your steps to creating a 100% reproduction rate with the minimum amount of steps possible. Everything you include in your bug report has to be relevant. On the other hand, don't make any assumptions about what is and what isn't relevant if you are unsure. If you have found that a graphical issue that can be reproduced in both resolutions of 1024*720 and 800*600, make sure this information is in the bug report (don't ever forget to mention any conditions under which the bug cannot be replicated, this information can hold important clues). If in doubt, include extra information as a note in the bug.

Neutrality
Stay neutral. Give as objective an account of the issue as possible. Humour or sarcasm can be misinterpreted to create ill feeling and negative feedback is an unhelpful approach. If you feel you need to give a personal opinion try to distance yourself from your statement to make it sound more humble and less self-righteous. Phrases that show consideration for the end-user instead of over-emphasising personal preferences will be much more readily received by the developers than personal attacks or overbearing judgements. E.g. "It is this tester's opinion that the control scheme is unintuitive and may cause the end-user considerable frustration" sounds better than "The controls are fubared".

Writing a Bug Report: Steps to Reproduce

Once you have narrowed down your repro steps you can put them into the bug report. Remember to number the steps and to begin them with the booting of the game. Never, under any circumstances, give instructions for using the debug menu in your repro steps. Firstly, once a game has entered Beta, you should never put a bug in that was found via use of a debug menu (it's extremely bad practice). Secondly, your steps should always be directed towards a player of the game, not to a developer. If you always imagine that a player will be reading your repro steps, they will always be clear and accurate. Thirdly, it is good practise that regression of the bug should be performed without the use of debug commands. This means if the bug is on the last level of a game, a full playthrough is required to reliably confirm if it is fixed or open.

Here's an example of the format I would use to write a bug:

Steps to Reproduce:
1. Boot [title name] [platform] [settings]
2. Load [level]
3. Move to [location]
4. [perform action]

Result:
[results]

Your first line should always be an instruction to boot the game on a certain platform. This is to avoid any important steps being missed out, such as booting in a certain language or at a certain resolution. If the bug does require a certain setting in order to be reproduced, add that information to this line. If the bug can be performed in any resolution, or in any language, add this as a note to the bug description.

Unless the bug occurs immediately after boot (e.g. on the title screen or on the Front End), your second line will usually be an instruction to load a particular level. This line should never involve debug commands. For this reason it is better to phrase this line as "Progress to mission 3" instead of "Load e1_m3", because it makes it clear you have found the bug during normal play. It is acceptable to attach a save file to the bug report that will allow a developer to recreate and observe the bug more quickly. In this case it's fine to say "Load bug1364.sav (see attachments)", just remember to try to re-name the save file using the same numerical ID as the corresponding bug entry in the database (it just makes it easier for developers to find it if they download and save it into a local folder).

Finally, it helps to add a line for the 'result' of the repro steps. This reinforces the overall reproduction process and makes it easy to copy+paste this section of the bug report if needed.

Here's a full example of some repro steps:

Steps to Reproduce:
1. Boot AdventureQuest PC
2. Progress to Denbrooke
3. Talk to Whipsnade to add the Caves to your map
4. Travel to the Caves via the map
5. Collect the Silver Key from the central cavern
6. Locate the Prison gate within the caves and open it using the Silver Key
7. Quickly kill the guard menacing the civilian (lunge attack works best for this)

Result:
The civilian is not slain and remains in a 'cowering' pose, even though the guard is dead and no enemies are in sight.

As you can see from the above example, no assumptions have been made about the reader. Literally any player of the game should be able to reproduce the bug under their own steam, without needed to query any of the steps. Now, prepare to get amourous with your devs! If you are writing clear repro steps for them, they will love you for it!

Writing a Bug Report: QA terminology

A list of terms used frequently in development and testing

AI (a.k.a. Artificial Intelligence)
A set of behavioural responses programmed into computer-controlled characters and enemies designed to challenge and further immerse the player. Most AI is handled in the game code, but in some single-player or co-operative games, more tightly scripted elements like pathfinding can be put together in a level editor and fall under the responsibility of the game or level designer.

Alpha (channel)
A part of the texture that defines its area as being transparent or translucent in the game. Colours uncommon in most environments are used to designate the areas for Alpha removal, such as pure blue, magenta or bright green. If an Alpha is not functioning correctly for a game texture, one of the aforementioned colours may be visible.

Alpha (version)
An incredibly early build of the game that has enough there to boot up, but lacking many features. A working blueprint of the game that gives some idea of structure and layout. Many features will be unimplemented at this stage; expect crashes, hangs and soft locks due to missing/non-functional files, basic or placeholder artwork and rough audio.

Beta
The stage at which a game is feature and content complete and in a playable state. In theory the game can be tested from start to finish (perhaps by use of some debug commands to circumnavigate known issues). It is by no means bug-free, but most major issues should have been addressed.

Blocker
Any bug that prevents the user from progressing in the game. Blockers are absolute, i.e. if the level or game can be restarted or absolutely anything can be done (no matter how unintuitive or out of character) to bypass a critical issue, the issue in question is not a blocker. In short, blockers prevent full testing of the game and receive a very high fix priority.

Brush
In level editors that use Binary Space Partitioning (or BSP) such as the Half-Life 2 engine, any three-dimensional solid that is not an 'entity' or a model. Most of the major geometry in a level of this type -such as the ground, walls, ceilings, etc.- will be made out of brushes.

BSoD (a.k.a. Blue Screen of Death)
An acronym to describe a crash resulting in a blue screen with an error message or a call stack. Not just limited to PCs; BSoDs can occur on most consoles.

Build (see also Version)
All the files an end-user would need to play the game, a fully-compiled version of the game. If you have a single member mastering your builds manually, they will probably only compile a new build when needed. Most prototyping and testing will probably be done locally using work-in-progress files.

C2D / C2W (Crash to Desktop/Windows)
An acronym used to describe when a game closes itself abruptly and returns the user to the desktop environment of their computer.

Call stack
For the purposes of simplification (please correct me if I'm wrong), a list of a program's subroutines in operation at the time of a crash. Essentially a more detailed error message, a call stack appears on-screen as an alternative to a dump file or basic error message.

Clip
A term used commonly in testing to describe when a player avatar or object passes through an object. The term is through to be derived from id Software's 'noclip' debug mode, that removed player collision detection from the game. In games development, the term clipping actually means something entirely different.

CNR (cannot reproduce)
When regressing a bug, what the QA Lead would write in the bug notes if any of a bug's repro steps cannot be performed because of another bug. It is good practise to also cite the bug(s) preventing regression. A developer may also write this in a bug if they cannot reproduce the issue on their system.

CNT (cannot test)
The acronym used on a checklist or testplan to show that a check cannot be performed because of another bug or because a feature has not been implemented. It is good practise to also cite the bug(s) preventing this check.

Collision detection
The code that prevents two objects from intersecting with eachother. In 3D games, collision between models is handled by a 'bounding box' that may or may not be identical to the visible boundaries of the model. Similarly, in level design, artificial 'invisible walls' may be added to prevent the player from exiting the playable area of the game world. In 2D games this may be handled by 'sorting' (see depth sorting).

Console
A drop-down text input menu some games use to perform debug (and in-game) operations.

Crash
When a game closes itself abruptly. After a crash the user may be taken to a 'dump' screen or a screen with an error message or error code. Following this, the system should reboot itself automatically.

Critical path
A term used to describe the main, often necessary, route through a game. This can extend to unlockable upgrades, major paths through a level or conversation strings with key characters. It does not refer to optional extras such as bonus levels, non-essential upgrades and incidental dialogue. The closer a bug is to the 'critical path', the more likely the end-user is to find it and the bigger its fix priority.

Debug
A mode or set of commands that allows a developer or tester to perform operations that either have not been fully implemented or cannot normally be performed within the rules of the game. Examples of this include cheats like 'god' mode, 'noclip/fly' mode, 'notarget/invisible' mode, as well as modes that help artists assess their work, such as 'wireframe' mode. Any bug found when using debug mode should be re-attempted without the use of debug to ensure its validity before a bug report is entered into the bug database.

Depth sorting
A method of creating the illusion of a third dimension for 2D games. When two 2D objects occupy the same 2D co-ordinates, based on the values of these objects, depth sorting will determine which object appears to pass under the other. If a character in a top-down 2D game appears to pass over or inside of a tree, this would be a depth sorting bug.

Diffuse map
The essential layer of a texture that defines the combinations and arrangements of colours on a surface. The diffuse map makes a flat surface look like a brown door made out of wood, for example. Any other maps are simply accentuating and improving on this effect.

Draw distance
Draw distance is the distance within which objects in the world are drawn. Objects beyond this distance are 'culled' (not drawn) and will only be drawn as the player avatar approaches them. If the draw distance in a game is set too close to the player, objects will 'pop-up' suddenly, breaking the immersiveness of the gameplay.

Dump
The process that occurs after a crash whereby game data relating to the crash is recorded into a 'dump file'. Programmers can look at these dump files and attempt to work out which part of the game produced the error.

Entity
In some level editors (including Half-Life 2), entities are special additions to a level that control various properties relating to the world, the player and all other objects within the world. Entities are usually invisible, but can be assigned to brushes (as brush-based entities). For instance, (in Half-Life games) a rope down a cliffside can only be climbed once a 'func_ladder' has been placed beside it. Without the entity allowing a player to climb, the rope is just a non-interactive piece of scenery. Entities control most 'special' rules in a game (such as scripted sequences, physics puzzles, fogging, breakable scenery, etc.) and even some basic rules (such as light sources, transparent glass and doors).

Environment art
Any level art not created in level editing software. Generally speaking, any separate model placed inside a level to increase the level of detail. Usable weapons, character models, vehicles and structures are not considered environment art.

Exhaustive playthrough
An extensive playthrough of a single-player or co-operative game undertaken by a tester whose focus is to experience every aspect of the game. This includes: completing every level/mission/quest, talking to every character, collecting every item, unlocking every upgrade, etc.

Fogging
Fogging is often used in conjunction with draw distance. A maximum and minimum density for the fog in a level is set and these values both correspond to distances from the player avatar. Objects moving between the minimum and maximum fog densities will appear to fade in and out of the fog. This is a much smoother visual technique than just having objects pop-up in the distance.

Geometry
A useful term to describe large areas of 3D level art in abstract forms, such as cliffs and caves. E.g. "Missing/untextured face in cliff geometry".

Hang
If the game ceases to respond to user input (i.e. it freezes) this is called a hang. A hang is different to a crash because a hang usually requires the user to manually reboot the game or the system.

Hard Lock
Any hang or crash that requires the user to reboot both the hardware and the game to resume control of the game. All hard locks are class A issues.

HoM (Hall of Mirrors)
In some engines, the visual effect seen when viewing the face of an object with no texture assigned to it. This effect can be seen most clearly if falling out of the level.

Lag (note: not the same as latency)
While latency is, generally speaking, the measured or demonstrable time taken for data to be sent, received and retrieved, 'lag' is a term used to express a noticable and unacceptable delay between a user's input and their resulting actions in-game. Lag can be attributed to too much client-side traffic, too much server-side traffic or -less likely- game/engine code.

LOD (Level of Detail)
The name given to the optimisation technique of swapping in higher-detail versions of models as the player approaches them in the game world. If done correctly, this changeover appears to happen seamlessly because the model usually switches to a lower or higher detail version at a distance where the player cannot discern any difference between the two.

Loser playthrough
A playthrough of a single-player or co-operative game undertaken by a tester to ensure that failure in the game functions correctly on all levels. This includes: dying to every enemy, failing every level/mission/quest in every conceivable way, attempting purchases/transactions without an adequate stock of money/points/items, manually quitting levels via the menu, etc.

Milestones
For modders, milestones are targets for the team to hit that help plan out the mod's development timeline and motivate team members. If realistically achievable and communicated well to the team, milestones can have a very positive effect on productiveness and momentum. Bugs fixes can be scheduled for specific milestones to help with individual task management and development planning.

NAB (not a bug)
The comment you're likely to see if a developer decides that a perceived bug is actually an intended part of the design. Synonymous with 'as designed'.

NMI (need more info)
The comment you're likely to find if one of your bugs is sent back to you by a dev or QA Lead for more info. Be more specific, write clearer repro steps, whatever it takes for a developer to be able to understand and fix your bug.

Normal map
A layer within a game texture that provides extra information for shading a flat surface. Normal maps make a flat surface appear more bumpy and allow them to respond to light sources with more detail. On models, normal maps are usually derived from a higher detail version of the same model.

Poly (a.k.a. Polygon)
Any single face of a model. A high-poly model is a very detailed model with many different faces, used for models close to the player; the opposite being true of a low-poly model. In the term 'polycount', this word usually refers to triangles (or 'tris'), since when game engines analyse model topography all faces are broken down into triangles as their mathematical properties can be easily evaluated in calculations.

Portal
The ambiguous term some people use to describe a hole in a level's geometry that allows the player to view out of the world. In games development terminology, portal actually means an entirely different thing.

RC (a.k.a. Release Candidate)
The stage at which a game is thought to be ready for a major public release. The game should be feature-complete (as far as features planned for this specific release) and free of major bugs (i.e. all A classes and most if not all B classes). It should be playable without the use of any debug commands (if a multiplayer, any cheats should already have been removed - Hi Valve!). You can have a release candidate for any public version of the game, e.g. you could have a Beta 4.0 RC1 (the first effort at a version of Beta 4.0 for public release) or a v2.0 RC4 (the 4th attempt at a version 2.0 of the game for public release).

Regression
When a developer attempts to fix a bug, the fix must then be checked by a third party (usually a tester). This process is called regression, and is there to ensure that a) the bug is actually fixed in this build, and b) the fix has not created any additional bugs or exploits.

Roadrunner playthrough
A playthrough of a single-player or co-operative game undertaken by a tester to ensure basic functionality throughout and that the game is completable from start to finish. Roadrunners do the bare minimum to progress within the game in the fastest possible time, including avoiding enemies, taking shortcuts, skipping cutscenes and dialogue, even failing scripted sequences or deliberately dying if it saves time.

Soft Lock
If the user cannot freely control their character (i.e. they are stuck or cannot return to the game area) but can still access the game menus, this is known as a soft lock. Soft locks require the user to restart the level or the title to resume control of the game. The majority of soft locks are class A issues.

Specular map
The layer of a texture that defines the reflectivity of a surface to create a mirror effect.

Texture (or texture map)
A 2D image stretched across a single or multiple surfaces in the game world. Figuratively speaking, the 'paint' applied to any surface in the game to give the resulting visual effect. In more modern 3D games, multiple layers of textures are combined on a single flat surface to give the impression of reflectivity, transparency or 'bumpiness'. These layers each do different things and have different names: see Alpha, Diffuse, Normal, Specular.

Version (see also build)
Any time a change is made to the game files and the change is committed to the master game files, the game's version number increases by 1. SVN systems track this number, making it possible to roll back to a stable version if a problem is encountered. With an SVN system your mod will likely run into thousands of builds before completion, since every minor change can be committed to the master files with ease.

Conclusion

So you've read the tutorial (or maybe you haven't and you're just wondering whether you should bother), here's a list of things you should be able to take away with you:
You are no longer a playtester, you are a QA tester
If I hope to have achieved anything with this piece, it's that your average person playing a mod and raising concerns is now attacking the game mechanics and finding ways to really break the game. I'd love to see the day when the average mod or addon you download doesn't have any blockers, any crashes, any missing files, any holes in levels, anywhere the player can get stuck... All of these can be found by actually testing the game. You now have the tools to do just that, so find a mod you like and see if they need any help testing!

Finally testers and developers see eye-to-eye!
Improve the communication between devs and testers! If any devs have read this they should hopefully be documenting their fix attempts and writing proper bug reports (ones that can be found by searching a bug database for keywords!). Similarly, any debutant testers should now have a good understanding of games development terminology and how to write clear and communicative bug reports. Both parties play a key role in a game's development, so it's of the utmost importance that they are able to communicate with eachother effectively.

Professional skills
Finally, all of the information in this tutorial is taken from QA practises used within the industry. By applying these to your mod you're really lifting the bar on your product, whether it's a single map release or a full-blown total conversion. It also means if you do decide to go for a testing job in the industry, with this information in your head you will actually know what you're talking about and you will actually have practised the same skills expected of you in a professional QA department.

I hope you've enjoyed the tutorial and can apply some of these points to your own mod projects. If you have any questions or comments, leave them in the comments area below. Private messages can be sent via the Mod Database or to my email, which is given at the start of the tutorial.

About the Author:
LinkedIn Profile

Extra reading:
'Testing your mod' (QA management tutorial, coming soon)

Post comment Comments
AgeNt_
AgeNt_

very useful for beginning moders! gj

Reply Good karma Bad karma+1 vote
Arxae
Arxae

and for experienced ones to :)
required read for every modder :)

Reply Good karma Bad karma+1 vote
OkeiDo
OkeiDo

Interesting read, I myself have tested a few mods now and still am so I'm quite familiar with some parts in this tutorial. However, it felt like some parts were mostly for single player games/mods but it's good that you managed to cover all parts. Good job and keep your tutorials comming! I will definitely show this to my fellow testers.

Reply Good karma Bad karma+2 votes
Crispy Author
Crispy

Hi! I know it's some time since you made this comment, but I've updated the Bug Hunter section with some ideas for multiplayer testing anyway.

Reply Good karma+1 vote
Ciasco
Ciasco

This is a very good read. I have been testing games for a while, and this clearly explains all of the characteristics of a bug that a game tester must explain to the developer. This is a must read for any developer/game tester, since it shows how much organization can help you in the bug-fixing process.

Reply Good karma Bad karma+1 vote
Gibberstein
Gibberstein

I thought page three was a bit too deep and overcomplicated for most mod teams - it doesn't need to be so formal in a small team. Might be valid for one of the few really large mod teams though.

The rest of it is spot on though - if all bug reports were done to this standard I'd be a very happy developer :)

Reply Good karma Bad karma+2 votes
Crispy Author
Crispy

Funnily enough this piece started off as part of a bigger tutorial, which I then decided to split into one for QA testers and one for QA Leads. When I wrote the introduction to what will now be the management tutorial, I put in a warning that small mod teams don't need to exhaust their efforts on fancy bugtracking software if they're only going to have 500 bugs in their database. There's no point writing up playthrough testplans if all you're changing is some weapon models and variables.

The same is true of testing to a certain degree. As far as a tester is concerned, the main points to focus on whether you're a big or small mod fall into two areas:
1) Bughunt actively, don't wait for bugs to come to you
2) Communicate information with the whole team in mind, whether it's a bug report or a fix.

What parts from this tutorial you choose to apply to your own testing is down to the scale of the project. Ultimately the QA Lead should be setting the benchmarks for your mod's bugging practise, which is what the next tutorial will look at.

Reply Good karma+1 vote
JohnBart
JohnBart

Thank you for making this tutorial Crispy. :-)
I have already sent a link for our beta testers, as there is a LOT to learn from this article.

Personally, being the beta team leader of our mod, I can't wait to read your upcoming QA management tutorial. ;-)

Reply Good karma Bad karma+1 vote
formerlyknownasMrCP
formerlyknownasMrCP

OH MY GOD this is so awesome.

Looking forward to your QA Lead guide

Good work Crispy.

Reply Good karma Bad karma+1 vote
Crispy Author
Crispy

Thanks for the kind words.

If you have any D class bugs of your own, I am taking all feedback on board to work on improving the tutorial and making it more relevant to more teams. Based on the comments so far, I'll be looking at adding in more stuff relevant to multiplayer testing and also trying to make it clearer what is applicable to smaller mods.

Reply Good karma+1 vote
Forceflow
Forceflow

Very interesting read, Crisp.

Reply Good karma Bad karma0 votes
Post a comment

Your comment will be anonymous unless you join the community. Or sign in with your social account: