Developer for Nuclear Dawn mod (May 2005-Dec 2008) :: QA Tester for SEGA Europe (Dec 2007-Aug 2009) :: Production Tester for Splash Damage (Aug 2009-)

Report article Mod Tester's Handbook

A reference guide for testing mods. Includes info on how to find bugs, how to narrow down the steps to reproduce and how to write good bug reports. Also includes a 'game development terminology' reference section.

Posted by Crispy on Jul 25th, 2008 digg this super bookmark Page 6 of 11    
Intermediate QA/Testing.


Writing a Bug Report: The Bug Summary

The bigger the project, the more important the summary. The purpose of the summary is not only to give a concise description of the bug, but also to provide keywords that will allow the same bug to be found in the database at a later date. The below example is a fairly professional method of doing it. Cherry-pick any useful information you find in this section, but it doesn't necessarily have to be replicated. If you have testers that won't be able to follow a single format, use a system that you know will be adopted readily and can work for your team.

A typical summary in a good professional bug database might look something like this:

[GFX] E1_L02: Emergence > Docks - Northernmost crane model has incorrect texture (black)

This summary contains a vast amount of information that tells me, in just one line:
- what area of development the bug affects
- its location within the game
- which part of the aforementioned location if affected
- exactly which asset is affected
- and the symptoms the issue is presenting

Here the issue is that an environment model (a crane) is showing up as completely black in-game for some reason. The tester may or may not think they know the reason for this, as far as the summary goes this is irrelevant, the focus needs to be on communicating the essential pieces of information in an objective, clear and searchable format.

[GFX] E1_L02: Emergence > Docks - Northernmost crane model has incorrect texture (black)

I have added some colours to the example summary to highlight the different key areas. When using a basic (usually web-based) bug database or forums, both of which rely heavily on text-based searches, the more information you can put into the summary line, the better. On a forum, the summary line will obviously be the title of the thread.

[ ] : > - (structure)
These symbols are used to give the summary a uniform structure and allow the eye to locate different pieces of information quickly. Every bug always includes certain items of info, and these items always appear in the same order. The symbols are there to separate these pieces of info and to guide the eye through the summary line. When creating your summary format you can use any symbols in any order, so long as they are easy to remember and do the job of making the summary clear.

GFX (category)
To help with searches and search filters, you can choose some important information to put at the beginning of the summary. The category of the bug (i.e. which area of functionality it affects)
is very useful, especially for developers trying to locate bugs that affect their own work. With this information at the beginning of all bugs, I can search via a keyword like "black", and then alphabetise all the results to bunch up all the bugs by category from 'AI' to 'SFX'. If I see this bug in a list of perhaps 20 other bugs, I immediately know that this one in particular is a graphics issue, so as a reader I am well situated to understand the rest of the summary that bit better.

E1_L02 / Emergence / Docks (location)
This information leads you through the game to the location of the bug. In this case you have the location of the bug within the game world itself: "E1_L02" means it is in the 2nd level of the first episode, "Emergence" is the title of the level/mission/quest, "Docks" is the actual physical location within the game world. It's important to have both "Emergence" and "E1_L02" in the title because while the name of the level is easier to remember, the label will be more useful when searching through folders or code and it also contains information about its location within the game's progression flow.

This part of the summary isn't only for locations within a level, though, it is used to give information about the location of the bug within the game.
A bug could be located within a particular feature: "General: Classes > Soldier -"
or a menu: "Front End: Options > Sound -"

Northernmost crane model has incorrect texture (black) (summary)
Although the summary is only 7 words long, it contains 4 adjectives to link the 2 nouns (the 'texture' and the 'model'). There is information about what generally is affected (a model of a crane), which specific crane model is affected (the one furthest north), and how specifically the model is affected (that the texture is somehow wrong and showing up black).

A less meaningful way of summarising the bug would be "crane in docks is black". This simply doesn't convey as much information; it doesn't tell an artist or level designer which of the multiple cranes in the docks area has an error, and it doesn't describe the error in much detail. Saying the
crane is black could mean that it is in shadow and there is a problem in fact with the lighting in the area.

Don't be afraid to add keywords in parentheses if you think they will help other team members find the bug in the database. You don't have to masterfully work them into the summary sentence, it's fine to add them in brackets so long as the meaning is clear. Finally, if you can economise the word count in the summary without compromising the perceived meaning, it helps to keep things short. Removing superfluous words like "the" and "is/are" and letting an adjective or past participle follow the noun is usually acceptable. You can also sometimes eliminate "which"/"that" by use of the gerund/present participle.

E.g. "The door is broken before the player activates the switch which stops progression" could be turned into:

"Door broken before switch activated, blocking progression"

Comments
AgeNt_
AgeNt_ Jul 25 2008, 1:59pm says:

very useful for beginning moders! gj

+1 vote     reply to comment
Sigma
Sigma Jul 25 2008, 4:09pm replied:

and for experienced ones to :)
required read for every modder :)

+1 vote     reply to comment
OkeiDo
OkeiDo Jul 25 2008, 7:21pm says:

Interesting read, I myself have tested a few mods now and still am so I'm quite familiar with some parts in this tutorial. However, it felt like some parts were mostly for single player games/mods but it's good that you managed to cover all parts. Good job and keep your tutorials comming! I will definitely show this to my fellow testers.

+2 votes     reply to comment
Crispy
Crispy Feb 16 2009, 3:23am replied:

Hi! I know it's some time since you made this comment, but I've updated the Bug Hunter section with some ideas for multiplayer testing anyway.

+1 vote     reply to comment
Shiggy
Shiggy Jul 26 2008, 1:57am says:

This is a very good read. I have been testing games for a while, and this clearly explains all of the characteristics of a bug that a game tester must explain to the developer. This is a must read for any developer/game tester, since it shows how much organization can help you in the bug-fixing process.

+1 vote     reply to comment
Gibberstein
Gibberstein Jul 26 2008, 10:02am says:

I thought page three was a bit too deep and overcomplicated for most mod teams - it doesn't need to be so formal in a small team. Might be valid for one of the few really large mod teams though.

The rest of it is spot on though - if all bug reports were done to this standard I'd be a very happy developer :)

+2 votes     reply to comment
Crispy
Crispy Jul 27 2008, 10:11am replied:

Funnily enough this piece started off as part of a bigger tutorial, which I then decided to split into one for QA testers and one for QA Leads. When I wrote the introduction to what will now be the management tutorial, I put in a warning that small mod teams don't need to exhaust their efforts on fancy bugtracking software if they're only going to have 500 bugs in their database. There's no point writing up playthrough testplans if all you're changing is some weapon models and variables.

The same is true of testing to a certain degree. As far as a tester is concerned, the main points to focus on whether you're a big or small mod fall into two areas:
1) Bughunt actively, don't wait for bugs to come to you
2) Communicate information with the whole team in mind, whether it's a bug report or a fix.

What parts from this tutorial you choose to apply to your own testing is down to the scale of the project. Ultimately the QA Lead should be setting the benchmarks for your mod's bugging practise, which is what the next tutorial will look at.

+1 vote     reply to comment
JohnBart
JohnBart Jul 26 2008, 3:47pm says:

Thank you for making this tutorial Crispy. :-)
I have already sent a link for our beta testers, as there is a LOT to learn from this article.

Personally, being the beta team leader of our mod, I can't wait to read your upcoming QA management tutorial. ;-)

+1 vote     reply to comment
Mr_Cyberpunk
Mr_Cyberpunk Jul 27 2008, 6:15am says:

OH MY GOD this is so awesome.

Looking forward to your QA Lead guide

Good work Crispy.

+1 vote     reply to comment
Crispy
Crispy Jul 27 2008, 10:17am says:

Thanks for the kind words.

If you have any D class bugs of your own, I am taking all feedback on board to work on improving the tutorial and making it more relevant to more teams. Based on the comments so far, I'll be looking at adding in more stuff relevant to multiplayer testing and also trying to make it clearer what is applicable to smaller mods.

+1 vote     reply to comment
Forceflow
Forceflow Aug 1 2008, 8:53am says:

Very interesting read, Crisp.

+1 vote     reply to comment
Post a Comment

Only registered members can share their thoughts. So come on! Join the community today (totally free) and do things you never thought possible.

Level
Avatar
Avatar
Offline Since
Aug 16, 2009
Country
United Kingdom United Kingdom
Gender
Male
Age
25
Member Watch
Track this member
Tutorial
Browse
Tutorials
Report Abuse
Report article
Bookmark
Digg Super bookmark