Developer for Nuclear Dawn mod (May 2005-Dec 2008) :: QA Tester for SEGA Europe (Dec 2007-Aug 2009) :: Production Tester for Splash Damage (Aug 2009-)

Report article Mod Tester's Handbook

A reference guide for testing mods. Includes info on how to find bugs, how to narrow down the steps to reproduce and how to write good bug reports. Also includes a 'game development terminology' reference section.

Posted by Crispy on Jul 25th, 2008 digg this super bookmark Page 9 of 11    
Intermediate QA/Testing.


Writing a Bug Report: Steps to Reproduce

Once you have narrowed down your repro steps you can put them into the bug report. Remember to number the steps and to begin them with the booting of the game. Never, under any circumstances, give instructions for using the debug menu in your repro steps. Firstly, once a game has entered Beta, you should never put a bug in that was found via use of a debug menu (it's extremely bad practice). Secondly, your steps should always be directed towards a player of the game, not to a developer. If you always imagine that a player will be reading your repro steps, they will always be clear and accurate. Thirdly, it is good practise that regression of the bug should be performed without the use of debug commands. This means if the bug is on the last level of a game, a full playthrough is required to reliably confirm if it is fixed or open.

Here's an example of the format I would use to write a bug:

Steps to Reproduce:
1. Boot [title name] [platform] [settings]
2. Load [level]
3. Move to [location]
4. [perform action]

Result:
[results]

Your first line should always be an instruction to boot the game on a certain platform. This is to avoid any important steps being missed out, such as booting in a certain language or at a certain resolution. If the bug does require a certain setting in order to be reproduced, add that information to this line. If the bug can be performed in any resolution, or in any language, add this as a note to the bug description.

Unless the bug occurs immediately after boot (e.g. on the title screen or on the Front End), your second line will usually be an instruction to load a particular level. This line should never involve debug commands. For this reason it is better to phrase this line as "Progress to mission 3" instead of "Load e1_m3", because it makes it clear you have found the bug during normal play. It is acceptable to attach a save file to the bug report that will allow a developer to recreate and observe the bug more quickly. In this case it's fine to say "Load bug1364.sav (see attachments)", just remember to try to re-name the save file using the same numerical ID as the corresponding bug entry in the database (it just makes it easier for developers to find it if they download and save it into a local folder).

Finally, it helps to add a line for the 'result' of the repro steps. This reinforces the overall reproduction process and makes it easy to copy+paste this section of the bug report if needed.

Here's a full example of some repro steps:

Steps to Reproduce:
1. Boot AdventureQuest PC
2. Progress to Denbrooke
3. Talk to Whipsnade to add the Caves to your map
4. Travel to the Caves via the map
5. Collect the Silver Key key from the central cavern
6. Locate the Prison gate within the caves and open it using the Silver Key
7. Quickly kill the guard menacing the civilian (lunge attack works best for this)

Result:
The civilian is not slain and remains in a 'cowering' pose, even though the guard is dead and no enemies are in sight.

As you can see from the above example, no assumptions have been made about the reader. Literally any player of the game should be able to reproduce the bug under their own steam, without needed to query any of the steps. Now, prepare to get amourous with your devs! If you are writing clear repro steps for them, they will love you for it!

Comments
AgeNt_
AgeNt_ Jul 25 2008, 1:59pm says:

very useful for beginning moders! gj

+1 vote     reply to comment
Sigma
Sigma Jul 25 2008, 4:09pm replied:

and for experienced ones to :)
required read for every modder :)

+1 vote     reply to comment
OkeiDo
OkeiDo Jul 25 2008, 7:21pm says:

Interesting read, I myself have tested a few mods now and still am so I'm quite familiar with some parts in this tutorial. However, it felt like some parts were mostly for single player games/mods but it's good that you managed to cover all parts. Good job and keep your tutorials comming! I will definitely show this to my fellow testers.

+2 votes     reply to comment
Crispy
Crispy Feb 16 2009, 3:23am replied:

Hi! I know it's some time since you made this comment, but I've updated the Bug Hunter section with some ideas for multiplayer testing anyway.

+1 vote     reply to comment
Shiggy
Shiggy Jul 26 2008, 1:57am says:

This is a very good read. I have been testing games for a while, and this clearly explains all of the characteristics of a bug that a game tester must explain to the developer. This is a must read for any developer/game tester, since it shows how much organization can help you in the bug-fixing process.

+1 vote     reply to comment
Gibberstein
Gibberstein Jul 26 2008, 10:02am says:

I thought page three was a bit too deep and overcomplicated for most mod teams - it doesn't need to be so formal in a small team. Might be valid for one of the few really large mod teams though.

The rest of it is spot on though - if all bug reports were done to this standard I'd be a very happy developer :)

+2 votes     reply to comment
Crispy
Crispy Jul 27 2008, 10:11am replied:

Funnily enough this piece started off as part of a bigger tutorial, which I then decided to split into one for QA testers and one for QA Leads. When I wrote the introduction to what will now be the management tutorial, I put in a warning that small mod teams don't need to exhaust their efforts on fancy bugtracking software if they're only going to have 500 bugs in their database. There's no point writing up playthrough testplans if all you're changing is some weapon models and variables.

The same is true of testing to a certain degree. As far as a tester is concerned, the main points to focus on whether you're a big or small mod fall into two areas:
1) Bughunt actively, don't wait for bugs to come to you
2) Communicate information with the whole team in mind, whether it's a bug report or a fix.

What parts from this tutorial you choose to apply to your own testing is down to the scale of the project. Ultimately the QA Lead should be setting the benchmarks for your mod's bugging practise, which is what the next tutorial will look at.

+1 vote     reply to comment
JohnBart
JohnBart Jul 26 2008, 3:47pm says:

Thank you for making this tutorial Crispy. :-)
I have already sent a link for our beta testers, as there is a LOT to learn from this article.

Personally, being the beta team leader of our mod, I can't wait to read your upcoming QA management tutorial. ;-)

+1 vote     reply to comment
Mr_Cyberpunk
Mr_Cyberpunk Jul 27 2008, 6:15am says:

OH MY GOD this is so awesome.

Looking forward to your QA Lead guide

Good work Crispy.

+1 vote     reply to comment
Crispy
Crispy Jul 27 2008, 10:17am says:

Thanks for the kind words.

If you have any D class bugs of your own, I am taking all feedback on board to work on improving the tutorial and making it more relevant to more teams. Based on the comments so far, I'll be looking at adding in more stuff relevant to multiplayer testing and also trying to make it clearer what is applicable to smaller mods.

+1 vote     reply to comment
Forceflow
Forceflow Aug 1 2008, 8:53am says:

Very interesting read, Crisp.

+1 vote     reply to comment
Post a Comment

Only registered members can share their thoughts. So come on! Join the community today (totally free) and do things you never thought possible.

Level
Avatar
Avatar
Offline Since
Nov 14, 2009
Country
United Kingdom United Kingdom
Gender
Male
Age
25
Member Watch
Track this member
Tutorial
Browse
Tutorials
Report Abuse
Report article
Bookmark
Digg Super bookmark