Developer for Nuclear Dawn mod (May 2005-Dec 2008) :: QA Tester for SEGA Europe (Dec 2007-Aug 2009) :: Production Tester for Splash Damage (Aug 2009-)

Report article Mod Tester's Handbook

A reference guide for testing mods. Includes info on how to find bugs, how to narrow down the steps to reproduce and how to write good bug reports. Also includes a 'game development terminology' reference section.

Posted by Crispy on Jul 25th, 2008 digg this super bookmark Page 4 of 11    
Intermediate QA/Testing.


Bug hunter

Tips for finding and producing bugs
Here are some methods you can adopt to find bugs in your game:

Be a loser
Fail challenges. Get yourself killed. Give up midway through a fight or a scripted sequence. Reload the game. Do you lose a life? What happens if you lose all your lives? Is the game saving properly? When you reload are you in the right place, with all the equipment you had before? Are all the NPCs where they should be? Is your mission progression saved?

Be fast
Find the quickest way to complete the game and keep pushing the boundaries. Can you get stuck in a closing door, or kill an enemy before they are ready to fight you? Can you trigger tutorial messages on top of eachother by completing tasks in record time? Can you get somewhere before the game has had time to draw it, maybe even fall out of the level? Use as many movement exploits as you can find to give you speed boosts, jump boosts and so on. In multiplayer a lot of the time by pushing the game to its limits you will find you can reach an area quicker than any other player or team and benefit from a considerable (and unfair) advantage.

Be curious
Bump into walls. Bump into enemies. Fall off cliffs. Trundle through rivers. Attack walls and lunge into the environment. Try to avoid savepoints, cutscenes, scripted sequences. Jump to supposedly inaccessible areas. Do you fall out of the level? Can you walk through enemies or scenery? Do you get stuck? Do the lighting and other graphics display correctly in all areas? Does the sound play correctly in all areas and on all surfaces?

Be destructive
Break the game mechanics and the game experience. Attack everyone and everything with everyone and everything you can get your hands on: friends, foes, flora, fauna, inanimate objects, breakables, collectibles... everything. Button spam. Spam the pause menu. Spam everything and anything. Trigger as many events as possible at the same time. If you can input text: spam as many characters as possible. Go round the the track the wrong way. Get in people's way. Do things out of sequence (both in-game and in menus). If it's multiplayer: piss people off. Spawncamp, spawnrush, spawnblock: be a hyperactive kid with ADHD.

Single-player versus multiplayer testing
Now, not all of these processes can be applied to multiplayer as equally as single-player. In fact, due to their largely scripted nature on only one machine, it's a lot easier to test singleplayer than multiplayer games that span multiple players on multiple machines all doing completely different things. But, in general, you can apply the same theories to multiplayer, the key difference is that you need to be able to communicate with eachother at all times because a lot of multiplayer bugs come from the fact that people are interacting in unforeseen ways and with unforeseen consequences. For example:

Be a loser
If you don't perform well (for whatever reason: intoxicated, tired, newbie, griefer), will it dramatically affect the enjoyment of the rest of the players? If you get off to a slow start, can you still complete with other players? If you join the game late, can you still compete with other players who have been playing the match from the start? If you die lots does the game get more difficult or are you able to able to improve as you play regardless of number of deaths?

Be fast
Try to discover ways to increase your movement speed. Can weapon knockback be used to gain an advantage? (e.g. rocket-jumping) Can common engine exploits be taken advantage of in your mod? (e.g. wallstrafing, bunnyhopping, strafejumping, scripts) What about running to the enemy as fast as possible; does both side arrive in the middle of the level at the same time, or is there a route that needs adjusting to balance things?

Be curious
Try changing settings during a match. Try quitting and re-joining games; could this be used to gain an advantage in any way? Try joining the other team. Try using abilities at distance; is it balanced? Do third-person animations play correctly and match what the other player is seeing first-hand on their screen? Can all victory conditions be met as expected? Are the victory conditions communicated clearly to beginner players? Can you use any materials (or even other players) to access anywhere you shouldn't be able to? Is all the information you are reading correct? (e.g. scoreboards, weapon/ability/class descriptions)

Be destructive
What happens if you quit the match at different times? What happens if everyone in the game performs an action at the same time? (shoot, micspam, connect to a server, quit the game, change team/class, etc.) Can you spawncamp? Can you obstruct your own teammates? Can you block any paths unfairly? Can you kill teammates? (if so, is there anything in place to deter you from this?). If you go AFK or deliberately hide away somewhere, does the gameplay suffer?

Comments
AgeNt_
AgeNt_ Jul 25 2008, 1:59pm says: Online

very useful for beginning moders! gj

+1 vote     reply to comment
Sigma
Sigma Jul 25 2008, 4:09pm replied:

and for experienced ones to :)
required read for every modder :)

+1 vote     reply to comment
OkeiDo
OkeiDo Jul 25 2008, 7:21pm says:

Interesting read, I myself have tested a few mods now and still am so I'm quite familiar with some parts in this tutorial. However, it felt like some parts were mostly for single player games/mods but it's good that you managed to cover all parts. Good job and keep your tutorials comming! I will definitely show this to my fellow testers.

+2 votes     reply to comment
Crispy
Crispy Feb 16 2009, 3:23am replied:

Hi! I know it's some time since you made this comment, but I've updated the Bug Hunter section with some ideas for multiplayer testing anyway.

+1 vote     reply to comment
Shiggy
Shiggy Jul 26 2008, 1:57am says:

This is a very good read. I have been testing games for a while, and this clearly explains all of the characteristics of a bug that a game tester must explain to the developer. This is a must read for any developer/game tester, since it shows how much organization can help you in the bug-fixing process.

+1 vote     reply to comment
Gibberstein
Gibberstein Jul 26 2008, 10:02am says:

I thought page three was a bit too deep and overcomplicated for most mod teams - it doesn't need to be so formal in a small team. Might be valid for one of the few really large mod teams though.

The rest of it is spot on though - if all bug reports were done to this standard I'd be a very happy developer :)

+2 votes     reply to comment
Crispy
Crispy Jul 27 2008, 10:11am replied:

Funnily enough this piece started off as part of a bigger tutorial, which I then decided to split into one for QA testers and one for QA Leads. When I wrote the introduction to what will now be the management tutorial, I put in a warning that small mod teams don't need to exhaust their efforts on fancy bugtracking software if they're only going to have 500 bugs in their database. There's no point writing up playthrough testplans if all you're changing is some weapon models and variables.

The same is true of testing to a certain degree. As far as a tester is concerned, the main points to focus on whether you're a big or small mod fall into two areas:
1) Bughunt actively, don't wait for bugs to come to you
2) Communicate information with the whole team in mind, whether it's a bug report or a fix.

What parts from this tutorial you choose to apply to your own testing is down to the scale of the project. Ultimately the QA Lead should be setting the benchmarks for your mod's bugging practise, which is what the next tutorial will look at.

+1 vote     reply to comment
JohnBart
JohnBart Jul 26 2008, 3:47pm says:

Thank you for making this tutorial Crispy. :-)
I have already sent a link for our beta testers, as there is a LOT to learn from this article.

Personally, being the beta team leader of our mod, I can't wait to read your upcoming QA management tutorial. ;-)

+1 vote     reply to comment
Mr_Cyberpunk
Mr_Cyberpunk Jul 27 2008, 6:15am says:

OH MY GOD this is so awesome.

Looking forward to your QA Lead guide

Good work Crispy.

+1 vote     reply to comment
Crispy
Crispy Jul 27 2008, 10:17am says:

Thanks for the kind words.

If you have any D class bugs of your own, I am taking all feedback on board to work on improving the tutorial and making it more relevant to more teams. Based on the comments so far, I'll be looking at adding in more stuff relevant to multiplayer testing and also trying to make it clearer what is applicable to smaller mods.

+1 vote     reply to comment
Forceflow
Forceflow Aug 1 2008, 8:53am says:

Very interesting read, Crisp.

+1 vote     reply to comment
Post a Comment

Only registered members can share their thoughts. So come on! Join the community today (totally free) and do things you never thought possible.

Level
Avatar
Avatar
Offline Since
Nov 14, 2009
Country
United Kingdom United Kingdom
Gender
Male
Age
25
Member Watch
Track this member
Tutorial
Browse
Tutorials
Report Abuse
Report article
Bookmark
Digg Super bookmark