Proper Testing Setup and Operation tutorial

[page=Introduction>

I'll start this off giving some of my experience, just so you know that I know what I'm talking about. I've beta/alpha tested these games/mods, in no particular order. Massive Assault Network, Xpand Rally, Joint Operations, Lineage 2, Far Cry, Call of Duty, Call of Duty: United Offensive, Natural-Selection, Desert Combat, War of the Ring, Joan of Arc, Horizons, Guild Wars, Kohan 2, There, Wish, and Ragnarok Online.

I am aiming this to be more for the leader, or QA lead for your mods. So let's begin!!!

[page=Understanding Gameplay and setting up a Testing Team]
First thing first, understand the gameplay. When you test something you should know whether something is a bug or a feature. You must understand everything about the design, because when you are going through bug reports, you must know which are true bugs, and which are some features of the game (and those you pass off to the appropriate designer if you feel that the feature is not presented correctly). If you have a design document, read it multiple times. Question things, and figure out why things are in the design doc. It is always good to be curious in these situations. If you feel that it won't contribute to anything, then it should be cut. Don't waste precious development and testing time later on, which if you would have examined it, realized it could have been cut early on.

Second, set up for a testing team. Since you now understand the design document, it's time to set up to receive testers. The best method, of course is forums. Here is a guide line to what your forums should be like. Customize them to what you seem is right, or what goes with the flow. The key is that you want everything to be organized, easy to find, and you won't have to go searching through tons of sub-forums to find them.

Beta Forums
-General
- News, Announcements, Files
- General Discussion and Feedback
- Suggestions
- Questions
- Technical Support
-Bug Forums
- Coding Bugs
- Mapping Bugs
- Make individual topics for each map to keep it clean
- Gameplay/Balancing
- Sounds and Graphics
- Missing/Broken Files

Your forums should look something similar to that. Of course, if you want to keep your forums smaller, than you should probably just do something like this.

Beta Forums
- Files and Discussion
- Bugs

I do not recommend that method, but if you need to hold back on sub-forums, then you gotta do what you gotta do.

This is also where the planning for how/when you want to test. Ask yourself a few questions

Do I want to run weekly playtests?
How many testers should I need? *
Will there be enough servers?
How will I gather feedback?

* = You should make this relative to the number of servers. If you have one server, with 20 player slots, then you should do slightly lower than double, or a maximum of double. So for one server, with 20 player slots, you should have a MAX of 40 testers. I do recommend in the 30's for that situation however.

[page=Assembling the Team]
Third, now you need to assemble a team. I do not recommend going on your average Counter-Strike forum and asking for play testers, no offense to Counter-Strike forums users, or players, as they usually will just play to play. You need people who have experience testing games/mods. Also, your fans are an excellent place to go. If you see someone on your forums who contributes to suggestions, feedback (important!), etc etc then contact them for becoming a tester, ASAP! If they are your true fans, then they will give you good feedback, and you can see how well the mod meets their expectations. In fact, I would suggest simply asking them if they would be willing to test, and not make them go through the sign-up procedure as they have proven theirselves.

Produce a Sign-Up sheet on your site. How you do that is your business, but I suggest not making it go to your personnel home e-mail account. Here are some questions that should definetely be on there.

Real name:
Alias:
CPU:
RAM:
Graphics Card:
Sound Card:
Internet Connection:
Have you ever tested before? If so, name what you have tested.
How many hours per week can you devote to testing?
Write an example of a bug report (can be real or fake).

The last three questions are key. They filter out the players from the testers.

Now you will probably be wondering, what's the difference between a player and a tester?

Well the answer is simple. A player simply goes in a game, and only wants to win. Their mindset is along the lines of "Come on, this is a game and let's just play! I don't want to test balance right now, I want to kill those monkeys!".

A tester is someone who will take themselves ouf of the game to test something. It may seem boring, but it does wonders for your mod. Here is the mindset of the tester. "Okay guys, cease firing. We need to do some testing on wheter the benefits of walking or less than the benefits of crouching. Meet at point X all teams."

In the end, players will be the ones who should be playing your game. Testers are the ones that prepare it for the players. I hope you can understand that concept.

When you sort through these tests, keep the number of testers you want in mind. When you read them, take the select few's application, and fliter it into a special place. It is okay if you filter in more than the number of testers you want. If you have less than, simply go through the applications again, and find the top of those that are available. If you think it is safe to go under what you want (maybe some applications are widely decorated) then you can. It really is your call.

For the hardware, you want variation. Find a very low comp, and find a very high-end comp. Of course, everything in between. You need to see if the game even runs on those machines, because if your mod is dependent on custom shaders, and you need a DirectX 8 card for them as you can't get it any lower, you need to find that early on. Then once you are 100% sure there is no other way, tell the community, give them the reason, screenshots, whatever. You need to show them that there was no way around it, and if it was removed for the lower end cards, the gameplay itself would change.

For the "What have you tested" part, just because they say "no" doesn't mean they wouldn't be a good tester. If you find someone that is highly decorated with retail games, put them in the special place, and examint it throughly later.

For the hours part, you want someone who can do a minimum of roughly 10-15 hours. That is between one to two hours a day, which isn't extreme in testing.

The last question is a huge one. If you find that somebody has never tested before, but everything else is looking good, and they can report a bug nicely, then let them in. They need to get experience somewhere. If you notice that someone has a highly decorated tested games list, but a very sloppy bug report, I would question wheter he is a true tester/player, or whether he made the tested games up to improve his chances.

[page=Organizing playtesting]
Fourth, organizing playtesting. This part can get really tricky. Especially if you have more than two teams facing each other. A method which is used a lot in MMORPG's is that at each testing session they test different character "builds" which is a predefined character set-up purposely made to try to find imbalances. For example:

Okay testers, at tomorrow's playtest I want half of you to join Team A, and get Equipment X, and the other half go Team C with Weapon Z with Attachment Y.

Excuse the variables, but you get the point. The whole point of testing is trying to break the game, and imbalance it.

I would suggest having an organized playtest bi-weekly, or weekly if you have lots of time. You also need to remind them. E-Mail them the day before reminding them. Don't expect them to read and sign-up for playtests in a thread in the forum, because some do forget that they are testing the mod and you can bring them back.

[page=Running the playtest]
Fifth, running the playtest. There should be careful thinking and testing that you should do before-hand. Don't throw out random things, think about how everything works together, and figure out if you can imbalance the equation. And no, I'm not talking about The Matrix here people!

You should tell everyone what you plan to do, a few days before hand. That way they can get a feel for what they will be using so that they can use it like a proficient player would, and then you really know what it will be like. Make a nice big post about it somewhere, explaining why you choose it, and what you plan to proove. Also, tell them what you want them to do during the playtest, strategy-wise.

You and the developers should be there 1 hour, to 30 mins before testing begins. Then a few mins before the test begins (or if the testers are in the middle of a hot game, let them finish it, as they usually go "player mode" before testing to get it out of their system, lol.). You should talk with the testers, find what's on their mind, answer questions etc etc. Also, before you officially begin, remind them of the character builds, and strategies you want used that night. Then, do your thing once you know everyone is set and understands.

[page=Gathering Feedback and builds]
Sixth, Gathering Feedback. Once your playtest is over, make a thread for comments on the playtest. Do not post your feelings in that thread, as some testers sometimes will go "yea I agree" without reading it, since they trust you as the leader. Let them figure things out and don't put ideas into their heads. After a day or so, post your feelings and comment on what other people think.

If you feel a change is worthy of getting changed, edit the post and put a * in the topic line, because you and the devs should have admin control over the beta forums. Once the bug has been fixed but a ! in the topic. If a tester posted a bug that has been fixed already, or is being fixed tell them. Do not lock/delete threads in the beta forums, unless it violates ToS! That is important. Some bugs do come back, and nip you in the heiny. Of course, you don't want a whole new thread, as you may mistaken it for a whole new bug, with different explanations of the bug and such.

Seventh, Distributing a new build This is a topic which is challenged by some. Some mod teams always think their mod will be leaked. If you choose the right people, it won't. One thing you need to do is show that you trust your testers, or they won't trust you! Do not take away the ability to run a server for example.

The build should be kept on a password protected FTP, and should be zipped up. If you can do a small patch, do it. Some testers don't like downloading 100 megabyte files over, and over, and over.

You need to do a little internal testing. Maybe a days worth, or maybe even a few game rounds worth. Make sure it doesn't crash right away, nor there are any blatent bugs that are an easy fix. Delay a build if you feel that it would create insufficient testing environment. The mod is buggy, so if you notice a small graphic glitch that would require re-texturing, let it slide until next build/patch.

Do not let users make a .torrent, nor distribute over any P2P, unless it is private and you control it. WASTE is an example of a private system which you can control. Use google if you want more information on it. FTP should be your first option, but make sure you won't go over your bandwith limit!!!

So you think you are done? Now you want to release your mod, as version 1.0. When do you know it's ready? Well, you should be able to go weeks without crashing. Minimum of 2-3. A crash on a player's computer is not nice, and their quality bar lowers for what they think of your mod. If you still have crash bugs, fix them before release. Maps should feel balanced. They should have variance, and they should all feel distinct. If in your testing, you notice on one map, Team A always wins, do more testing on the map to make it balanced. The game should generally feel balanced. You should be able to play a game, and when you die you should be able to realize that you had a fighting chance, but failed and not get mad because the gun is way too inaccuate. Note however that there is a difference between rampant bitching, and balance observations. Some get mad and need to blame it on something. In those situations, don't really bother unless a lot of people are complaining, then you know they are rampantly bitching about balance, which isn't all that great.

You should have almost no sound and sprite errors. Graphical bugs are fairly tricky. If objects are popping, or clipping try what you can to stop it, but usually that is the engine's fault.

You should go through few release candidates. If you have like 10 or so release candidates, then I would stop calling them release candidates, or stop do a lot more for the next release candidate and release it. You should have the final release candidate for a week or two, to make sure it feels right. There most likely will be bugs gone unchecked, but that is why you can make patches. You should closely follow these steps for testing patches as well, but don't give them a new build like a week after 1.0 is released... Let them have fun and be in "player-mod" for a few weeks. Let them enjoy what they have tested. Make sure to give all the testers a special thanks in your credits!!!

Rmember though, Gameplay > ALL

Achieving perfect balance is bad, by the way. That means when one player slips up, it could cause a snowball effect. Have it as close to being balanced as you can. I consider being balanced as having every team in the game be able to win, having fun, and if you lose it's because you know the other team was better and not that it was the game's fault.

[page=Conclusion]
It looks like my fingers are starting to bleed, and I can't think of too much more stuff to add to this. I am not going to explain how to test your mod, as it always changes. I mean always. However, there should be a few mandatory engine bugs that you should look for and see if you can correct. Sometimes an engine is picky about certain things with all mods, and you should be on the look out.

Thank you for reading, please do not hesitate to PM me via ModDB for any questions you all may have. I'm very open to all types of questions on testing. :)

~Iced_Eagle