Ever had that itching desire to break out of hell and into the highest reaches of heaven? In Party of Sin, you take control of the Seven Deadly Sins in a coopetitive puzzle-platformer for 1-4 players. Master a large, dynamic team of anti-heroes -- the Seven Deadly Sins -- as you forge your destiny on a quest to troll humanity. Envy, Greed, Sloth, Pride, Wrath, Lust, Gluttony are all multi-dimensional with special powers useful in many situations. Swap characters on the fly as you change tactics based on the situation: ALL the Sins are useful in combat, ALL the Sins aid in puzzle solving, and ALL the Sins have coop interactions, both Good and Evil. Adventure: A full 6 to 8 hours of gameplay is provided, in both solo and coop modes. Play over 20 levels on your adventure through Hell, Purgatory, Earth and Heaven. Upgrade your Sins in the shop by collecting God's forbidden apples. Five bosses like the Demon Narwhal and the Airship Captain stand in your way.

Report article RSS Feed Test Data Collection and Analytics

Attention all indie development teams stuck in development hell (who isn't?!): here's a 3-step guide on how to crank up the objectivity and productivity of your development process a couple notches through the use of thorough, statistical test-result gathering.

Posted by AlfredG on Jun 6th, 2012

The core change to our design pipeline I mentioned last week was the incorporation of more testing. The idea is that building confidence in our level design through testing will save us a lot of work in the long run and also help establish clear goals for design as we wrap up production. As part of this new effort, we've written some new test procedures and I've also worked on some stats gathering. I want to explain that a little more clearly in this blog post.

Some Definitions

Creator / Designer: The person who built the level and will make changes to it
Observer: The person administering the test, taking notes
Player / Participant: The person actually playing the game

1-Record Everything

We used to rely on handwritten notes or accounts of tests in order to make judgements. The main problem with this is that the creator never gets to see what the player actually did, and has to rely on what the observer has noted (unless they happen to be the same person). This added a lot of subjectivity to the test results and also meant the creator was working blind. You don't really realize how much of a problem this is until you implement the solution: Record Everything.


Hard Drives these days are cheap and we all have plenty of disk space. With tools like FRAPS and a run-of-the-mill webcam, you can gather a ton of data on someone playing. In our case we use the webcam to record the player's facial expressions and we use FRAPS to record the game itself.

The webcam model I chose. Logitech C920.

Instead of relying on notes and accounts from observers, I'm now advising that everyone on the team record their playtests. The creator is able to review the video himself (at 2x or 4x speed to save time) and our observers no longer need to take notes and can instead interrogate the player about what they are thinking. There have already been instances where a player has solved a puzzle in an unintended way, and the creator never would have known from the observer's notes. Having the video on hand makes everything simpler.

2-Data Gathering

Part of the new test procedure involved a data-sheet on the player. We wanted to know who the player was, what skill level of gamer they considered themselves to be, what platforms they used, etc. This data would help us make important design decisions. Along with this player info, we also wanted to gather data on the gameplay itself. How long did a player spend as Envy? How many times did they die? This is doable using our recorded video, but unfortunately it's pretty tedious.

This week, I automated this process using a quick-and-easy PHP service and some C# WebClient calls from the game. The game client now sends hundreds of events every minute and they get stored in a central database. We can use this data to easily calculate how much time a player spends as each sin, or how many times they died. We can even cross-tab this data by location and find the hot spots or time-consuming parts of the game.

The message format is fairly simple. The game sends event, each event containing the timestamp, current location and sin of the player that sent it, as well as an event type and detail field. This gets dumped into a MySQL database by a PHP script. I found out early on that spamming our web-server every few seconds is bad, so I implemented event batching.

Here is the source code for the PHP service. You can guess that the C# side of things is fairly simple. The code uses the UploadValuesAsync method to pass along parameters whenever there is an event we want to track.

3-Analysis

Now we're sitting on a ton of data, but we need to analyze it somehow. This is the step I'm at now. I've downloaded RapidMiner and started analyzing the data. I have a background in Business Intelligence so this isn't completely new to me, but game data has some new challenges.

Here is what a test run of hell3 looks like in RapidMiner. As you can see the boss near the end has a much higher point density. This is because players kept dying to the Narwhal!


The main information I want to get out of our test runs are reports like:

  • Percentage of time spent as each sin
  • # of deaths by location
  • # of deaths by sin
  • Average time spent at specific puzzles
  • Average time spent at battles
  • Average time to complete a level
  • Location of jumps leading to deaths

I'll be working on producing these types of reports in the next few days and will let you all know how it's going!

Dan

Post comment Comments
Fib
Fib Jun 7 2012, 2:04pm says:

Great post! Some indie developers don't realize the importance of testing, test data collection, and statistical analysis on the test data. It can be a lot of work but it will help the developer make better decisions about the design. It's very important!

+2 votes     reply to comment
LowBit
LowBit Jun 12 2012, 6:20pm says:

Your Business Intelligence background is really paying off. We've been running some playtest sessions on my own project, but nothing even half as sophisticated as this. I'll definitely be shoving this article under the noses of the rest of my team.

More, please!

+2 votes     reply to comment
Post a Comment
click to sign in

You are not logged in, your comment will be anonymous unless you join the community today (totally free - or sign in with your social account on the right) which we encourage all contributors to do.

2000 characters limit; HTML formatting and smileys are not supported - text only

Icon
Party of Sin
Platform
Windows
Developer
Crankshaft Games
Publisher
Surprise Attack
Engine
Custom Built
Contact
Send Message
Official Page
Partyofsin.com
Release Date
Released Dec 11, 2012
Game Watch
Track this game
News
Browse
News
Report Abuse
Report article
Related Games
Party of Sin
Party of Sin Single & Multiplayer Platformer
Related Engines
Custom Built
Custom Built Commercial Released Sep 1, 2007
Related Groups
Crankshaft Games
Crankshaft Games Developer & Publisher with 7 members