Hello and welcome again! The survey has been going for about a month now, and I think it is time for a preliminary analysis of the results I have received so far.
The good news is I have received a lot of general feedback, regarding the gun and it’s mechanics. As of writing this, over 20 people have given shared thoughts after completing the first test chamber, and even more have commented on reddit, facepunch and in private messages.
The bad news is, the overwhelming majority of players didn’t really stick around. The reported numbers drop dramatically after the first chamber. According to my data, only one person have actually gotten farther then test chamber 4 and completed the mod, and even that person have skipped test chamber 5! (that was my favourite chamber :c)
After a bit of thinking, I suspect this is primarily due to the learning curve being a much more important factor then I have originally considered. It is no coincidence that the players start to quit right around the time where I switch from “tutorial” to “proper” puzzles. The original Portal spends plenty of time establishing it’s mechanics before throwing the player into deep water, and it seems like my mod should do the same. That’s why I’m not very worried about the later test chambers that got little to no feedback - I will get them tested in the future, after easier tests will be designed to lead up to them properly.
That said, let’s delve into data and see what we have learnt!
Quantitative feedback
The surveys given to players after each chamber included a number of closed-answer questions. These question varied between some chambers, but one that appeared in every single one of them regarded how well did the players receive the chamber. I have used this data to produce the following chart:(I only included test chambers 1 to 4, since no later chamber managed to get more then 1 completed survey)
In the question the players had to rate the chamber on a 5-degree scale ranging from “extremely fun” to “not fun at all”. I have calculated the “fun percentage” by assigning each option an index value from zero to four, multiplying the number of answers in each category by that index, and then dividing the whole thing by the overall number of votes times two and then subtracting 1 from the result. This way a test chamber that everyone rated not fun gets index of -100%, one that everyone loved gets 100%, and the rest is somewhere in between. The bad news is while none of the chamber has fallen to the negative side, there was a significant and measurable enjoyability drop going from the first two chambers to the latter two. I intended for test chambers 1 and 2 to be simple tutorials, and then 3 onwards would ramp the difficulty, until chambers 6 and 7 which would introduce new mechanics, and the 8 would be the final test of the alpha, putting all that to use. Instead, it seems that people have found the tutorials to be more fun than the “real” puzzles! This shows that perhaps I have terminated players “education” too quickly and threw them into the deep water, without letting them get accustomed to the mechanics properly. The second chart illustrates what I mean well:
This “accessibility score” was calculated from two other five-point question in the similar manner to fun score from the previous chart. The two questions I used were “how well do you think you understand the mechanics of the chamber” and “how hard did you find this chamber”. As you can see, the first to chamber do very well, but the latter two fall down significantly, with the last one going into the negatives! While during the design of the chambers I considered logical difficulty to be an upside, seeing how much it contributes to the dropdown of players, I am now starting to think that it really needs to be lead up to properly to really be fun. This means making all the proper kind of easy explanatory puzzles to “arm” the player with knowledge and confidence to ‘even the fight’, so to speak.
The last chart I made was built based on questions regarding the kind of difficulty that players have encountered. The three options were: “Logical thinking: having to stop and think about the solution”, “Skill-based: executing the solution to the puzzle once you knew it”, and “Technical: coping with glitches and unpolished aspects of experience”. Naturally, logical thinking is the most desirable kind of difficulty in your game, with technical issues being obviously the least desirable. The results for the four test chambers are as follows:
As you can see, while logical thinking wasn’t really doing that bad, chamber 3 in particular proved to be quite difficult for people in terms of skill (perhaps because, as I learned from the open-ended questions, a lot of people have gotten through it with an alternative glitch-based solution!). However one thing that I found even more alarming was to how big extent the glitches affected people experiences with the game. When in the best case scenario ¼ of your playerbase finds your game’s biggest challenge to be dealing with glitches, you know you have to make sure it’s all roughed out!
Qualitative feedback
Aside from all the closed-choice questions, my surveys also included a number of open-ended questions, consisting of fields where players were free to type whatever they wanted. For the most part, I would say that this is the majority of the really useful data has come from. People gave a lot of small suggestions on how to improve the chambers and the mod, some of which I probably wouldn’t think of myself. A lot of the complaints were also directed at technical issues, such as crashes and getting stuck in walls. I am trying to come up with ways to fix these and other issues, and this will be the main focus for me in the upcoming weeks!
I have compiled a Trello list full of small tweaks and fixes made primarily from these suggestions, and I will be going through it over the course of the next few months.
Conclusion
Generally, people's suggestions were really good - I only got one troll submission and even it was kinda funny. Even with the relatively low turnout, the feedback I received have really influenced many conceptions I had about the project and I am very happy that I have went with this survey.
Oh, and one more thing for everyone who has not yet tried the mod: all the surveys are still open and I look at them regularly! It is not too late for you to check it out and help me make my mod better.
Until next time!
~WhyNott
the way you're going about this survey, with this amazingly postive attitude, has partially restored some of my faith in humanity
This comment is currently awaiting admin approval, join now to view.