TNM is a huge singleplayer mod for Deus Ex based on the gaming community, finally released after 7 years of development.

Report article RSS Feed The Nameless Mod Post-Mortem Part 3

What went wrong with the development of TNM? The mod's creator recalls the lessons learned.

Posted by stenchy on Jul 24th, 2009
Article


What Went Wrong

1. We spent no time on pre-production

In stark contrast to Deus Ex's luxurious six months of pre-production, we literally had none. Actual game assets were being produced from day one. Before we'd even decided what the game was going to be about or what sort of story we would write, artists were churning out character textures and level designers were building maps. A month or two of recklessly disorganized brainstorming segued into a confused, hurried documentation process where we struggled to keep up with our own ideas. Since the project was entirely anarchistic in the beginning, nobody had to wait for permission to start pumping out assets and little to no coordination took place between different designers or artists.

In and of itself, "don't eliminate pre-production" may seem like a pretty obvious piece of advice, but what we were actually trying to do, in hind-sight, was iterative design. We were experimenting with the engine and the tools at our disposal, testing the capabilities of our team, and throwing our every idea at the game with no sense of restraint in order to see what worked and what didn't. The approach could have worked out a lot better than it did if we'd made a conscious effort to evaluate the decisions we made and deliberately pick out the elements we wanted. Our team size and structure was even quite suitable for an iterative design model; we had around ten highly dedicated people working on it back then and there was no management or budget to worry about. We were free to mess around and distill the useful ideas from the poor.

Unfortunately we made the major mistake of not eliminating anything at all. Everything people created and every idea anybody had went straight into the design documents. In that way, we managed to waste what was actually a pretty healthy pre-production process because we treated it as full production. A lot of the bad ideas and decisions that made it into the game back then. Although we managed to replace or fix later down the line, some of it can still be found in the final product because it's simply too time-consuming to fix. File that one under "how not to do agile development".


2. We didn't understand all the code

They say those who don't know history are doomed to repeat it, but sometimes you know going into a project that you're going to face some major problems and there's very little you can do about it. The sometimes makeshift nature of parts of Deus Ex's code was one such problem we faced. Now, far be it from me to criticize the Deus Ex programmers — they did an amazing amount of work and unlike us, they were working under deadlines. But by their own admission, the artificial intelligence in Deus Ex wasn't as well designed as it should have been and suffered even further for being constructed precariously on top of the Unreal Engine code.

To us, the problem wasn't just that we didn't want to mess too much with the AI code and that we never had access to Unreal's source code to fix some of the more persistent bugs we encountered, but that we didn't fully understand how the code was meant to be used in the design. Our AI problems are the most illustrative example of that, but unfortunately not the only one. Deus Ex's AI, it turns out, is pretty stable when it just has to be friendly or when it just has to be hostile. When you start mixing friendly and hostile (and perhaps even neutral, if you hate yourself) behaviors in the same area, the illusion begins to fall apart. What we did was mix friendly, hostile, and neutral characters in the same levels and then switch their alliances around depending on the player's previous choices. It blew up in our faces and it never stopped blowing up.

I suppose there were two ways we could have solved this problem: One was to sit down and rewrite all of Deus Ex's AI code as early as possible. This wasn't an option due to our extremely limited programming resources, so the remaining solution was to look very carefully at how Deus Ex's characters had their AI set up and how the different settings were used then stick to that. Once we finally realized that, implementing such a solution was tantamount to scrapping everything and starting over from scratch.


3. We had to compromise on quality

By the time TNM was released, our standards were pretty high. We screened our actors thoroughly for audio quality. We rebuilt entire levels dangerously late in development if there were too many problems with them. We'd long since stopped implementing any feature if we didn't feel that we could take it all the way. But of course, beggars can't be choosers, we sometimes had to compromise because we couldn't pay people. The two areas where I think this is most obvious in the final product are level design and voice-over. For the voice-over, it was a hard and bitter struggle to secure solid actors for the major characters, but I think we largely made it. In terms of minor characters, the picture looks a little different: Many of them were recorded by ourselves or our girlfriends or by random fans. Even our good actors occasionally had sub-standard recording quality, which becomes obvious far more often than it should.

In terms of level design, the problems were technical as well as aesthetic. Some of our level designers were quite good at producing pretty architecture, but didn't seem to grasp the design principles that made Deus Ex's missions so exciting. Others had brilliant ideas and a great understanding of design, but their levels looked boxy and uninteresting. Some levels that were otherwise nice and well designed had been constructed entirely off-grid, causing substantial technical issues later on. It's painfully apparent that we never had an art director or a concept artist, making our levels vary erratically in style and tone. Far too late did we understand the concept of dividing level design tasks amongst a designer and an environmental artist. Even then we never had the resources to do it in the first place.

Though this problem was emphasized because we were forced to make do with the generosity of volunteers, I imagine that too short a supply of talented team members who meet your standards of quality is a problem even commercial projects face. I suspect that if I had a fool-proof solution for it, I could use it as the basis for a whole new article. In the future, we aim to conquer the problem by counting on a small team of only the most skilled and dedicated rather than designing a project that demands the participation of 50 contributors of varying talent and enthusiasm. In short, we'll embrace the indie ethos and stop pretending we're an AAA studio.


4. We didn't test enough

We tried though. We gradually brought over twenty testers in to play the game starting several years before we were finished and set up an online bug tracking system for them to report back to us with. We set up a forum specifically for the testers and sent out guide documents to everybody detailing how to install the test build, what sort of bugs they could expect, how to report them, etc. We evaluated our bug database and set up priority fix lists towards the end of the project to make sure all the really problematic bugs were fixed before release. When TNM was released, it had no major bugs that we knew about.

Yet the first journalist we sent the release candidate out to, before the download link was publicized, found a bug in the kill counting code which would close off an entire map prematurely due to a recent fix between the two last release candidates. We then spent an entire week fixing bugs in some sort of infernal post-crunch, followed by another month of slower, steady fixes. By the second patch, we've fixed around 240 problems and we've more or less brought TNM into the state we thought it was in when we released.

There are three things we could have done to improve our quality assurance. First and most importantly, we should have been more careful to test our own work during development. Far too often, features and assets went into the game without any testing. In a game as heavily dependent on emergent gameplay as Deus Ex, it's critically important to apply very thorough testing to everything you implement. If we had tested each of our new additions in combination with as many different factors as we could think of, that would've significantly reduced the amount of combinations that slipped through the cracks. Secondly, we should have set up some auto-testing: one of our programmers coded up auto-testing scripts a couple of months before we were done, which was far too late — we only managed to apply it twice before release. Since our plot had so many branches and variations along the way, being able to fast-forward through certain common branches would have been a great help earlier in the project.

Finally, we should have managed our testers more strictly. Unfortunately I'm not sure most of our testers would have responded very well to tighter management, but I suspect five dedicated and well managed testers would've been far more useful than a score of testers left mostly to their own devices. We tried to get our testers to commit to particular kinds of playthroughs, hoping we'd get all the branches covered, but it never worked because most people who volunteered to test weren't interested in committing to anything beyond playing the game and writing down the major bugs they found. In summary, our primary mistake was that we skipped the organized alpha testing and went straight to beta testing.

5. Our project is a PR nightmare

The final and perhaps the hardest problem we faced was making people understand that our game wasn't shit. We'd spent a lot of time making sure that people wouldn't be alienated by our game and that we had plenty of depth and internal consistency to keep everybody interested, but we'd completely underestimated what a huge turn-off the basic idea of the mod was to so many people. Every site where we published our trailer and every forum thread where people began to discuss the mod, one sentiment would immediately surface like a knee-jerk reflex: What an idiotic concept. Why would anybody spend seven years working on this fan-boy circle-jerk of a game?

We were pretty crest-fallen. We'd gone to such lengths to make sure TNM was a game, not just a joke, and many people wouldn't even give it a chance because they immediately assumed the worst. But to make matters worse, we came to realize that we'd frontloaded all the Internet references, fan culture, memes and the in-jokes right in the first mission of the game. Part of this was unavoidable: the first mission served by necessity to introduce the player to our setting so all the opaque references and Internet semiotics were presented to you immediately. Once out of the first introductory hub area the setting would quickly slip into the back seat to leave room for the plot itself, but too many people seemed to never reach it having lost all interest long before then. The Nameless Mod is free to download; we have no demo. Convincing people to invest their time in playing through TNM, that introduction area is all we have, and it seems to be doing a rather poor job.

Perhaps our greatest mistake was to tell people that The Nameless Mod was inspired by a real community that existed on the Internet at one point in time. I suspect people in general would be a lot more susceptible to our quirky cyberspace setting if they thought we'd just invented it as a Snow Crash-esque sci-fi take on the Internet. Then they wouldn't associate us with the reviled genre of "forum fan fiction" to begin with and once playing the game they wouldn't be expecting in-jokes everywhere. Much of the feedback we've received has implied that people constantly see in-jokes and obscure references when by far most of the game's fiction was either invented specifically to suit the plot or the setting or twisted so far out of its original shape that it no longer bears any resemblance to the events or the people it was inspired by.

A well-known games journalist graciously defending our concept wrote: "no wonder everyone makes games with space marines being gruff". I'd like to think people are generally open to new concepts, but I'll admit that our premise and our setting do us no favors. Not, however, because TNM is too weird — there have been many successful games far stranger than The Nameless Mod and our world is quite recognizable when it comes down to it. But at the same time it reminds people of a genre of fiction which is almost never executed well. If we'd managed to set ourselves further apart from that genre with all the PR material we sent out, we'd done more to change people's expectations before firing up the game which would have resulted in a much easier sell.


In Closing

In some ways, The Nameless Mod is a triumph for agile development and iterative design processes. Not only did we learn how to make the game we were trying to make, we learned almost everything we now know about game development and project management. We learned these things by trial and error and our errors pushed us to seek out the information we needed and apply it to our own work.

It's unfortunate that we wasted so much time messing up and making bad design decisions. Part of this was because we didn't manage feature creep as well as we could have. Part of it was because we got carried away with perfectionism, constantly recreating old assets to match our new standards. If we'd known what we were doing right from the start, TNM could probably have been released in 2006.

It's been an incredible project though and it's changed some of our lives substantially. I think we ultimately came out on top of most of our mistakes and we're very excited to move on to the next challenge. With all the feedback we've received on The Nameless Mod, hopefully we won't repeat any of the mistakes we've made. And now we have this post-mortem to help us remember them.

Previously: Post-Mortem Intro, What Went Right

Written by Jonas Wæver
Edited by Ryan Anderson

Follow The Nameless Mod:

ModWatchFacebook
Icons by Iiro Jäppinen



Post comment Comments
formerlyknownasMrCP
formerlyknownasMrCP Jul 24 2009, 1:11am says:

Don't feel so bad about the PR stuff, think about it like this, you were on EVERY news stand in Australia at one point in big bold white writing "THE NAMELESS MOD".

The QA stuff I completely agree with. The fact is you needed professional QAs for a project of this scope, not people that wanted to "play the game early"- and you certainly needed a QA manager to ensure the bugs that were being submitted weren't bugs that had 1. been submitted already and 2. weren't written clear enough for the programmers to understand.. the other problem is that they're not supervised meaning QAs can and will slack off if they're not kept an eye on.

Still the end product was amazing and its a mod that is one of a kind. Well done and I wish you guys all the best now that you're all indies :D.

+4 votes     reply to comment
methy
methy Jul 24 2009, 7:27am says:

Not so much a circle-jerk, at least when I was active at ModDB, the perception was that TNM was written by PlanetDeusEx, for PlanetDeusEx, similarly to how Hammy-bob Mod was written by ModDB, for ModDB.

This must have had a positive impact somewhere along the line - not many mod projects ever see a team of ten dedicated developers, let alone when they're just starting out. Of course, the negative is that people thought it would be a joke, and even worse, a joke they weren't in on.

At the end of the day though, you guys created a mod that is unique in scope and quality, proved the mod community is capable of professional production values, without taking the usual 'indie' shortcuts. It is well known, respected, and considered one of, if not, the best in its class by the majority of those who have played it.

It is one of the most important mods to come out since the first wave of quality mods, and it's something you can all be proud of for the rest of your lives. Congratulations.

+1 vote     reply to comment
nerdenstein
nerdenstein Jul 24 2009, 7:34am says:

I cant think of anything else to say =D Mr_Cyberpunk and methy have said it all for me.. It is a great, unique mod! Okay so u screwed up in the Testing and the pre-production but hey you said it urselfs. u learned from ur mistakes through trial and error and u can take all the new skills u have learned in doing TNM and use them in ur future projects. Good Luck to u guys and i'll be keeping an eye out for u in the future =)

Btw. Excellently written Post-Mortems Jonas ;)
I enjoyed reading them almost as much as i enjoyed playing TNM ;)

+1 vote     reply to comment
enablerbr
enablerbr Jul 24 2009, 8:44am says:

i guess the real test for this learning process. will be when the next project is started. will the mistakes of the past be repeated or left as distant memorys.

i like the deadication shown by the team to stick with the project to the end. where some other team projects have just faded into nothing.

+1 vote     reply to comment
Trestkon
Trestkon Jul 24 2009, 9:38am replied:

Very true :-) We actually *have* started the next project, and are so far managing avoid the major mistakes of early TNM development, let's hope we can continue that way!

+1 vote     reply to comment
Trestkon
Trestkon Jul 24 2009, 9:40am says:

I think I'm going to frame your comment and hang it on my wall, methy. A most excellent way to start my morning :-)

+1 vote     reply to comment
methy
methy Jul 24 2009, 11:28am replied:

I'm glad, Trestkon. If you make the next one even better, you can have my left nut and frame it on your wall. No pressure.

+1 vote     reply to comment
Trestkon
Trestkon Jul 24 2009, 12:00pm replied:

Err, I don't think that would match my couch...you better keep it.

+1 vote     reply to comment
Mkilbride
Mkilbride Jul 24 2009, 12:32pm says:

While I've never played your mod, nor Dues Ex.(Not my type of thing really), I applaud your efforts for undertaking such a project.

Working under such conditions, completing such a thing, and at such a level of quality from what I've seen, is a feat rarely seen in the modding community.

+2 votes     reply to comment
Phalxor
Phalxor Jul 24 2009, 6:53pm replied:

definite dedication, 'bout to give 'er a run through

+1 vote     reply to comment
Jonas
Jonas Jul 24 2009, 8:38pm replied:

Remember to patch to 1.0.2! :P

+2 votes     reply to comment
Danboe
Danboe Jul 25 2009, 7:13pm says:

Very interesting insight into such a huge project.

+1 vote     reply to comment
Post a Comment
click to sign in

You are not logged in, your comment will be anonymous unless you join the community today (totally free - or sign in with your social account on the right) which we encourage all contributors to do.

2000 characters limit; HTML formatting and smileys are not supported - text only

Icon
Deus Ex Icon
Platform
Windows
Game
Deus Ex
Contact
Send Message
Official Page
Thenamelessmod.com
Release Date
Released Mar 14, 2009
Mod Watch
Track this mod
Feature
Browse
Features
Report Abuse
Report article
Related Mods
The Nameless Mod (Deus Ex)
The Nameless Mod Deus Ex - Single Player Adventure
Related Games
Deus Ex
Deus Ex Single & Multiplayer Adventure
Related Groups
Off Topic Productions
Off Topic Productions Developer & Publisher with 5 members
Summer of Mods & Indies
Summer of Mods & Indies Official event with 137 members