Conversation System (News Pack 1/2)

Information flow got a bit severed in the last time which had been due to various problems including a blown RAM stick totally bringing down development for some time until fixed. Besides this a lot of work went into different areas for one the game engine itself as well as the Epsylon game based on it. This made it difficult to make news posts in between so all has been collected now and split into two news posts one focusing more on the game engine changes one more on the Epsylon game changes. This one here is going to show now the game side Conversation System.

Face Rig and Model/Animation Overhaul

Before the Conversation System could be worked on the old model had to be replaced with a new one. The new model features a bunch of changes compared to the old one. The model includes a jacket which is fully rigged and animated. Also the face has been changed and fully rigged with a face rig able to do emotions as well as mouth animation. Getting mouth poses done had been a bit tricky as I had to first sift through a lot of documents on the Internet containing lots of babbling ending up a lot of experimentation and lost time until I had something going. Another point had been the overhaul of the textures. The old model had a lot of separate textures which did not help the matter a lot. Now the model has only 1 body texture and 1 texture for the necklace (as it contains transparency). Last but not least the animations had been redone to match the character. Not all animations are redone so far so it's still a work in progress. The new animations though should do the trick now. With this work out of the way the next step could be taken care of.

Speech Animation

Since there is supposed to be quite a bit of conversation and because I'm still without some helping hands I had to design a Speech Animation System in a way I can get the job done rather quick in the future although maybe not as good as a complex system. For this I made first a Speech Animations System including an editor.

In general this editor allows to create speech animation using phonemes and words. For each speech animation a list of Phoneme can be defined. This simply defines what IPA symbol triggers this phoneme and what mouth pose to use as well as how long in average this phoneme is. Then with this list of phoneme a list of words can be created. Words simply define how a given word is spoken using IPA notion. This way you can use dictionaries for your language of choice to get the animation for a given word done quickly. You are though not forced to use IPA if you don't want to. Important is only what symbol you assign to your phoneme the rest follows then. All this is then stored as *.spa.xml files. Every character in the game has one (mostly shared ones) Speech Animation assigned. This way different languages for your game can also be created by just creating a new Speech Animation with a new list of words. With this done the next step could be taken.

Conversation System

With the speech animation up and working the Conversation System could be implemented. For this again a new editor has been created. Doing this editor took most of the time but it is worth it.

In general the conversation system composes of a list of Files and Topics. A File is an entity in the game world and can be anything from people over objects all the way to events or in general anything of importance in the game. For every file there exists a list of Topics. Topics define what kind of questions you can ask about a game entity. The available topics can change depending on the progress of the player. The idea is to give the player a larger amount of stuff to ask for to make it more difficult to progress by questioning people. In general people are not going to answer to a lot of your babbling so figuring out what to ask is one part of the job. Every topic consists of a list of Actions. These are the basic building stones of the conversation. There are various actions ranging from switching the camera shots, making actors talk, making actors gesticulate, making them do faces as well as conditional actions. Most actions just set up parameters in the actors involved in the conversation and return immediately. This allows to easily trigger various actions of different actors in a conversation even on a conditional basis. The actions itself are processed sequential. This allows to easily create dynamic conversations even with variable characters present on stage as actions wait on each other without having to get yourself worked up over timing. In the most simple case you just drop in a bunch of actor speak actions and you have already a working conversation without a lot of work. To achieve this the Conversation System has a few additional properties.

For actions you only specify what you want to happen but not what happens precisely. So for example at one place in the conversation you want your actor to make a certain Gesture. It doesn't matter if he's standing around or sitting at a table. The type of gesture is the same but depending on where the conversation takes place or what the actor is doing right now the gesture plays out differently. To achieve this gestures are grouped. For each gesture you can define then how it looks like for whatever situation it could happen in. For the conversation this is fully transparent so you can just say what kind of gesture you want without having to worry about where the conversation takes place. For each use of a gesture in the conversation you specify also the length. This allows to scale gestures easily to match your conversation without a lot of work.

For Face Poses this works similar. You specify what face pose to use and the face pose defines then how to produce the face in question. Each actor has an own Face Pose Animator. This is done using engine animators and therefore works similar. The face pose then defines how to set the controllers. This way you can just change the actor (and his face pose animator) and you get an entire different result without having to screw around with the conversation definition.

For animating the mouth the speech animation comes back. As mentioned above each character has an own speech animation which is usually a shared one). Typically lip-sync is done by keying mouth poses to a voice audio clip using visemes. While rather precise this is a lot of work especially if you want to have a bunch of conversation lines. For this purpose only words are used. As mentioned above in a speech animation words are defined and how they are animated. For the actual conversation you then only specify where words are located (and how long they are) in respect to a voice audio clip. This is very fast although not as accurate. But chances are slim one notices if a mouth position inside the word is not 100% on cue. On the other hand lip-syncing to a voice audio this way is very fast. Furthermore you get another bonus. By changing the speech animation (for example by changing the character) the animation of the words change too without you having to make any changes at all to your conversations.

Now conversations without Camera Shots would be boring. These work similar to gestures in that you define (using camera shot actions) what camera shot to use. Each camera shot then defines how it looks like. These can be static or moving camera shots. Moving shots are defined using a linear path by defining the start and end value for various camera properties. These are then interpolated in a linear way. Whenever you change a camera shot you define also how long the shot is. This scales then the movement. This way you can stretch camera shots over multiple spoken lines of one or more actors if required. The camera placement works using an orbit system hence you can specify an orbit center, orbit distance, orbit orientation as well as a look-at point. You can also alter the field-of-view (for example for dolly zooms). Both the orbit center and look-at position can be attached to individual character in the conversation or a reference point. This way you can do all kinds of movie style camera shots and use them easily across your conversations. In addition parameters can be also defined using curves which gives full control if required.

Conversations can be created for any number of characters from 1 (for monologues) all the way up to group conversations. Adding and removing characters is also easy to do. You can reference characters by their unique game name or by a number (the how-many-th character in the conversation). Actions can also have conditions triggering only if certain actors are present or absent. So you can also make one character bad-mouth another if he's not present in a conversation and so forth.

Last but not least you can test your conversations live in the editor. For this the editor supports a small Information System as used in the game. Instead of having global flags every file in the game has their own set of flags (called Information). You can easily manipulate and watch the effect on the sets of flags this way in the editor. For convenience purpose you can load/save flag configurations and character configurations for quick testing.

After all this talk some video. In this video I created quickly (an afternoon job) a small static conversation between two characters (using the same model for this test). Don't look at my puny attempt at a voice acting there... I know I suck at it ;) . Had been just for a quick test. Also I used only 3 simple test gestures consisting of 2-3 frames each to test the system. This can all though be easily improved later on with the right skills. So here the video (sample at the end of the video, watch large as the video is smaller).

The Best: It's Script only!

Yep, that's true. This conversation system has a nifty advantage over other game engines in that it is fully implemented with scripts. This means the system is implemented (in the game, not the editor) using game scripts using the engine resources in clever ways. Hence you can modify and extend this system in whatever way you want. And if required you can totally replace the conversation system with your own.

Outlook

This had been the larger of the two news posts. The next one will be posted a week later in the Drag[en]gine Profile and is going to show some of the changes in the game engine which also had been required to implement the conversation system.