Clever Patching with VFS Magic feature

Some time ago there has been an article concerning Desura and Linux. In this article a hefty discussion went on about where to install and especially how to deal with patching, different users and multiple game installations. Various solutions flew around the room mostly ending with a full copy of the game for each user for each patch version he wants to use. While this is for sure a possibility (and for Desura most probably the only solution) it is quite the cluster-fuck on the disk, wasteful and error prone. With the Drag[en]gine though and the concept of Launchers some interesting usage scenario is possible which is not only a very clean solution but also a very versatile one. I'd like to use this article to give a little insight in the powers of Clever Patching with VFS Magic.

A long time ago there had been once a news post which described a bit the Virtual File System (VFS) used in the Drag[en]gine. There exists also a Wiki Page about the VFS giving informations about this clever system. The VFS used by the Drag[en]gine is an entire different beast than the typical VFS one finds in other engines including AAA ones. The basic idea had been to allow for total transparency and some very interesting magic that can be pulled off. As it stands the patching and multi-user problem is one of the situations this particular VFS provides an interesting solution for.

So how works the VFS in a nut-shell? Games are never run directly on the operating system file system. Rather everything runs through the VFS. The game resides thus entirely in the VFS at all times which provides a consistent file system across all architectures. At the beginning the VFS is empty. The VFS is populated by adding a list of Containers. Containers can have different types which makes them very powerful. The most common container maps a disk directory into the VFS. Another typical container is the Archive container which maps the content of an archive file into the VFS. Others exist which are outlined in the wiki page but these two are all we need to implement a clever patching system. As a side note multiple VFS can co-exist which allows for even more magic to be cast if required.

Important to the understanding of this solution is the understanding of how containers act in a VFS. Every container has a Root Path. This tells where in the VFS the content of the container is visible. The content of the container does though not replace the directory like mounting does under Linux of Windows but it is actually layered on top of the existing content. While reading a file the containers are tested in the reverse order they have been added with the first container providing the given file winning. To get an understanding of this process have a look at the following graphics (click to enlarge).

Three containers have been added. The first is a Disk Directory container with the base game directory. This can be a disk directory or a CD directory. The second is an Archiv container pointing to a 7z file containing a patch for the game. The last container is another Disk Directory container pointing to the base of some game modification. As you can see different kinds of containers can easily be mixed. Three sample files are pointed out with the colors showing from which container the file would be taken. This allows for easy installation of games using CDs or DVDs without pain. Using the CD directory as the first and game installation directory as the next container all game content is accessible to the game no matter what files are installed on the hard disk or not. Try deleting some directories in other games. The games won't be running anymore as the game expects the files on disk. In the Drag[en]gine though this is fully transparent since what is no more on the hard disk container will be found on the CD container and this is all transparent to the game. A very simple yet very powerful system.

The same holds true for writing files but with a little difference. Every container can be read-only. Containers that are read-only can not win thus only the first writable container where the file is under the root path wins. This allows for a nice little trick. Imagine you open a game file in the editor, do a little modification and safe it back to the same file. Now imagine this file is on your CD-Rom. What happens? See the following graphics (click to enlarge).

As you can see with the READ operation the file from the CD is loaded. In the save operation the file is though now written to the hard disk directory instead as it is the first writable one. The next time you open the very same file it would come from the hard disk directory thus the one you modified. All this is totally transparent to you and the game. This system is the base for the solution to patching and multi-user/version support.

Now all will be put together to get a working system. The key for all this is the Launcher, a concept important to the Drag[en]gine. Launchers are responsible to provide the game with a working environment and this also means they are responsible to set up the VFS for the game. With other words the launcher adds the appropriate containers to the VFS before running the game. For the game the choice of containers is totally transparent and this is the key to success.

The configuration looks now like this. The base game is installed into /opt/games/epsylon/data. This is the same for everybody allowed to play this game. Next all patches are located as single 7z archive files under /opt/games/epsylon/patches with each archive file one patch version. This again is the same for all. The nice part is that nothing is yet patched as patching will be done virtually. Hence to play a game with patch version N as depicted in the graphics below the launcher just adds first the CD directory and the game directory and then all patches less than and equal to the requested patch level to the VFS. If the game is run now it plays with patch level N. You can run any patch version like this without creating duplicate copies. Downgrading is also a snap as just a few patch files less are added to the VFS before running. This is lightning fast, eats the bare minimum of disk space and safe as you can not mess up anything. On top of this containers for game modifications can be placed to play a game with a certain modification. This way modifications can not harm the game in any way as they are fully contained within their directory yet they can be precisely injected into the host game files where required. Eventually to support multiple users to have their own personal files even in case games try to write to their game directory (which is bad according to FHS) a user overlay container is added last. This points to some directory in the users home directory and is writable. Anything written by the game is thus redirected to this user directory while still visible to the game as if it would have been written to the game directory. This allows also for other nice tricks like using different overlays for different game versions and game modifications at the discretion of the users The graphics below shows a sample configuration.

This all is handled by the launcher hence neither the user nor the game developer gets in contact with this. You can just select which game to play, which version, which modification and which overlay (should you choose to use multiple ones) and go playing. No hassle with multiple installations of a game nor head-aches with modifications getting in the way or requiring specific game versions. To make it even more simple individual game profiles can be created to start a specific setup straight away without thinking.

I hope this little article gave you an overview about the huge possibilities the Drag[en]gine VFS System provides you and how in particular it helps with patching, modifications and multi-user support. Until next time.

PS: In the last News Post I mentioned adding more documentation. The first one concerning the Navigation System (Wiki) as described there is now available on the Wiki.