DV2 file

Tools created by vometia. Mirrored here for archival purposes as the link is now dead. Their original thread discussing DV2 can be found here. Also includes the modding guide.

dv2 - a utility for reading and writing DV2 files as used by Larian's
Dragon Knight Saga.

  Author: Vometia (a.k.a. Christine Munro), Oxford
  Date: December 2019
  Version: 0.1.something - IOW it's what it is.  Which is a mess.

Rationale
------------------------------------------------------------------------
DKS will only read data from its DV2 files, which makes modding the game
a little tricky if we have no means of updating their contents.  There
are other utilities such as QuickBMS which will unpack a DV2 with the
relevant script but few that will rebuild them.  The only example of
the latter I am aware of is an Ego Draconis-era utility that builds the
older Version 4 format DV2s.  While DKS is happy enough to load them,
the utility's provenance is unknown.

The slight awkwardness is that I don't "do" Windows, and this isn't
something that can realistically be done in a Bash script, so it's
written in C.  On FreeBSD.  I have subsequently made sure it will
compile on Linux (tested on Linux Mint v19.2) which seemed to
necessitate writing my own variant of nftw(3).  It will compile on
Cygwin once zlib-devel is installed; however, building a static Windows
executable that does not depend at all on Cygwin using e.g. MinGW proved
less straightforward and the idea was not pursued.

What it does
------------------------------------------------------------------------
Unpacks DV2 files, automatically ascertaining if they're V4 or V5, which
as far as I can tell only differ in the header structure.  Which is
unfortunately not reliably documented anywhere, but it seems fairly
consistent.

And packs DV2 files: it only supports V5, though that shouldn't really
matter unless someone has a pre-DKS, pre-patch Ego Draconis they simply
*must* mod, in which case some small alterations to pack.c will do the
job (i.e. change header_v5_t to header_v4_t and remove the two h_unknown
values).

Usage
------------------------------------------------------------------------
Its usage is fairly straightforward: either

     dv2 [-options] -p <file>.dv2 <directory>
  or dv2 [-options] -u <file>.dv2 [<directory>]
  or dv2 [-options] -l <file>.dv2

If no main command option (-p, -u or -l) is specified the program will
also attempt to ascertain what to do by its name, so if it is named
pack, unpack or list (which may or may not be prefixed with "dv2") it
will perform the appropriate function.

The various options are:

  -p: pack.  Create a new DV2 file from whatever <directory> is.
      Whatever is specified as the input directory will be stripped from
      the DV2's internal pathnames.

  -u: unpack.  Extract the contents of a DV2. <directory> is optional
      and if unspecified, the current directory will be used.

  -l: list.  Similar to -u but does not create any files; however, it
      is not the same as "-t -u" so if you need to test the integrity
      of a DV2 file, use the latter.

  -c: compress (or don't).  Use zlib compression when creating a DV2,
      which reduces the space required to about 60%; and don't attempt
      to inflate the contents of a DV2 when unpacking.

  -t: test.  Run through the motions but don't create anything.  File
      output is still created but sent to /dev/null.

  -d, -dd, -v, -vv, -q, -qq: turn on (extra) debugging, verbose messages
      or be quiet.  Serious errors that cause the program to abort will
      be shown regardless.

DV2 format
------------------------------------------------------------------------
This is not officially documented, but what has been found to work is as
follows:

  HEADER - 22 bytes (14 for V4)

  uint32  Header version: 4 or 5
  uint32  Undocumented variable, usually (always?) set to 1.  V5 only.
  uint32  Undocumented variable, usually (always?) set to 4.  V5 only.
  uint8   Aligned?  Guesswork, maybe legacy, always 0.
  uint8   Packed?  Again a guess, always 1, not used.
  uint32  Start of file data, the next 32K boundary after contents etc.
  uint32  Size of string space used to store internal filenames.

  FILE/PATH NAMES - See header for storage size

  A chunk of zero-terminated strings occupying the space described in
  the header.  These are relative to the DKS/data directory and use
  backslashes as separators.

  FILE COUNT - 4 bytes

  uint32  The number of files in the archive.  Obviously this should
          match the number of strings and entries in the directory.

  INTERNAL DIRECTORY - file count * 12 bytes

  A list of records, one per file.  There should be the same number and
  in the same order as the filenames.

  uint32  Start of data in archive.  This is relative to the start of
          data specified in the header, i.e. they need to be added.
  uint32  File data size within DV2 archive.
  uint32  Unpacked file size: this is zero if the file is not
          compressed.  It could be inferred that this means that the
          same DV2 can contain a mix of both compressed and uncompressed
          files, though I have not attempted to do so.

  PADDING - up to 32K

  A whole load of nothing up to the next 32K boundary.

  FILE DATA - everything else

  Compressed or not.  This is an important distinction as uncompressed
  files need to begin on 32K boundaries (this appears to be to speed
  loading of e.g. textures) while compressed files do not.  It doesn't
  seem to matter either way with compressed files though obviously
  leaving gaps everywhere rather defeats the purpose of compressing
  them.

REVISION HISTORY
------------------------------------------------------------------------
26-Nov-2019 vom Early proof-of-concept unpacker, uncompressed DV2s only.

 6-Dec-2019 vom Added support for decompression.

 9-Dec-2019 vom Added more comprehensive error handling/debugging.

10-Dec-2019 vom Added DV2 creation; indifferent success with zlib meant
                it would write uncompressed DV2s only.  Removed some
                FreeBSD dependencies.

13-Dec-2019 vom Fixed bugs with zlib and figured out more details of DV2
                format, i.e. correct file alignment.

14-Dec-2019 vom Tried to compile on Linux.  Failed.  Turns out it
                doesn't like nftw(3) or something about it; write
                somewhat compatible vftw from scratch as a replacement.
                It may have bugs.

                Added slightly less hopeless command processing.

15-Dec-2019 vom Now compiles properly on Linux and Cygwin.

                Investigated version that would work on Windows without
                run-time dependency on Cygwin.  Turned out to be more
                effort than planned so abandoned idea.

                Added support for endianness, though unlikely to be
                needed.  Not tested.

17-Dec-2019 vom Added some "missing" features to vftw(3) to make it
                useful elsewhere, requiring partial rewrite.  Found and
                fixed some bugs but very tired so not confident they've
                all been discovered.

18-Dec-2019 vom Added list function to make browsing DV2s more
                efficient.  Similar to running "-t -u" but doesn't
                do anything other than examine file directory at
                beginning of DV2 archive.  In other words, continue
                to use -t -u to verify integrity.

                Further work on CLI parsing though still very basic.

TODO
------------------------------------------------------------------------
Probably debugging; whether or not there are serious bugs, some pieces
of code seem overly complex due to their remit evolving rather than
being decided at the outset and may benefit from a rewrite.

On the other hand, "if it ain't broke, don't fix it", though "ain't
broke" is yet to be proven.

LEGAL
------------------------------------------------------------------------
I suppose I need to include this bit.  Copysomething 2019 me.  Do what
you want with the code as long as it isn't commercial or trying to take
credit for it i.e. I assert my my moral right to be known as the author
or something.  Also use at your own risk: if you don't like that idea
then, y'know, don't.