MapMixer is designed to give you more control over the way you launch and manage your game session in Unreal Tournament 3. It replaces the the game's Main Menu and let's you launch either a quick-offline match or a complete session where every map is specifically customised - or randomised. It's a whole new way to manage your UT3 experience.
Custom Gametypes
A MapMixer Custom Gametype is simply a collection of settings for a particular installed Gametype. A basic example of one is Instagib Deathmatch. In its simplest form, it's just Deathmatch with the Instagib Mutator enabled. There are numerous advantages of having it as a separate gametype. The most important being that you can personalise all the core settings for that style of play, without having to create a separate Rules List or enable the Mutator manually. Remember that each Custom Gametype is based off an installed Gametype.
While not mandatory, it's important and useful to understand that a Custom Gametype should only contain settings that help distinguish it from another gametype. With the IDM example, it is the Instagib Mutator that separates this from plain deathmatch, so setting a goal score here may not always be appropriate and best suited for a Game Rules List.
Another advantage of a MapMixer Custom Gametype is that you can choose which maps to enable for this Gametype. You can either choose from a set of Map Prefixes or a specific Map List. This means that if you use your Custom Gametype when Autocreating a Session Map List (using a Map Source List), it won't select Maps for that gametype that you have deemed inappropriate. (Of course in that example, Map Source Lists also allow you to override the map selection per Source - but this is just an example.)