These instructions assume you own X3: Albion Prelude on Steam. They should be easily adaptable to any other install, like GoG.
1. Install X3: Albion Prelude.
2. Go to the game properties and make sure the game is in English.
3. The Bonus Package is incompatible with Litcube's Universe and Mayhem. To deactivate it, go to the right column of the game's page, open the DLC manager window, and untick the Bonus Package.
4. When the game is fully installed, copy your folder ...Steam\steamapps\common\x3 terran conflict into a new folder ...\X3 Mayhem 3. For ease of understanding, we will keep this name, but you can of course rename this folder to anything you want.
With this method, you can very easily go back to vanilla, LU, or any other mod. Mayhem 3 cannot be just removed, it modifies too many things in the file structure of X3.
5. Because your folder is no longer dependent on Steam, you have to download the NoSteam.exe file and copy it to your new ...\X3 Mayhem 3 folder and overwrite X3AP.exe.
6. Install Litcube's Universe from the official website. You need four things: the base install, the patch, the GUI and the soundtrack. You don't need the Export/Import package.
Install everything in your root folder ...\X3 Mayhem 3.
7. From the Mayhem 3 download page, download the main archive and the Graphics pack. The ambient soundtrack is optional. Extract both archives to your root folder ...\X3 Mayhem 3 and overwrite. When a new patch is released, you just have to download the updated main archive, and extract / overwrite like you just did now. There are no "patches", it's always the full package.
8. Mayhem 3 is now installed! Because it implements procedural galaxies, you must first generate a new world with an external program called the Galaxy Generator that you can find in your root folder: ...\X3 Mayhem 3\Mayhem Galaxy Generator.exe. If you have any problem running this software, please deactivate your Antivirus.
All the controls you see are optionals. You can just click Generate to have a default galaxy, or use the Randomize button; or customize your next galaxy to your liking. When you're finished, click Run to launch the game with the galaxy you have selected.
For a first playthrough, the default parameters are just fine and you can just click Generate as soon as you open the program.
Then select the galaxy you just generated and click Run to launch the game.
Note that "Use the alternate soundtrack file" checkbox is for advanced users or those who have downloaded the ambient soundtrack optional file. This option picks different music files specified in mayhem_data\custom_musics.txt to fill the galaxy instead of the original ones that are specified in mayhem_data\custom_musics_lu.txt . If these files don't exist, there will be no music.
You can now start your first Mayhem playthrough! Pick your favorite race, answer some questions that will customize your game experience, and be sure, at the end of the form, to click the "What do I do now?" button to have some early guidance.
Also don't forget to read the in-game encyclopedia. It contains a lot of information about the new features of Mayhem.