How to open and extract homeworld.big:
You need a small utility, for example BigViewer and Spooky RAT
Once extracted, the following files can be edited in notepad:
*.shp -- the DNA of a ship
*.lod -- this manages the scaling model of a ship
*.script -- modify them to change general gaming features
*.level, *.missionsphere, *.dist -- they control what a map looks like
To edit *.mex files which determine guns of a ship, you need a mex editor.
To edit *.peo files which are actually the model of a ship, I still have no idea
To edit *.lif files which are textures of ship, you can use 3D Exploration to save them as other formats.
Useful links
Forums.relicnews.com
Strategyplanet.com
Oocities.org