If you have visited ModDB or IndieDB in the past week, you have probably noticed our servers seem to be going up and down like a yo-yo. Unfortunately that is the case due to a recent upgrade we did. In this upgrade we provisioned a new webserver, new database server and the troublesome new SAN iSCSI filestore.
Whilst this upgrade should have made things lightning fast (and did for a while), it appears there is a critical connection bottleneck between the all important SAN box and our 3 webservers. The SAN which should easily do 1000mbit to all 3 boxes at once is instead choking at 30mbit which is waaaay lower than required. As a result everytime file transfers spike up (which is all the time) CPU load spikes and the system crashes. No data is lost but emails, media and downloads are all very unstable right now and there is no quick fix. So please bear with us while we address the problem as efficiently as possible.
UPDATE: 1st August
We will be replacing our backend SAN in the coming 2 weeks. Site performance for the next 2 weeks will likely be sluggish to allow for migrating terabytes of data from the faulty SAN to a replacement device.
Thanks all for your patience and understanding!