UPDATED: We are replacing our back end storage system shortly to resolve downtime issue and slow speeds.

Posted by INtense! on Jul 31st, 2010

If you have visited ModDB or IndieDB in the past week, you have probably noticed our servers seem to be going up and down like a yo-yo. Unfortunately that is the case due to a recent upgrade we did. In this upgrade we provisioned a new webserver, new database server and the troublesome new SAN iSCSI filestore.

Whilst this upgrade should have made things lightning fast (and did for a while), it appears there is a critical connection bottleneck between the all important SAN box and our 3 webservers. The SAN which should easily do 1000mbit to all 3 boxes at once is instead choking at 30mbit which is waaaay lower than required. As a result everytime file transfers spike up (which is all the time) CPU load spikes and the system crashes. No data is lost but emails, media and downloads are all very unstable right now and there is no quick fix. So please bear with us while we address the problem as efficiently as possible.

UPDATE: 1st August

We will be replacing our backend SAN in the coming 2 weeks. Site performance for the next 2 weeks will likely be sluggish to allow for migrating terabytes of data from the faulty SAN to a replacement device.

Thanks all for your patience and understanding!

Post comment Comments
MrMattWebb Jul 23 2010 says:

Wow, all the images are gone o_O

good luck :)

+2 votes     reply to comment
INtense! Author
INtense! Jul 23 2010 replied:

Not gone, just not displaying

+7 votes   reply to comment
Dragonlord Jul 23 2010 replied:

Best image compression ever... 0bytes in size :D

+31 votes     reply to comment
Anonymousperson Jul 25 2010 replied:

I laughed at this... then felt like a nerd
Which isn't really a bad thing in retrospect, I suppose

+6 votes     reply to comment
NullSoldier Jul 23 2010 says:

I for one welcome our new image eating SAN iSCSI overlords.

+11 votes     reply to comment
1337-1 Jul 25 2010 replied:

All hail the overlords.

(quick, the overlords are hungry, we need more images!)

+4 votes     reply to comment
ytres Aug 1 2010 replied:

All your images are belong to us!

+2 votes     reply to comment
Dremth Jul 23 2010 says:

It's alright guys. As much as I'd love to be on ModDB 24/7, we users need outages every now and then so we can do other things like sleep, eat, and go to the bathroom.

+13 votes     reply to comment
INtense! Author
INtense! Jul 23 2010 replied:

we've been getting record traffic this week too due to some massive mod releases (total war, terminator, firearms) so those are the guys i'm worried about. We really want the site to be up and running 100% so we can help them distribute their mods to as many people as possible.

+17 votes   reply to comment
chubber Jul 23 2010 replied:

not to mention Dark Times II: Rising Son coming out tomorrow...

+2 votes     reply to comment
Nilghai Jul 23 2010 says:

I'm just glad to hear that nothing is lost. good luck with it Intense :)

+3 votes     reply to comment
NGS616 Jul 23 2010 says:

The provider of your file-store should compensate you with a little free service.

Good luck, though.

+3 votes     reply to comment
INtense! Author
INtense! Jul 23 2010 replied:

they've fulfilled a few SLA claims, but i'd much rather the site work while they sort out the bottleneck!

+4 votes   reply to comment
qpac Jul 23 2010 says:

That explains some things. Well, I hope it gets better soon.

+3 votes     reply to comment
mastersrp Jul 23 2010 says:

Glad to know. I haven't really felt the impact though, but I haven't been downloading anything. Media seemed to work for me without a hitch too... Weird.. Good thing that you're fixing what's wrong though!

+2 votes     reply to comment
INtense! Author
INtense! Jul 23 2010 replied:

you're lucky then... things have been very patchy!

+3 votes   reply to comment
icantfly1 Jul 23 2010 says:

Ah so that's why some downloads didn't work..
Anyway, thanks for informing us!

+3 votes     reply to comment
ʇɐuʞɐʇǝp Jul 23 2010 says:

I'm lucky. I get the same as mastersrp.

+2 votes     reply to comment
Salsa_Shark Jul 23 2010 says:

I’ve had some trouble with moddb this past week, I was blaming Telstra for it, I guess this isn’t the case, ho well, it’s nice that you let us know.

+4 votes     reply to comment
chubber Jul 23 2010 says:

I couldn't even load moddb at all at one point

+3 votes     reply to comment
ammd Jul 23 2010 says:

I know this is offtopic but we need a "clear all" button for all those updates.^^

+4 votes     reply to comment
SCΛRECROW Jul 25 2010 replied:

That's a good idea.

+2 votes     reply to comment
galaxy366 Jul 25 2010 replied:

that is a great idea xD

+2 votes     reply to comment
Druza Jul 23 2010 says:

Thanks for actually explaining this to us instead of telling us nothing or just saying it was "technical difficulties".

+6 votes     reply to comment
BlabVoid Jul 25 2010 says:

Last night I was having trouble posting a news post because of this, wouldn't let me post it because it said the image file path wasn't valid.

Now I know why. :P

+1 vote     reply to comment
Dragonlord Jul 25 2010 replied:

Yeah, it's rather unfortunate this situation. News post delayed now nearly a week since I can't get it up. Maybe I've got more luck in the days to come :/

+1 vote     reply to comment
Cremat0r Jul 25 2010 says:

Oh ,nice, I was kicked like 10 times off Moddb, it gets annoying after a while :S

+1 vote     reply to comment
jstubbles Jul 25 2010 says:

Unfortunately coinciding with my HTML editing of our page, hah.

Hopefully it gets fixed up good! :)

+1 vote     reply to comment
USSJboy Jul 25 2010 says:

Wow. Thats kinda interesting. So far I've had no problems on Moddb. Speed is still good, pics load and downloads still going strong.

Cheers :)

+1 vote     reply to comment
xXMaNiAcXx Jul 25 2010 says:

Nice to know that the problem wasn't only with me, and it seems to be really stable now, just keep up with your work and make the server up until I finish to read my updates :D

+1 vote     reply to comment
masterchief396 Jul 31 2010 says:

Sorry to hear that I was wondering what was going on. I hope things get cleared up.

+1 vote     reply to comment
lancerfour Aug 1 2010 says:

Hope all goes well, and thanks for the update.

+1 vote     reply to comment
Post a comment

You are not logged in, your comment will be anonymous unless you join the community. Or sign in with your social account:

Post news
Related Groups
Indie DB
Indie DB Official with 2,243 members
Mod DB
Mod DB Official with 4,015 members