Report article RSS Feed Server troubles

UPDATED: We are replacing our back end storage system shortly to resolve downtime issue and slow speeds.

Posted by INtense! on Jul 31st, 2010

If you have visited ModDB or IndieDB in the past week, you have probably noticed our servers seem to be going up and down like a yo-yo. Unfortunately that is the case due to a recent upgrade we did. In this upgrade we provisioned a new webserver, new database server and the troublesome new SAN iSCSI filestore.

Whilst this upgrade should have made things lightning fast (and did for a while), it appears there is a critical connection bottleneck between the all important SAN box and our 3 webservers. The SAN which should easily do 1000mbit to all 3 boxes at once is instead choking at 30mbit which is waaaay lower than required. As a result everytime file transfers spike up (which is all the time) CPU load spikes and the system crashes. No data is lost but emails, media and downloads are all very unstable right now and there is no quick fix. So please bear with us while we address the problem as efficiently as possible.

UPDATE: 1st August

We will be replacing our backend SAN in the coming 2 weeks. Site performance for the next 2 weeks will likely be sluggish to allow for migrating terabytes of data from the faulty SAN to a replacement device.

Thanks all for your patience and understanding!

Post comment Comments
MrMattWebb
MrMattWebb Jul 23 2010, 2:05pm says:

Wow, all the images are gone o_O

good luck :)

+2 votes     reply to comment
INtense!
INtense! Jul 23 2010, 2:06pm replied:

Not gone, just not displaying

+7 votes     reply to comment
Dragonlord
Dragonlord Jul 23 2010, 2:23pm replied: Online

Best image compression ever... 0bytes in size :D

+31 votes     reply to comment
Anonymousperson
Anonymousperson Jul 25 2010, 1:13am replied:

I laughed at this... then felt like a nerd
Which isn't really a bad thing in retrospect, I suppose

+6 votes     reply to comment
NullSoldier
NullSoldier Jul 23 2010, 2:21pm says:

I for one welcome our new image eating SAN iSCSI overlords.

+11 votes     reply to comment
1337-1
1337-1 Jul 25 2010, 5:31am replied:

All hail the overlords.

(quick, the overlords are hungry, we need more images!)

+4 votes     reply to comment
ytres
ytres Aug 1 2010, 12:21am replied:

All your images are belong to us!

+2 votes     reply to comment
Dremth
Dremth Jul 23 2010, 2:31pm says:

It's alright guys. As much as I'd love to be on ModDB 24/7, we users need outages every now and then so we can do other things like sleep, eat, and go to the bathroom.

+13 votes     reply to comment
INtense!
INtense! Jul 23 2010, 2:36pm replied:

we've been getting record traffic this week too due to some massive mod releases (total war, terminator, firearms) so those are the guys i'm worried about. We really want the site to be up and running 100% so we can help them distribute their mods to as many people as possible.

+17 votes     reply to comment
chubber
chubber Jul 23 2010, 7:56pm replied:

not to mention Dark Times II: Rising Son coming out tomorrow...

+2 votes     reply to comment
Nilghai
Nilghai Jul 23 2010, 2:54pm says:

I'm just glad to hear that nothing is lost. good luck with it Intense :)

+3 votes     reply to comment
NGS616
NGS616 Jul 23 2010, 2:57pm says:

The provider of your file-store should compensate you with a little free service.

Good luck, though.

+3 votes     reply to comment
INtense!
INtense! Jul 23 2010, 3:11pm replied:

they've fulfilled a few SLA claims, but i'd much rather the site work while they sort out the bottleneck!

+4 votes     reply to comment
qpac
qpac Jul 23 2010, 3:17pm says:

That explains some things. Well, I hope it gets better soon.

+3 votes     reply to comment
mastersrp
mastersrp Jul 23 2010, 3:40pm says:

Glad to know. I haven't really felt the impact though, but I haven't been downloading anything. Media seemed to work for me without a hitch too... Weird.. Good thing that you're fixing what's wrong though!

+2 votes     reply to comment
INtense!
INtense! Jul 23 2010, 4:02pm replied:

you're lucky then... things have been very patchy!

+3 votes     reply to comment
icantfly1
icantfly1 Jul 23 2010, 4:49pm says:

Ah so that's why some downloads didn't work..
Anyway, thanks for informing us!

+3 votes     reply to comment
ʇɐuʞɐʇǝp
ʇɐuʞɐʇǝp Jul 23 2010, 6:43pm says:

I'm lucky. I get the same as mastersrp.

+2 votes     reply to comment
Salsa_Shark
Salsa_Shark Jul 23 2010, 10:23pm says:

I’ve had some trouble with moddb this past week, I was blaming Telstra for it, I guess this isn’t the case, ho well, it’s nice that you let us know.

+4 votes     reply to comment
chubber
chubber Jul 23 2010, 10:33pm says:

I couldn't even load moddb at all at one point

+3 votes     reply to comment
ammd
ammd Jul 23 2010, 10:34pm says:

I know this is offtopic but we need a "clear all" button for all those updates.^^

+4 votes     reply to comment
SCΛRECROW
SCΛRECROW Jul 25 2010, 3:58am replied:

That's a good idea.

+2 votes     reply to comment
galaxy366
galaxy366 Jul 25 2010, 9:05am replied:

that is a great idea xD

+2 votes     reply to comment
Druza
Druza Jul 23 2010, 11:00pm says:

Thanks for actually explaining this to us instead of telling us nothing or just saying it was "technical difficulties".

+6 votes     reply to comment
BlabVoid
BlabVoid Jul 25 2010, 2:25am says:

Last night I was having trouble posting a news post because of this, wouldn't let me post it because it said the image file path wasn't valid.

Now I know why. :P

+1 vote     reply to comment
Dragonlord
Dragonlord Jul 25 2010, 9:33am replied: Online

Yeah, it's rather unfortunate this situation. News post delayed now nearly a week since I can't get it up. Maybe I've got more luck in the days to come :/

+1 vote     reply to comment
Cremat0r
Cremat0r Jul 25 2010, 2:35am says:

Oh ,nice, I was kicked like 10 times off Moddb, it gets annoying after a while :S

+1 vote     reply to comment
jstubbles
jstubbles Jul 25 2010, 4:52am says:

Unfortunately coinciding with my HTML editing of our page, hah.

Hopefully it gets fixed up good! :)

+1 vote     reply to comment
USSJboy
USSJboy Jul 25 2010, 6:29am says:

Wow. Thats kinda interesting. So far I've had no problems on Moddb. Speed is still good, pics load and downloads still going strong.

Cheers :)

+1 vote     reply to comment
xXMaNiAcXx
xXMaNiAcXx Jul 25 2010, 4:34pm says:

Nice to know that the problem wasn't only with me, and it seems to be really stable now, just keep up with your work and make the server up until I finish to read my updates :D

+1 vote     reply to comment
masterchief396
masterchief396 Jul 31 2010, 6:44pm says:

Sorry to hear that I was wondering what was going on. I hope things get cleared up.

+1 vote     reply to comment
lancerfour
lancerfour Aug 1 2010, 12:28am says:

Hope all goes well, and thanks for the update.

+1 vote     reply to comment
Post a Comment
click to sign in

You are not logged in, your comment will be anonymous unless you join the community today (totally free - or sign in with your social account on the right) which we encourage all contributors to do.

2000 characters limit; HTML formatting and smileys are not supported - text only

News
Browse
News
Report Abuse
Report article
Related Groups
IndieDB
IndieDB Official group with 1,985 members
ModDB
ModDB Official group with 3,789 members