UPDATED: We are replacing our back end storage system shortly to resolve downtime issue and slow speeds.

Posted by on

If you have visited ModDB or IndieDB in the past week, you have probably noticed our servers seem to be going up and down like a yo-yo. Unfortunately that is the case due to a recent upgrade we did. In this upgrade we provisioned a new webserver, new database server and the troublesome new SAN iSCSI filestore.

Whilst this upgrade should have made things lightning fast (and did for a while), it appears there is a critical connection bottleneck between the all important SAN box and our 3 webservers. The SAN which should easily do 1000mbit to all 3 boxes at once is instead choking at 30mbit which is waaaay lower than required. As a result everytime file transfers spike up (which is all the time) CPU load spikes and the system crashes. No data is lost but emails, media and downloads are all very unstable right now and there is no quick fix. So please bear with us while we address the problem as efficiently as possible.

UPDATE: 1st August

We will be replacing our backend SAN in the coming 2 weeks. Site performance for the next 2 weeks will likely be sluggish to allow for migrating terabytes of data from the faulty SAN to a replacement device.

Thanks all for your patience and understanding!

Comments
MrMattWebb
MrMattWebb

Wow, all the images are gone o_O

good luck :)

Reply Good karma Bad karma+2 votes
INtense! Author
INtense!

Not gone, just not displaying

Reply Good karma+7 votes
Dragonlord
Dragonlord

Best image compression ever... 0bytes in size :D

Reply Good karma Bad karma+31 votes
Anonymousperson
Anonymousperson

I laughed at this... then felt like a nerd
Which isn't really a bad thing in retrospect, I suppose

Reply Good karma Bad karma+6 votes
NullSoldier
NullSoldier

I for one welcome our new image eating SAN iSCSI overlords.

Reply Good karma Bad karma+11 votes
1337-1
1337-1

All hail the overlords.

(quick, the overlords are hungry, we need more images!)

Reply Good karma Bad karma+4 votes
ytres
ytres

All your images are belong to us!

Reply Good karma Bad karma+2 votes
Dremth
Dremth

It's alright guys. As much as I'd love to be on ModDB 24/7, we users need outages every now and then so we can do other things like sleep, eat, and go to the bathroom.

Reply Good karma Bad karma+13 votes
INtense! Author
INtense!

we've been getting record traffic this week too due to some massive mod releases (total war, terminator, firearms) so those are the guys i'm worried about. We really want the site to be up and running 100% so we can help them distribute their mods to as many people as possible.

Reply Good karma+17 votes
chubber
chubber

not to mention Dark Times II: Rising Son coming out tomorrow...

Reply Good karma Bad karma+2 votes
Nilghai
Nilghai

I'm just glad to hear that nothing is lost. good luck with it Intense :)

Reply Good karma Bad karma+3 votes
NGS616
NGS616

The provider of your file-store should compensate you with a little free service.

Good luck, though.

Reply Good karma Bad karma+3 votes
INtense! Author
INtense!

they've fulfilled a few SLA claims, but i'd much rather the site work while they sort out the bottleneck!

Reply Good karma+4 votes
qpac
qpac

That explains some things. Well, I hope it gets better soon.

Reply Good karma Bad karma+3 votes
mastersrp
mastersrp

Glad to know. I haven't really felt the impact though, but I haven't been downloading anything. Media seemed to work for me without a hitch too... Weird.. Good thing that you're fixing what's wrong though!

Reply Good karma Bad karma+2 votes
INtense! Author
INtense!

you're lucky then... things have been very patchy!

Reply Good karma+3 votes
icantfly1
icantfly1

Ah so that's why some downloads didn't work..
Anyway, thanks for informing us!

Reply Good karma Bad karma+3 votes
ʇɐuʞɐʇǝp
ʇɐuʞɐʇǝp

I'm lucky. I get the same as mastersrp.

Reply Good karma Bad karma+2 votes
Salsa_Shark
Salsa_Shark

I’ve had some trouble with moddb this past week, I was blaming Telstra for it, I guess this isn’t the case, ho well, it’s nice that you let us know.

Reply Good karma Bad karma+4 votes
chubber
chubber

I couldn't even load moddb at all at one point

Reply Good karma Bad karma+3 votes
ammd
ammd

I know this is offtopic but we need a "clear all" button for all those updates.^^

Reply Good karma Bad karma+4 votes
SCΛRECROW
SCΛRECROW

That's a good idea.

Reply Good karma Bad karma+2 votes
galaxy366
galaxy366

that is a great idea xD

Reply Good karma Bad karma+2 votes
Druza
Druza

Thanks for actually explaining this to us instead of telling us nothing or just saying it was "technical difficulties".

Reply Good karma Bad karma+6 votes
BlabVoid
BlabVoid

Last night I was having trouble posting a news post because of this, wouldn't let me post it because it said the image file path wasn't valid.

Now I know why. :P

Reply Good karma Bad karma+1 vote
Dragonlord
Dragonlord

Yeah, it's rather unfortunate this situation. News post delayed now nearly a week since I can't get it up. Maybe I've got more luck in the days to come :/

Reply Good karma Bad karma+1 vote
Cremat0r
Cremat0r

Oh ,nice, I was kicked like 10 times off Moddb, it gets annoying after a while :S

Reply Good karma Bad karma+1 vote
jstubbles
jstubbles

Unfortunately coinciding with my HTML editing of our page, hah.

Hopefully it gets fixed up good! :)

Reply Good karma Bad karma+1 vote
USSJboy
USSJboy

Wow. Thats kinda interesting. So far I've had no problems on Moddb. Speed is still good, pics load and downloads still going strong.

Cheers :)

Reply Good karma Bad karma+1 vote
xXMaNiAcXx
xXMaNiAcXx

Nice to know that the problem wasn't only with me, and it seems to be really stable now, just keep up with your work and make the server up until I finish to read my updates :D

Reply Good karma Bad karma+1 vote
masterchief396
masterchief396

Sorry to hear that I was wondering what was going on. I hope things get cleared up.

Reply Good karma Bad karma+1 vote
lancerfour
lancerfour

Hope all goes well, and thanks for the update.

Reply Good karma Bad karma+1 vote
Post a comment
Sign in or join with:

Only registered members can share their thoughts. So come on! Join the community today (totally free - or sign in with your social account on the right) and join in the conversation.

News
Browse
News
New
Post news
Share
Related Groups
Indie DB
Indie DB Official with 2,368 members
Mod DB
Mod DB Official with 4,131 members