Chaotic Dreams

Chaotic Dreams Official Forums
It is currently Thu May 25, 2017 11:57 pm

All times are UTC




Post new topic Reply to topic  [ 2 posts ] 
Author Message
 Post subject: Downtime explaination
PostPosted: Thu May 08, 2014 11:53 pm 
Offline
Chaotic Dreams Team
User avatar

Joined: Sun Aug 29, 2004 4:50 pm
Posts: 440
Location: Proxy King Lair
Whoever commonly visits these forums may have noticed a few days of downtime...

So, here's what happened -- the short version, because I really don't feel like giving all the details, since I've just spent 4 days working on this, it's 1:50 AM and I'm tired.

On May 5th, 12:55:24 UTC, one of the hard drives failed. Fortunately, it was in RAID 1, and thus no data was lost ...
... until literally 4 minutes and 4 seconds later, when the other HDD started failing.

The initial point of failure was one of MySQL database files, which made the daemon do an emergency exit. That's when I noticed something was wrong -- the websites couldn't connect to the MySQL server, and just trying to restart the MySQL server didn't work (it failed with a read error).

Fortunately, I've been doing daily database backups (off-site), so that wasn't a disaster.
The rest of the data on the failing drive was still okay, so I could back that up before both drives were replaced (merely as a conveniency -- it was not critical).
I reinstalled the system after that, restored relevant configuration and started up the servers again. It took a lot of time.

I hope the database backup I was working off of was early enough that there wasn't any incorrect data due to some sort of unnoticed corruption; I basically just took May 4th (I do have backups for every day, but just took the latest "probably fine").

tl;dr: Both HDDs (mirrored) on the server failed a few minutes apart. I had backups.


Top
 Profile  
 
PostPosted: Fri May 09, 2014 3:09 am 
Offline
Chaotic Dreams Team
User avatar

Joined: Thu May 09, 2002 2:55 pm
Posts: 7996
Hi Tim.

I did notice, and I did notice your continued dedication to fix 'er up.

Thank you Mr. Sir.

_________________
@RFlaggTOC


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 2 posts ] 

All times are UTC


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
cron
Powered by phpBB® Forum Software © phpBB Group