Last thursday, our server suffered from a catastrophic hard drives failure. We were using software RAID on two SATA disks, but because of our stupid datacenter’s team (DigitalRoot not to name it…) shutting down the server by removing the power plug to move it to another location, the RAID setup became unstable. However, we continued to use it…

BAD IDEA!!!! After some weeks, it totally broke up. We lost everything… :grr: Good occasion to test our backup system. After a rapid re-install (from scratch), we recovered all backed-up data from the backup server, and we were up and running some hours later… :stress:

Lessons Learned:

• Perform an image of the base system – to do
• Perform backup of: /etc, /home, /databases – OK
• DO NOT USE software raid – OK (sata raid controller installed)
• Backup: perform periodic “copy and delete” sync. By that, I mean not only copy all modified files to the backup server, but also remove them from it when they are removed on the production server. Indeed, after the data recovery, I had a lot of unread e-mails… When I read and deleted them, some of them were in fact already backed up. :dead: