Even Geeks Suffer from Data Loss

Yesterday the Ur-Guru pointed me to a post on Coding Horror entitled “International Backup Awareness Day.” Coding Horror is normally a blog, and the permalinks to posts don’t normally look like the one I just pasted in there. Depending on when you read this, clicking on that link might get you a “404: Page Not Found” error. Thanks to catastrophic data loss, Coding Horror is only pretending to be a blog right now.

Here’s the story. Jeff Atwood, geek extraordinaire, hosted his blog on a virtual machine (VM) on a server at a web hosting company. VMs are great for developers, because you can simulate different operating systems in order to test your software on them, and you can take snapshots and re-create them easily.

Unlike Jeff Atwood or the Ur-Guru, I’ve never worked with a VM. To use them you need more RAM than I have at my disposal. I can see how it might be handy to have one to test backup software on without cluttering up the registry of my main machine, though. But apparently backing up the contents of a VM doesn’t work quite the same way as backing up your ordinary operating system and files. This is a setup for a very dangerous situation: when you think you have backups, but don’t. (Have I said “Test your backups” lately? Test your backups.) And this, it seems, is exactly how Jeff lost his blog:

1. The server experienced routine hard drive failure. (Ed. note: hard drive failure is described as “routine.” In data centers, where drives are spinning 24/7, that’s exactly what it is.)
2. Because of the hard drive failure, the virtual machine image hosting this blog was corrupted.
3. Because the blog was hosted in a virtual machine, the standard daily backup procedures at the host were unable to ever back it up.
4. Because I am an idiot, I didn’t have my own (recent) backups of Coding Horror. Man, I wish I had read some good blog entries on backup strategies!
5. Because there were no good backups, there was catastrophic data loss. Fin, draw curtain, exeunt stage left.

Now, I don’t know what blogging platform Jeff was using. Given that he’s one of those extreme geek types, it could be something really obscure, even something he created himself. (Power geeks are like that; they’re as likely to insist on developing their own tools as to use anyone else’s.) I don’t even know what operating system his VM was running. But I know that there were ways to back up this blog when I was using Blogger (published by FTP to my own website), and there’s no excuse for not backing up WordPress blogs, since there are handy plugins to make it easier. And any offline blog editor like Windows Live Writer or Ecto will save local copies of your posts, so you can back them up along with the rest of the data on your hard drive. (Back in the days before Windows Live Writer, I used to write my blog posts in Microsoft Word, but you don’t actually want to paste from Word into anything that uses HTML. It did, however, mean that I had local copies.)

Jeff was able to re-build the text portion of his blog in HTML thanks to a fellow extreme geek who’s been archiving the Internet, but lost many of the images (which are not, apparently, on his hard drive, or not readily identifiable from among those on his hard drive). I shudder to think just how much work this must have been—and how much more work it will be to convert it back into blog format if he chooses to do that.

The lessons Jeff Atwood learned from the demise of Coding Horror are not just for geeks.

What can we all learn from this sad turn of events?

1. I suck.
2. No, really, I suck.
3. Don’t rely on your host or anyone else to back up your important data. Do it yourself. If you aren’t personally responsible for your own backups, they are effectively not happening.
4. If something really bad happens to your data, how would you recover? What’s the process? What are the hard parts of recovery? I think in the back of my mind I had false confidence about Coding Horror recovery scenarios because I kept thinking of it as mostly text. Of course, the text turned out to be the easiest part. The images, which I had thought of as a “nice to have”, were more essential than I realized and far more difficult to recover. Some argue that we shouldn’t be talking about “backups”, but recovery.
5. It’s worth revisiting your recovery process periodically to make sure it’s still alive, kicking, and fully functional.
6. I’m awesome! No, just kidding. I suck.

So when, exactly, is International Backup Awareness Day? Today. Yesterday. This week. This month. This year. It’s a trick question. Every day is International Backup Awareness Day. And the sooner I figure that out, the better off I’ll be.

Have you checked your backups lately? Now might be a really good time.