I was listening to Marketing Over Coffee a few weeks ago and heard a sad tale of woe from co-host John Wall. His blog, Ronin Marketeer, was down for four days. Hosed, in fact.

Literally.

On February 20th, there was an an accident during the annual inspection of the fire-suppression system at WestHost’s data center. If you rewrite WestHost’s account of the incident in the active voice, it amounts to “The vendor forgot to remove an actuator before the inspection, and this triggered a release of Inergen all over the data center.”

According to Wikipedia, Inergen is non-toxic…to humans. It is, alas, highly poisonous to computer servers. And John’s blog just happened to be on one of the worst-affected machines.

Bye-bye blog.

The good news is, WestHost actually had backups of its clients’ sites. Not all hosting companies actually back up your website for you. (You can usually make your own backups through the control panel, but you may or may not be able to automate this process. Of course, if you have a traditional HTML site and edit the files on your computer before uploading them, you should be able to back up your local versions easily.)

The better news (theoretically) was that John, being a smart guy with lots of IT experience, had recently made his own backup of his blog. That meant he was starting out in better shape than Jeff Atwood over at Coding Horror, who had to rely on other people to piece together bits of his lost blog for him.

But the first attempt to restore Ronin Marketeer left a bit to be desired. When I sent a sympathetic inquiry to John after hearing the podcast, he sent me a link to a post he had titled “When Even a Backup Is Not Enough.”

As you can see, everything is all f’d up here.

Over a week ago disaster struck at my hosting company, during a fire alarm test the suppression system was triggered, hosing all the servers. This blog was dead for a full week.

We were offered to move our hosting from the version 3 infrastructure to v4, and I took up the offer since it got my domain back 2 days earlier. Unfortunately the new environment is not the same – even though I have a full backup of my Database that supports this blog, the new system does not allow you access to the directory where that data is kept.

I’m no expert in MySQL, but it looks like I’ve gone from having my own instance to sharing one on the server with everyone else.

The end result is that all my archives are gone for now and my Google juice vanishing as there’s no access to any of my archives. It looks like my only path is to install WP and MySQL on a box of my own, then do a WordPress export so I can then import it back in. I cannot believe that having the actual files is not enough for me to do a restore.

“My god,” you may be thinking. “If having the backup is no good, why bother making one?”

But if he hadn’t had the backup, the story would not have had a happy ending, and it does. John had to do some heroically geeky things, but he was able to get the blog back up and running. He did lose some comments, probably due to the nature of the restore process, but everything else seems to be intact. John started Ronin Marketeer in November 2006, and he’s a pretty prolific blogger. It would have been a serious loss, and no fun to try to reconstruct from the Google cache and the Wayback Machine.

I’m betting John will be especially interested in the WordPress backup plugin I’m going to be writing about next week. Everyone else certainly seems to be, and I’m very impressed so far.

If you search the WordPress plugin repository for “backup”, you’ll get—as of today—195 results. I wrote about two of those plugins, WordPress Database Backup and WordPress Backup by BTE, just about a year ago, and installed them on all 8 of my own WP sites, as well as insisting that my clients use WP-DB-Backup at minimum.

Both of those plugins back up different parts of a WordPress installation and then either save it there on the server or e-mail it to the admin. I get a lot of e-mails with database backups, as you can imagine. These aren’t large files, and it’s not too time-consuming to save them with other client files and let them get backed up as part of my regular backup routine.

But the BTE plugin backs up your uploads, plugins, and themes directories. And those can start to get pretty large after a while. Not large in absolute terms of how much room I have on my hard drive or backup drives, but large in terms of what it’s convenient to receive by e-mail, especially multiplied by eight or more. And then there’s the fact that the mail server for Author-izer.com, my primary business website, absolutely WILL NOT accept the plugins backup file, even though it’s a ZIP. It believes that file is full of malicious code out to attack me, and refuses it. (Ta ever so, mailer-daemon.) And then there’s the lack of versioning, because each week’s backups of those directories has the same name. These are minor annoyances, but real.

Now there’s a new plugin that combines the functions of these two stalwarts, with a few extras besides: Automatic WordPress Backup, sponsored by Melvin Ram’s Web Design Company, developed by Dan Coulter.

AWB lets you schedule daily, weekly, or monthly backups of your database, your wp-config.php file, your wp-content folder (themes, plugins, and uploads), and even your .htaccess file. Instead of e-mailing them to you, it uploads them to Amazon S3.

S3 stands for “Simple Storage Service.” It’s not actually quite as simple as all that, but the idea is that you only pay for as much storage and bandwidth as you actually use. Since a typical WordPress installation—even with a lot of plugins and uploads—isn’t very large, backing up via S3 shouldn’t cost more than a few cents each month.

Before you install the plugin, go to Amazon S3 and sign up for an account if you don’t have one already. (Signing up is free.) Once you get that confirmed, go to “Security Credentials” under the “Your Account” tab to get the information you’ll need to configure the plugin.

Then log into your WordPress dashboard and install the plugin normally. There’s a handy YouTube video that walks you through installation over on the AWB website. This is a nice touch. I just wish Amazon had done the same for S3! Once you activate AWB, you’ll be prompted to configure the settings. If you need to find them later, they have their own options submenu at the foot of the right sidebar.

Fill in your AWS Access Key and Secret Key, create an S3 “bucket” (the Ur-Guru was a bit disparaging about that term) to store your backup in, and decide what you want to back up, how often, and when to get rid of old backups.

I like both the option to automatically delete old backups and the option to make backups only once a month. There are sites that I don’t update any more often than that, even though I know I should.

When I first installed AWB on the test blog over at the Podcast Asylum, it didn’t seem to work. After you hit “Save Changes and Back Up Now,” you see a message telling you that there will be a link to download your most recent backup when you come back to that page—but there was never any link.

That was when I realized I didn’t know how to see what was on my Amazon S3 server. Amazon’s own site wasn’t too helpful; their AWS Management Console doesn’t work with S3 yet. Fortunately, there are plenty of other tools to let you get access to your S3 account. I picked S3Fox, a plugin for the Firefox web browser. Once I’d installed that, I was able to confirm that while my “testblog” bucket had been created, there was nothing in it.

Yet when I installed Automatic WordPress Backup here on the FileSlinger Backup Blog, it worked just fine. Was this a hosting issue, I wondered? (The Podcast Asylum site is on Dreamhost and the Backup Blog is on GoDaddy—and I don’t actually recommend either of them for WordPress hosting these days.) I got in touch with Melvin Ram, who walked me through installing the development version of the plugin, due for release next week sometime.

That fixed the problem: after clicking “Save Changes and Back Up Now,” I saw the following message:

That “Restore from a backup” tab is new in the development version; in the current version, 1.0.2, you have to download the backup and restore it manually. Not quite all the bugs are out of the restore process yet, though. I double-checked in S3Fox, and sure enough, the ZIP file was there.

I did notice, however, that while the ZIP file contained my wp-config.php file, my .htaccess file, and my wp-content folder, it was missing my WordPress database. (So was the one from fileslinger.com.) So I might want to wait through a few more development versions before I completely replace WP-DB-Backup with Automatic WordPress Backup.

Nevertheless, I think WDC has made a great start with this plugin, and that it’s going to be extremely useful once they’ve got the bugs out.

Yesterday the Ur-Guru pointed me to a post on Coding Horror entitled “International Backup Awareness Day.” Coding Horror is normally a blog, and the permalinks to posts don’t normally look like the one I just pasted in there. Depending on when you read this, clicking on that link might get you a “404: Page Not Found” error. Thanks to catastrophic data loss, Coding Horror is only pretending to be a blog right now.

Here’s the story. Jeff Atwood, geek extraordinaire, hosted his blog on a virtual machine (VM) on a server at a web hosting company. VMs are great for developers, because you can simulate different operating systems in order to test your software on them, and you can take snapshots and re-create them easily.

Unlike Jeff Atwood or the Ur-Guru, I’ve never worked with a VM. To use them you need more RAM than I have at my disposal. I can see how it might be handy to have one to test backup software on without cluttering up the registry of my main machine, though. But apparently backing up the contents of a VM doesn’t work quite the same way as backing up your ordinary operating system and files. This is a setup for a very dangerous situation: when you think you have backups, but don’t. (Have I said “Test your backups” lately? Test your backups.) And this, it seems, is exactly how Jeff lost his blog:

1. The server experienced routine hard drive failure. (Ed. note: hard drive failure is described as “routine.” In data centers, where drives are spinning 24/7, that’s exactly what it is.)
2. Because of the hard drive failure, the virtual machine image hosting this blog was corrupted.
3. Because the blog was hosted in a virtual machine, the standard daily backup procedures at the host were unable to ever back it up.
4. Because I am an idiot, I didn’t have my own (recent) backups of Coding Horror. Man, I wish I had read some good blog entries on backup strategies!
5. Because there were no good backups, there was catastrophic data loss. Fin, draw curtain, exeunt stage left.

Now, I don’t know what blogging platform Jeff was using. Given that he’s one of those extreme geek types, it could be something really obscure, even something he created himself. (Power geeks are like that; they’re as likely to insist on developing their own tools as to use anyone else’s.) I don’t even know what operating system his VM was running. But I know that there were ways to back up this blog when I was using Blogger (published by FTP to my own website), and there’s no excuse for not backing up WordPress blogs, since there are handy plugins to make it easier. And any offline blog editor like Windows Live Writer or Ecto will save local copies of your posts, so you can back them up along with the rest of the data on your hard drive. (Back in the days before Windows Live Writer, I used to write my blog posts in Microsoft Word, but you don’t actually want to paste from Word into anything that uses HTML. It did, however, mean that I had local copies.)

Jeff was able to re-build the text portion of his blog in HTML thanks to a fellow extreme geek who’s been archiving the Internet, but lost many of the images (which are not, apparently, on his hard drive, or not readily identifiable from among those on his hard drive). I shudder to think just how much work this must have been—and how much more work it will be to convert it back into blog format if he chooses to do that.

The lessons Jeff Atwood learned from the demise of Coding Horror are not just for geeks.

What can we all learn from this sad turn of events?

1. I suck.
2. No, really, I suck.
3. Don’t rely on your host or anyone else to back up your important data. Do it yourself. If you aren’t personally responsible for your own backups, they are effectively not happening.
4. If something really bad happens to your data, how would you recover? What’s the process? What are the hard parts of recovery? I think in the back of my mind I had false confidence about Coding Horror recovery scenarios because I kept thinking of it as mostly text. Of course, the text turned out to be the easiest part. The images, which I had thought of as a “nice to have”, were more essential than I realized and far more difficult to recover. Some argue that we shouldn’t be talking about “backups”, but recovery.
5. It’s worth revisiting your recovery process periodically to make sure it’s still alive, kicking, and fully functional.
6. I’m awesome! No, just kidding. I suck.

So when, exactly, is International Backup Awareness Day? Today. Yesterday. This week. This month. This year. It’s a trick question. Every day is International Backup Awareness Day. And the sooner I figure that out, the better off I’ll be.

Have you checked your backups lately? Now might be a really good time.

I was going to write about this last week, but I neglected to provide myself with a backup set of arms to type with, so the post got postponed due to physical incapacity. In the meantime, another example of why this is important landed in my lap.

If you’ve been reading this blog for a while, you’ll know that one of my businesses is podcasting. On March 21st I was recording the “Get Published!” workshop for the Bay Area Independent Publishers Association (BAIPA). I tend toward the minimalist in my recording equipment, relying primarily on my iriver IFP-795, a remarkably capable if slightly fiddly little device. But this workshop had two sets of breakout sessions, which meant I needed to scrounge up a couple of extra digital recorders in order to have all the sessions covered. One was an older Sony, one a new Marantz PMD-620.

Being the backup-centric person that I am, I decided to record the keynote speakers with both the Marantz and the iriver. This was partly to compare the quality of the recordings, but I also wanted to be absolutely sure that I had a usable recording of those presentations.

It’s a good thing I did it, too. Halfway through the second keynote speaker’s talk, the battery on the iriver gave out. (It hadn’t looked that low, or I would have put a new one in before he started.) So I had to take some extra editing time mixing in the recording from the Marantz, but at least I had the whole thing. (Because I was unfamiliar with the settings on the Marantz, the quality of the recording on the iriver was much better, so I used it where I had the option to.)

And I heartily wished I’d had multiple recorders for the breakout sessions, as two of them were cut short. In one case, the storage on the Sony filled up (there was no indicator to say how much room was left on it, but it only holds about 90 minutes). In the other, it appears the speaker accidentally switched off the Marantz. Given that it was hard enough getting sufficient devices to have one recorder for each session, perhaps what I really need is backup staff. I don’t think I should go into the conference recording business. There are people out there who do it better.

The experience was definitely a lesson in the importance of backups, and it seemed appropriate to tie it in with a few notes I’ve been getting lately from colleagues and backup service providers. The day before that conference, Arco (I thought they were a gas station) sent me a message saying “Are your digital photos safe on your computer?” and advertising EZBackup101. (Their $10 off coupon expired yesterday, though.) My colleague Stu Sweetow from AVC Video sent out a newsletter advising clients on archiving their audio-visual materials. (In particular, you need to transfer your VHS tapes to digital format.) And my pals at Mozy have been talking about doing a spring clean and consolidating your CDs into storage in the cloud. (Without mentioning just how long that will take to upload unless you have FIOS…) There are probably half a dozen companies that have approached me that combine online backup with photo or video sharing, and they’re all absolutely correct that you want to keep multiple copies.

What I’m talking about here is a little different, however. When it’s really important to have that photo, video, or audio recording, you want to make multiple copies. If you’ve taken your kids in for professional photos, you’ve probably noticed that the photographer takes several shots of each pose. You’ve probably also seen that professional video teams shoot with multiple cameras. That’s partly because video is more interesting if you can use multiple angles, but it also means that if one doesn’t come out, you’re still covered. Back in my academic days, I took two cameras along on some trips, one to get slides and one to get prints. (Anyone remember film?)

This sounds like an excuse to sell (or buy) more gadgets. But I bet most people have a cameraphone in addition to a camera. A “record” function on a smartphone, or a built-in mic on a laptop, in addition to a recording device. Something that can act as a backup when it’s really, really important to have one. Because the first rule of technology, as the Ur-Guru says, is “Never assume it will work.”

Here’s my second example, from Tuesday night, March 31st.

I had learned on Monday that the AJC was going to be hosting a Business & Technology Program in San Francisco with Craig Newmark (founder of craigslist) and Matt Mullenweg, founder of WordPress. How could I possibly pass that up? Like everyone else, I use craigslist, and you all know by now what a WordPress fangirl I am.

When I RSVPed, I asked about recording. I was told that they planned to record the event for a podcast, but seeing as I ran the Podcast Asylum, they might want to ask me some questions. They had, the development assistant explained, rented a Zoom H2 from a local community college. The Zoom H2 is a nice little device. I’m thinking of getting one, myself. It turned out there was a problem with this one, however: apparently the battery had exploded and fried the SD card, rendering it inoperative.

So, in fact, we ended up using my trusty iriver to record the event. Dreadful acoustics (270-degree floor-to-ceiling windows, hardwood floors, and feedback in the amplification system), but it came out all right.

If I buy the Zoom, or the Marantz, you can be sure I won’t be leaving the iriver at home.