I was listening to Marketing Over Coffee a few weeks ago and heard a sad tale of woe from co-host John Wall. His blog, Ronin Marketeer, was down for four days. Hosed, in fact.

Literally.

On February 20th, there was an an accident during the annual inspection of the fire-suppression system at WestHost’s data center. If you rewrite WestHost’s account of the incident in the active voice, it amounts to “The vendor forgot to remove an actuator before the inspection, and this triggered a release of Inergen all over the data center.”

According to Wikipedia, Inergen is non-toxic…to humans. It is, alas, highly poisonous to computer servers. And John’s blog just happened to be on one of the worst-affected machines.

Bye-bye blog.

The good news is, WestHost actually had backups of its clients’ sites. Not all hosting companies actually back up your website for you. (You can usually make your own backups through the control panel, but you may or may not be able to automate this process. Of course, if you have a traditional HTML site and edit the files on your computer before uploading them, you should be able to back up your local versions easily.)

The better news (theoretically) was that John, being a smart guy with lots of IT experience, had recently made his own backup of his blog. That meant he was starting out in better shape than Jeff Atwood over at Coding Horror, who had to rely on other people to piece together bits of his lost blog for him.

But the first attempt to restore Ronin Marketeer left a bit to be desired. When I sent a sympathetic inquiry to John after hearing the podcast, he sent me a link to a post he had titled “When Even a Backup Is Not Enough.”

As you can see, everything is all f’d up here.

Over a week ago disaster struck at my hosting company, during a fire alarm test the suppression system was triggered, hosing all the servers. This blog was dead for a full week.

We were offered to move our hosting from the version 3 infrastructure to v4, and I took up the offer since it got my domain back 2 days earlier. Unfortunately the new environment is not the same – even though I have a full backup of my Database that supports this blog, the new system does not allow you access to the directory where that data is kept.

I’m no expert in MySQL, but it looks like I’ve gone from having my own instance to sharing one on the server with everyone else.

The end result is that all my archives are gone for now and my Google juice vanishing as there’s no access to any of my archives. It looks like my only path is to install WP and MySQL on a box of my own, then do a WordPress export so I can then import it back in. I cannot believe that having the actual files is not enough for me to do a restore.

“My god,” you may be thinking. “If having the backup is no good, why bother making one?”

But if he hadn’t had the backup, the story would not have had a happy ending, and it does. John had to do some heroically geeky things, but he was able to get the blog back up and running. He did lose some comments, probably due to the nature of the restore process, but everything else seems to be intact. John started Ronin Marketeer in November 2006, and he’s a pretty prolific blogger. It would have been a serious loss, and no fun to try to reconstruct from the Google cache and the Wayback Machine.

I’m betting John will be especially interested in the WordPress backup plugin I’m going to be writing about next week. Everyone else certainly seems to be, and I’m very impressed so far.

Yesterday the Ur-Guru pointed me to a post on Coding Horror entitled “International Backup Awareness Day.” Coding Horror is normally a blog, and the permalinks to posts don’t normally look like the one I just pasted in there. Depending on when you read this, clicking on that link might get you a “404: Page Not Found” error. Thanks to catastrophic data loss, Coding Horror is only pretending to be a blog right now.

Here’s the story. Jeff Atwood, geek extraordinaire, hosted his blog on a virtual machine (VM) on a server at a web hosting company. VMs are great for developers, because you can simulate different operating systems in order to test your software on them, and you can take snapshots and re-create them easily.

Unlike Jeff Atwood or the Ur-Guru, I’ve never worked with a VM. To use them you need more RAM than I have at my disposal. I can see how it might be handy to have one to test backup software on without cluttering up the registry of my main machine, though. But apparently backing up the contents of a VM doesn’t work quite the same way as backing up your ordinary operating system and files. This is a setup for a very dangerous situation: when you think you have backups, but don’t. (Have I said “Test your backups” lately? Test your backups.) And this, it seems, is exactly how Jeff lost his blog:

1. The server experienced routine hard drive failure. (Ed. note: hard drive failure is described as “routine.” In data centers, where drives are spinning 24/7, that’s exactly what it is.)
2. Because of the hard drive failure, the virtual machine image hosting this blog was corrupted.
3. Because the blog was hosted in a virtual machine, the standard daily backup procedures at the host were unable to ever back it up.
4. Because I am an idiot, I didn’t have my own (recent) backups of Coding Horror. Man, I wish I had read some good blog entries on backup strategies!
5. Because there were no good backups, there was catastrophic data loss. Fin, draw curtain, exeunt stage left.

Now, I don’t know what blogging platform Jeff was using. Given that he’s one of those extreme geek types, it could be something really obscure, even something he created himself. (Power geeks are like that; they’re as likely to insist on developing their own tools as to use anyone else’s.) I don’t even know what operating system his VM was running. But I know that there were ways to back up this blog when I was using Blogger (published by FTP to my own website), and there’s no excuse for not backing up WordPress blogs, since there are handy plugins to make it easier. And any offline blog editor like Windows Live Writer or Ecto will save local copies of your posts, so you can back them up along with the rest of the data on your hard drive. (Back in the days before Windows Live Writer, I used to write my blog posts in Microsoft Word, but you don’t actually want to paste from Word into anything that uses HTML. It did, however, mean that I had local copies.)

Jeff was able to re-build the text portion of his blog in HTML thanks to a fellow extreme geek who’s been archiving the Internet, but lost many of the images (which are not, apparently, on his hard drive, or not readily identifiable from among those on his hard drive). I shudder to think just how much work this must have been—and how much more work it will be to convert it back into blog format if he chooses to do that.

The lessons Jeff Atwood learned from the demise of Coding Horror are not just for geeks.

What can we all learn from this sad turn of events?

1. I suck.
2. No, really, I suck.
3. Don’t rely on your host or anyone else to back up your important data. Do it yourself. If you aren’t personally responsible for your own backups, they are effectively not happening.
4. If something really bad happens to your data, how would you recover? What’s the process? What are the hard parts of recovery? I think in the back of my mind I had false confidence about Coding Horror recovery scenarios because I kept thinking of it as mostly text. Of course, the text turned out to be the easiest part. The images, which I had thought of as a “nice to have”, were more essential than I realized and far more difficult to recover. Some argue that we shouldn’t be talking about “backups”, but recovery.
5. It’s worth revisiting your recovery process periodically to make sure it’s still alive, kicking, and fully functional.
6. I’m awesome! No, just kidding. I suck.

So when, exactly, is International Backup Awareness Day? Today. Yesterday. This week. This month. This year. It’s a trick question. Every day is International Backup Awareness Day. And the sooner I figure that out, the better off I’ll be.

Have you checked your backups lately? Now might be a really good time.

Would you entrust your data to a company called “Danger”? Microsoft and T-Mobile did. And it was your data, if you were a Sidekick user.

The adventure began on October 10th. The headline in TechCrunch read “T-Mobile Sidekick Disaster: Danger’s Servers Crashed, And They Don’t Have A Backup.” Jason Kincaid, author of the TechCrunch article, was absolutely scathing on the subject:

This goes beyond FAIL, face-palm, or any of the other internet memes we’ve come to associate with incompetence. The fact that T-Mobile and/or Microsoft Danger don’t have a redundant backup is simply inexcusable, especially given the fact that the Sidekick is totally reliant on the cloud because it doesn’t store its data locally.

I’ve never used a Sidekick, but a mobile device that doesn’t store phone numbers, etc locally at all seems bizarre, and in fact I’m not sure that statement is quite accurate, given suggestions in other articles that if you keep the Sidekick charged and turned on, you would at least save anything in its current memory.

But then, I still have a “dumbphone,” so what do I know about how these things work?

By October 11th, T-Mobile had posted the following discouraging notice on its user forums:

Regrettably, based on Microsoft/Danger’s latest recovery assessment of their systems, we must now inform you that personal information stored on your device — such as contacts, calendar entries, to-do lists or photos — that is no longer on your Sidekick almost certainly has been lost as a result of a server failure at Microsoft/Danger.

Not surprisingly, the media has been all over the story. “Microsoft has said that the hardware failure that caused the problem took out both the primary and backup copies of the database that contained Sidekick users’ information,” Ina Fried wrote on October 12th. “But the question remains, why wasn’t there a true independent backup of the data?”

That would certainly be my question. Rafe Needleman, also writing for CNET on October 12th, concluded that you can’t trust the cloud because you can’t trust the people running it. The problem, in other words, is not one of technology. Tech support staff often refer to problems that start “between the keyboard and the chair.”

If it’s possible to create independent, redundant backups in your own data center, it’s possible to do it in the data centers used by cloud computing companies. The only difference  is that you can’t walk down the hall and see that they’ve done it. Some people will slack off when you aren’t there to hold them accountable, but that’s not true of everyone. As Lance Ulanoff concluded in his October 13th article, “Don’t Blame Cloud Computing for the T-Mobile Mess,”

Obviously, something went very, very wrong with T-Mobile and Microsoft’s Sidekick data set-up, but let’s not throw out the baby with the bathwater (or the cloud with the rainwater). The cloud isn’t the problem. Instead, I blame the people—as always.

But the Register, with typically British enthusiasm for a pun, declared “Danger Lurks in the Clouds” on October 18th. The danger is that all mobile devices will rely increasingly on a working connection to provide any functions at all. Nevertheless, author Bill Ray concludes:

Cloud-based servers are still more reliable than most of the kit knocking around users’ homes – the life expectancy of an Apple Time Capsule, for example, is just over 17 months according to the Time Capsule Memorial Register, so even those who are backing up locally shouldn’t be too smug.

That article concludes, in the Register’s usual tongue-in-cheek fashion, that paper is the only safe storage medium.

By October 20th, Microsoft and Danger had in fact been able to restore some of the data, as reported in CNET and on T-Mobile’s user forum. That’s a happier ending than Sidekick owners had been led to expect. I’m glad they got their data back, but if I’d been affected, I’d want more.

I’d want to know what the company was going to do differently from now on so that this wouldn’t happen again. And I’d want a free application that would let me back up all my contacts, calendar entries, etc, onto my computer. It wouldn’t even have to sync with Outlook or Google or Mac-whatever, as long as I’d be able to restore the data to my mobile device.

Finally, as an occasional naming consultant, I want to see Microsoft Danger rebranded. What incentive do you have to entrust something valuable to a company called Danger? What incentive do employees of a company called Danger have to be careful? Danger is a fun name for a company that makes games, but for data storage, it just sounds unreliable.

I woke up at 2 AM this morning. Don’t ask why, because I haven’t the faintest idea, but there I was. Since by 3:00 it was clear I wasn’t going to get back to sleep, I turned on the computers and attempted to be productive.

By about 10:30 AM I was nearly finished creating a mind map for the website redesign for a corporate client, based on their suggestions and another site they thought was a good example. I use MindJet’s MindManager Pro 6 software for this. The latest version of MindManager is 8, but I never bothered to upgrade. The program already has more features than I know how to use.

Anyway, I had opened an earlier map to check something that I wanted to add to the current map. I was finished with the earlier map and went to close it, and got prompted with one of those “Do you want to save changes?” dialogs. I hadn’t made any changes to that file, so I said “No.”

And then discovered that I’d just closed the whole program, and lost the last 10 minutes of work I’d done. Four hours of sleep, yeah. Does wonders for the brain function.

I save my work often, and I have most of my programs set to autosave at least every ten minutes. But it turns out I can make a lot of changes to a document in ten minutes, at least if it’s a mind map.

Now, I have Mind Manager set to automatically create .BAK files the way I have Word set to automatically create .WBK files. In Word, the .WBK files preserve an earlier version of the file; you can return to it if the current version gets corrupted somehow. I assumed the same idea held true with MindManager, but I couldn’t even get it to open the .BAK file, so I had no way to tell whether it had preserved my new changes.

I also checked Free Agent Sync on my F:\ drive, but that copy of the file, like the one on my C:\ drive, had not preserved the changes. So I had to go back and re-create all the work I’d just done. And in the process of doing so, I misspelled someone’s name, a thing I normally pride myself on not doing, and probably left a few other things out.

I’ve since gone back and set the autosave in MindManager (it’s under Tools | Options | Save) to 5 minutes. That might keep me out of trouble. But only if I get enough sleep.

If your hard drive fails and you don’t have up-to-date, functioning backups, you call in the data recovery experts. It’s kind of like sending your drive in for emergency surgery—down to the sterile environment the engineers have to operate in. Chances are, your drive is dead for good, but it might be possible to give your data a new lease on life.

On February 16th I received an invitation to interview Bay Area data recovery company DriveSavers, accompanied by a press release announcing their introduction of industry standards for data recovery. I’ve written about DriveSavers and their Museum of Bizarre Disk-asters in the past, and happily took the opportunity to interview Chief Information Security Officer Michael Hall on February 19th, 2009.

While the issue of “securing data, even during data recovery,” to quote the first message I got from Margie Schaffner at BLASTmedia, is certainly an important one, I knew that my readers would want to hear about more than just industry standards, so I put out a call on LinkedIn for questions to ask Michael. I organized the questions into four basic categories and consolidated the ones that overlapped.

Note: I have edited Michael’s responses (which I recorded) just slightly, mostly to make them more appropriate for the written form of this blog. In some cases, where he provided the answer to a later question earlier in the discussion, I have relocated what he said. I sent him a draft of this post to check for accuracy before releasing it, and a few responses have been filled in or corrected through those e-mail discussions.

Questions About DriveSavers

What differentiates you from Ontrack Data Recovery? Both companies have nearly identical taglines of being the world leader in data recovery services. (Note: when I asked the question, I expanded it to include competitors in general. When Michael answered the question, Michelle Taylor, Director of Communications at DriveSavers, edited it slightly, so the style of the response is more formal than the rest of Michael’s answers.)

The primary difference between us and any other data recovery company is our certified secure environment and unparalleled customer service. We adhere to very strict and stringent government security protocols, and we are also certified SAS 70 Type II compliant. We have the fastest standard service available and we meet those turnaround times about 99% of the time. Our actual facility has certified ISO 5, 6 & 7 cleanrooms, expert engineers and advanced technologies which enable us to maximize the success of every data recovery. We defend and protect our customers’ information from security breaches here. No one else is going to be able to hack into our network and gain access to our customers’ information. I’m not sure what Ontrack’s network infrastructure is like; I know that what differentiates us from all other data recovery companies is that we adhere to very high security standards and we have met or exceeded all those standards and we have certifications to prove it: they’re available for viewing on the website.

What determines your prices?

It’s based upon the capacity of the device itself and the turnaround time that the customer is requesting. We have a number of different options there. We have what we call the economy 5-7 day turnaround; we have a standard service, which is a 2-3 day turnaround; and we also offer priority service, which is an immediate turnaround. That means an engineer is going to be dedicated to that job from the time it hits our facility until it’s completed, to expedite the recovery process for them and get the dataset in their hands as quickly as we possibly can. So it’s basically capacity and how quick they want their data.

How are you going to make data recovery affordable for the average Joe or Jane who didn’t make a backup? (To which Sallie adds: how much slack do we want to cut people who don’t make backups?)

What we see with individuals and small businesses is that they almost never have an on-site IT person. They have a consultant who comes in and configures their backup, or their network topology, or their security, or their entire infrastructure. They put it in place, tell people how to use it, and then they walk away. They don’t come back and periodically check things. What we see more than anything is that when something’s been configured as a backup, no one’s ever taken the time to actually validate the configuration by doing a test restore to another device or checking the integrity of the data. What we see oftentimes is that whoever put the backup program in place, it worked fine for what the customer had at that point in time, but say they go from a simple database program to a SQL server, or they change their e-mail program to an Exchange server. Those files are open files; they’re constantly changing.

When they established their backup system initially, they didn’t have to worry about open files. So now they’re doing backups with open files, but they don’t have “backup an open file” option in their backup program. So they’re backing up an Exchange server and they’re backing up everything except the two open files that they need.  The same thing holds true with SQL. You can back up the entire SQL directory—except for the database. And then they have some sort of natural disaster—hardware failure or an earthquake or a power surge—and their hardware goes down. They think they’re fine until they try to do a restore, and then they realize that the whole system was configured incorrectly.

We see innumerable data recoveries that come to us because of that same scenario right there.

How do you know when the cost of recovering data is greater than the value of the data?

That’s something you have to determine yourself. The easiest way to look at it is, how long will it take you to re-create the data yourself manually? How many man-hours is that going to take? How many temps will you have to get all the paper trail that you’ve got back into electronic format—if you have a paper trail at all? How much are you willing to pay to keep your business running?

Are you hiring?

Not at the moment.

An Ounce of Prevention…

Isn’t prevention (e.g. Business Continuity Planning) a better investment than data recovery?

A Business Continuity Plan is imperative to any business, no matter how small or large. You have to have contingencies in place and have a pre-set plan: “If this happens I can do that. If that happens, I can do this.” That’s part and parcel of doing business. Is it more important to do that than pay for data recovery after the fact? Absolutely. If your Business Continuity Plan is written properly and it’s comprehensive and inclusive, you’ll probably never need to use us. Why wouldn’t you be proactive on the front end to take care of that. What we see, though, is even with Business Continuity Plans, there’s that 2% you can’t account for. Natural disasters. Simultaneous catastrophic hardware of the main device and the backup device. There are corporations that have us at the very bottom of their Business Continuity Plan. If all this does not work, here’s your last stop, and it’s a data recovery company.

What are you doing to integrate a “prevention” mode so that people can do encrypted, compressed off-site backups via the Internet (automatic of course), so that data recovery is more easily accomplished should it ever be needed?

We don’t offer offsite backup solutions for customers, but we are huge proponents of customers having that in place, and we can point people to different companies that handle that kind of program. We’re not going to offer it. Our primary focus is only on data recovery.

The Data Recovery Process

What is the most common reason for needing data recovery services?

About 80% of what we see is electromechanical failure. The reason we have the museum of Bizarre Disk-Asters is that it’s an unbelievably great visual representation of what can happen. But 98% of the time, that’s not what happens. You don’t have a fire, you don’t have a flood. A hard drive is a mechanical device. It’s not a question of whether it’s going to fail—it’s when it’s going to fail. A hard drive has a Mean Time to Failure rate attached to it; that’s the life expectancy for the device. 98% of the time either the drive dies on its own, or it’s been fried in a power surge. That will cause your drive to fail a whole lot quicker than being run over by a bus.

What kind of data is hardest to recover?

That’s a very open question. We work on all platforms. Any operating system. Any type of electronic device. Our rule of thumb is, if you can write a 1 and 0 to it, we’ll take it off of it. Some are more difficult than others, but I don’t know that I could definitively say “This is the hardest thing to recover.” Sometimes the hardest things to confirm [recovery of] are proprietary software applications that have been written specifically to a type of business or to an individual, where they’re not off-the-shelf applications that we can easily access and figure out. When we have situations like that, we try to work with the person who wrote the application or the customer to gain access to the application in order to confirm the data set for them.

What percentage of the time do you have success with recovery?

We have an overall success rate, but I’m not sure what it is.

The website says “the highest in the industry,” but doesn’t give any numbers. A later e-mail discussion with Michelle Taylor produced the following answer:

One of the most telling reasons we know we have the highest data recovery rate in the industry is that the majority of the drives we see at DriveSavers have visible signs of previous data recovery attempts. In some cases, these attempts have caused so much damage that the data is unrecoverable. But, in most cases we are able to retrieve data that others could not.

What state is the data in when you recover it? For instance, if you recover a Word document, do you get the whole thing? Paragraphs? Sentences?

Our intentions are to get the dataset back to the user in the state it was when they were using it. Sometimes that’s not physically possible. Usually that occurs when there’s damage on the device itself that renders a portion of it completely inaccessible. If the data has been physically scraped off the platter and it’s just dust in the bottom of the drive, we’re not going to get that back—no one is. Our rule of thumb is to get back the original data set in its original form.

What’s your opinion of online backup systems (like Carbonite) and how difficult is it to resurrect information if one is backed up in that manner?

Most online backup companies have step by step instructions for restoring the backup set back to the customers system. It is a good idea to test the procedure ahead of time so that you know exactly what is required on your end to complete the restoration.

And how easy is it for DriveSavers to recover data if the online backup service suffers a loss?

If an on-line backup service needs to use our services we still have the ability to recover the data. Usually they will be utilizing a raid configuration to store data. We have an enterprise division that is dedicated to performing data recovery on multi drive raid systems.

Is it easier or harder to recover data from the new solid-state disks in netbooks and laptops than from traditional hard drives?

Any time a new technology comes out, we spend a tremendous amount of R & D on it to make sure that we can recover information from solid-state devices, and we are able to do it.

Data Recovery Standards

Security standards are nice; do they map to an ISO standard?

Since this information was on both the press release and the DriveSavers website, it didn’t seem necessary to ask it again. The standards, and the certifications, fall into several categories:

• ISO 5 (Class 100) Cleanroom Certification
• Cisco Self-Defending Network Certification
• SAS-70 Type II Compliance Certification
• Encryption Training Certification
• Manufacturer authorization

There have been numerous cases of recovered data being sold or released without the owner’s consent. You have other cases of Geek Squad employees making private copies of sensitive information when they repair hardware for a customer. How do companies like DriveSavers talk to this? “Trust me” only goes so far.

All the certifications mean you don’t have to take their word for it. In fact, the page listing them is entitled “Demand Proof.” In addition, according to Michael, “We perform background checks on all our employees. They have to sign a security policy; we have everything in place to inhibit that from happening. Only certified cleared engineers have access to the customers’ information. We hire the most qualified and credible people.”

Explain the standards for the SOHO user who doesn’t understand what those certifications mean:

An ISO-certified cleanroom increases the chances of a good recovery, because we’re not introducing any kind of foreign objects to the media as we’re going through the recovery process. So you’re going to maximize your bet right off the bat. Secondarily, we’re in compliance with the international technology control audits. We have everything in place to ensure that integrity of their data is not compromised while it’s at our facility. We monitor our facility and our network 24/7. We’re certified to handle any type of encryption recoveries and we have the manufacturer authorizations to be able to work on the devices themselves.

For an individual customer or small-business user, even though it’s your individual drive, you have to bear in mind that even though it’s your personal drive, there’s a high likelihood that there’s information on that drive that you don’t want shared. How many people use an accounting program? If you’re using Quicken or QuickBooks or any program of that type, your credit information is on that program. Your bank account numbers are in that program.

If you’ve set up your computer properly, it shouldn’t be able to be hacked at your house, but if you send it off-site, who do you trust, and why? Anybody can say they have a cleanroom; can they show you the certification? Anybody can say they have a secure network; can they show you the certification?

Here’s another classic example. How do other data recovery companies handle recycling customers’ drives? How many times have you seen something in the news about “I bought this drive on eBay and it had another person’s information on it when I got it.” If a customer sends us a hard drive and it is completely physically done, it’s of no value to them, it’s out of warranty from the manufacturer, and they don’t want it anymore, and they tell us to recycle it, we’re going to physically degauss the hard drive with a Department of Defense-approved degausser to render it 100% inaccessible before we recycle it.

That’s as opposed to throwing a whole bunch of drives in a pile and taking them to a recycling center. Those drives get bought in bulk and then sold on eBay.

The security criteria and protocols that we have in place are just as important when dealing with individuals as with corporations. You hear of innumerable instances of laptops with 50,000 social security numbers getting stolen. If mine was on that laptop, I would be upset. But at the same time, my social security number is on my hard drive, and it wouldn’t matter if it was one of 50,000 or one of one. A lot of people have a file that shows their passwords, or their PIN number for their ATM machine.

Special thanks to everyone on LinkedIn who provided these great questions.