Last week I had a good talk with some people from KineticD, and I’m still exploring their product, which you’ll hear about next week. This week, however, we’re going on a little detour, courtesy of an experience with a client yesterday.

I was actually at the client’s house to work on other things (like adding Google Analytics to their Facebook pages), but while I was there, they asked me to check on backups, particularly for the Windows system. The backup drive they’d been using was wonky and they hadn’t replaced it yet.

What’s more, they’d been using Mozy to back up their Macs, but their Mac guy had just told them they couldn’t rely on it because of problems with restoring data. (To the Mozy people reading this, I have no direct experience with the problem and can’t tell you any more.)

I did suggest that the client check out KineticD for online backup, since it’s targeted at small businesses with multiple computers, but the $2/GB price structure would mean considerably more than they were paying Mozy, so they weren’t too enthralled with that idea.

So I suggested a NAS drive that they could back all the machines up to, Mac and PC, and printed out some datasheets on a couple of Buffalo’s models, since I’m familiar with their products thanks to my BFF Jay Pechek. I also believe that it’s good to have both local and online backups.

A little further conversation revealed that photos made up the great bulk of the 25+ GB the client was currently backing up with Mozy (and this was just the home office, mind you; the actual business has completely separate systems).

In that case, I said, why not use SmugMug to back up the photos?

SmugMug seems to be the best photo sharing service you’ve never heard of. Certainly my client hadn’t heard of it. But I keep hearing of it in more places. Professional photographers like it because you can sell photos directly from the site. And when I went over to check it out in more detail, I discovered that they actually offer a special backup service for your high-resolution RAW, TIFF, and PSD files, as well as your video. (Storage is provided by Amazon Web Services.)

Unlike my mother and the Ur-Guru, I’m not a serious photographer. I’m still learning how to use my camera, and I don’t shoot RAW. My hard drive is not filled with photos—though I confess that owning cats may change that. But anyone with children or grandchildren has photos to store, and the combination of a SmugMug account with the SmugVault service may well be the best deal available for both backup and sharing.

A basic SmugMug account is $39.95/year and lets you store unlimited .jpg, .gif, and .png files, so I could actually keep all of my cat photos up there at full resolution (about 4 MB apiece) without paying anything extra. And I think my client’s camera also shoots in JPG rather than RAW, so her 20 GB or so of photos would be cheaper to store on SmugMug than on Mozy, and she could display them, too.

For those of us who have amateur cameras, a Flickr Pro account might be sufficient for photo storage and backup. You get unlimited uploads of JPG files for $24.95/year. But the pros need to care for their RAW images, and the designers for their PSD files, and that’s where SmugVault comes in.

You activate SmugVault in your SmugMug control panel under “settings.”

When you click the “Get one!” link next to the SmugVault entry, you find yourself here:

So you need to hop on over to Amazon and sign up. You don’t need to have your own AWS account for this, just to have an Amazon account with a card on file so they can charge you your initial pro-rated payment. As usual with AWS, the billing is a bit confusing.

 

So it’s 22 cents per gigabyte for storage, but there are also upload and download (bandwidth) charges, and a minimum charge of $1/month. Compared to Mozy Pro’s 50 cents/gigabyte plus a monthly license fee, or KineticD’s $2/GB flat fee, that’s still pretty cheap, as long as you’re just uploading the files and leaving them there. As my podcaster friends have discovered, bandwidth charges are what will kill you with Amazon S3 and their other web services.

It’s easy to use the SmugVault interface. You just click the green “add files” button and create a gallery for the vault by selecting and uploading files from your drive. I selected 3 PSD files, one of them 41 MB in size, and they went up pretty quickly.

Once the files have been uploaded, you see more information in your main vault screen.

When you browse the archives, you can delete the files, hover over them for information, or click on them to be taken to their gallery, where you can download them.

Since I made this gallery private, people browsing around my site can’t see those files. You can also password-protect galleries, which is probably a good idea for your backups, since otherwise they won’t be very secure.

There are only two drawbacks. The first holds true for any online service: doing your first upload is going to be very time-consuming if you have a lot of photos. The second is that SmugMug doesn’t automatically upload your photos, so you have to remember to put them in your SmugVault. For some people, that’s a dealbreaker, but I’d still recommend it to anyone whose data consists primarily of photographs.

Disclosure: this post contains Amazon affiliate links. I bet you’re smart enough to figure out which ones they are.

As we were driving back from Palm Desert to Pasadena earlier this month, the Ur-Guru pointed out a problem with the current craze for cloud computing. No, not the problem I’ve talked about before—slow broadband or no broadband at all making it impractical if not impossible to store or transfer large files online—though that problem is a long way from being solved. It’s the reason the “prosumer” photographers with their 10-megapixel cameras are all lining up to buy the DroboPro. The Ur-Guru will be going home with about 70 GB of photos from this trip alone. (The RAW files are sitting on my Maxtor Shared Storage II drive until he tells me it’s safe to delete them; he’ll have more than a few backup routines to run when he’s back in his world-famous home office.) You try uploading that over your home cable or DSL line. I dare you.

But in spite of the bandwidth bottleneck in getting our data into the cloud, more and more of us are either storing our documents and photos online or creating them there in the first place. It’s not just the online backup services, but the photo sharing sites (the Ur-Guru likes Flickr, whereas my mother prefers Picasa), the social networks, the blogs, and the file sharing and collaboration tools like Google Docs, Box.net, and Huddle Workspaces. And then there are all those Kindle books hanging out on Amazon’s Whispernet servers waiting to be downloaded—and maybe sucked back up again when Amazon finds out it’s made a mistake. (That last episode points out another problem with online storage: if your only copy of the data lives on someone else’s server, is it really yours?)

But there’s a down side all this online storage, as connected as it makes us all, and as useful as it can be to get our critical data off-site in case of theft, fire, or natural disaster. The data you back up online, the photos you post to Flickr, the PDF of your resume that you just dropped into Box.net so it would show up on your LinkedIn profile, even the website you can’t conduct your business without—all of those live on servers in data centers. This is the good news and the bad news. Your website and mail server have to be up and running 24/7, of course. That’s just as true if you run them out of your home office as if you hire a web hosting company.

But what about the rest of it? Sure, some of it is there because we want anyone to be able to see it at any time, whether we’re around to show it off or not. That means it has to be on a computer that’s using power. But what about the data that really is just backup? Or even reference or archival material? My colleague Eve Abbott says that 80% of paper that gets filed is never referred to again. (And after a year has passed, it goes up to 90%.) I’m pretty sure that statistic holds true for digital documents, too. It’s easier to search through our electronic detritus, but if you go back and look at all the articles and white papers you’ve saved over the years, I bet you’ll be amazed at how many of them there are—and how few you can remember anything about.

That means that not only don’t we need immediate access to a lot of what’s going onto those 24/7 RAID-whatever servers in the high-security data centers with the enormously high power bills, we probably don’t need it at all. Most backup software is configured for lazy people: it backs up everything, automatically, so you don’t have to think about or do anything. Anyone who has experienced data loss is likely to err on the side of caution anyway, so unless there are severe limitations on that online storage space and harsh financial penalties for overrunning the quota, very few people are going to sit down and triage the contents of their hard drives before sending their data up into the cloud—never mind afterward.

The Ur-Guru was all set to go look up statistics for the cost of operating a data center so he could figure out the math of exactly how much it really cost (and how much power it used) to store everything online. Me, I started avoiding math as soon as I left high school—once was enough to take calculus. Besides, I don’t think we need the actual figures in order to reach some sobering conclusions.

Data stored on tape, CD, DVD, or a disconnected hard drive requires no power at all. The storage media aren’t particularly green—and never have been—but they aren’t using up electricity. When I shut down my computer, my USB drives power off. The Rebit that I use to back up my netbook spends most of its time unplugged on the computer stand, because the netbook itself spends most of its time there, except when I’m traveling or out at a business meeting. There’s no need to have it running all the time. Likewise the four Western Digital drives that the Ur-Guru brings with him on his visits wait patiently, unpowered, for him to return with a full SD card and offload new photos.

Because these external drives are not spinning all the time, we can expect them to live much longer than the drives in the RAID boxes in data centers. Those are designed to be easy to replace without interrupting the machine’s function or causing any data loss, and it’s a good thing, too: constant use wears them out. They have to be replaced by the boxload. That means more toxic electronic components to dispose of than if you were backing up at home.

Is there a way to get the benefits of online backup and storage without increasing the need for power at a time when we’re all trying to save energy? Possibly. In most cases, it’s not really necessary for backup to be continuous or for restoration to be instantaneous. In theory, then, data could be stored on removable drives and handled in much the same way the automated tape libraries in large corporation handle tapes. Retrieving that data would take a short while, since the appropriate disk would have to be found and inserted into a drive bay, but restoration would still be faster than with tape. But to utilize this kind of storage, service providers would have to invest in entirely different facilities and infrastructure. FalconStor could see a real boom in its sales of Virtual Tape Libraries if the world of online storage decides to go green.

I don’t think you’ll see everyone and his brother rushing into that, however. Setting up a VTL requires serious capital outlay, much more than renting space in a data center. That cost would inevitably get passed on to the user. It would take some serious dedication to reducing your carbon footprint to choose an expensive service that doesn’t use power just to store your data over a low-cost—or even free—service that does. Especially when you could be equally “green” by just making your backups at home and storing the drives in your safe-deposit box.

The Ur-Guru thinks the real solution probably involves a storage medium that hasn’t been invented yet, but says that all conversions to alternative power sources and more efficient systems require an outlay of capital. If we take that as given and are willing to invest in some R & D, then it would probably be possible to reduce the world’s collective power bill for data substantially even before we invent that new storage medium:

My idea for a VTL was not about using disks as tapes and having robotic arms load them, but rather using entire arrays of disks and having a robotic arm move entire sections in and out while some Drobo-like software makes duplicate copies when needed, caches data for users who are using restore more often, or use some statistic saying which user might be restoring in the next x amount of time, etc. It’d be quite a cool thing to work out and make work but with logic applied you could still design it.

What I want to take out of the equation is “physical storage units allocated to x users” and turn the data into a dynamic movable chunk so the data for one or more users can “move around” in the storage units that make up an entire grid, offloading any disks that contain data that is hardly ever accessed or updated and basically powering them off.

Maybe instead of robotic loader arms some kind of electronic solution might also work where entire arrays can be powered down or run on low power, only to be powered up when needed (to move “offloaded” data to “cache”).

That sounds like just the sort of mathematically complicated problem that the Ur-Guru would enjoy and that I went into the humanities to get away from. But it’s an intriguing idea. Any enterprising entrepreneurs out there interested in making it work?

Last week I talked about backups for two different Web 2.0 services: del.icio.us and LinkedIn. I chose those two because they’re the ones I use the most often.

This week I’m going to start by talking about Flickr, a popular photo-sharing service that doubles as a social network. I don’t post photos to Flickr myself, but the Ur-Guru does. (Yes, lots of them are pictures of me. What did you expect?)

I first noticed the existence of Flickr backup tools a couple of years ago. I had a bit of trouble understanding why you would need them. After all, the photos can’t get to Flickr unless you first have them on your computer (or a camera connected to the Internet). Surely if they’re worth sharing with the world, you’re going to save them on your hard drive or a CD, and they’ll get backed up with the rest of your data.

On the other hand, if something happened and you needed to re-upload your photos, remembering which ones you’d had there and which tags you’d used to identify each image could get to be a real challenge. That’s why there are programs like Flickredit, a Java-based program for editing, tagging, uploading, and backing up your photos and their associated metadata (copyright info, title, description, tags). If you’ve put hours into creating this metadata for your Flickr photos, I’d recommend checking it out.

Photobucket, another popular photo-sharing site, lets its pro users back up via FTP download. Regular users can order backup CDs or DVDs from the Photobucket Store.

Enough people who belong to multiple social networking sites have expressed a desire to import their profiles without typing everything over again that there’s now a Data Portability Project. There’s a long list of the benefits of data portability over on the Use Cases page. They look particularly useful for people who use a lot of job-search or social networking sites.

Interestingly, however, while the list mentions transferring, aggregating, and exporting contacts and other data, it doesn’t specifically address backup. If your data is that portable, however, it should be possible to port it onto your hard drive and back it up. And, of course, having the same information duplicated across several sites can also act as a backup, though if you delete something by accident, the deletion might propagate across all the sites. Which leads me to wonder whether there’s an “Undo Portability Project” in the making. (Repeat after me: synchronization is not backup.)

It will take a while before the Data Portability Project produces useful results, so remember to check out the possibilities for backing up your profile information and other data before you sign up. If you need to keep your profile info in a Word doc in order to keep from having to re-type it, then that’s probably what you should do. And if you can get new messages, photos, and the like from your friends as an RSS feed, remember to subscribe to your own feed in order to keep a copy.

In most cases, anything you post on these sites goes up there at your own risk, and it may well become the property of the social networking site once you put it there.

If you’re an avid user of MySpace, Facebook, or other social networks, why not share your method for backing up your profile and other data—or your reason for not bothering.

This is the post I was planning to write last week, your annual reminder that you need to archive your data at the end of each year. (If your fiscal year is different from the calendar year, you should create these archives then.)

I’ve written about year-end backups on several occasions before. Because (as I pointed out in December 2005) these aren’t really backups, I’m going to stop talking about “year-end backups” and start talking about “annual archives.” At the end of 2004, and again in December 2006, I described the kinds of data that goes into one of these archives. My focus up to this point has been on archiving your data for tax purposes, so those posts address primarily financial and business data.

The need to back up–and archive–all supporting documents relating to your business income and expenses has not gone away. I’ve just made 4 DVDs to add to the tax box. There’s one for each of my business personas (the FileSlinger, the Author-izer, and the Podcast Asylum) and one with the new promo photos the Ur-Guru took this year. (You can see some of them on Flickr.) The most time-consuming thing about making them was isolating 2007 data. In some cases I had already done this, but I haven’t been completely consistent.

Once it was done, I removed all finished projects from 2007 from my C drive to make room for 2008 projects. I’m not that pressed for storage space on my machine, but it’s annoying to have to look through folders for clients I’m finished with, or previous versions of documents I’m working on, when I want to get to my current work. So I use making the annual archive as an opportunity to tidy up my hard drive.

That’s all business as usual. But more and more people are using computers to do more and more things. You might well want to make an annual archive even if you don’t have to worry about tax audits. Here are a few examples of data that it pays to be able to save each year even if you’re a student, a stay-at-home parent, or retired.

Coursework and Student Records

You might want to go back and use that essay or project for something else one day, and chances are you’re going to remember it by what class you had to do it for. You might need your grades and transcripts in order to pursue an advanced degree or get a job. And you might need to provide someone with evidence that you really did take such-and-such a class. But you’re probably not going to need it all on your main hard drive, and you may not even need it on your main backup drive. Burn it to a CD or DVD, label it with the year, and archive it. (Preferably off site.)

Some class projects take up more space than others. If you’re studying video, you’ll probably need more than one DVD per year. You might consider using an external drive to store your annual archives. Toshiba has just announced 1.8-inch hard drives with capacities up to 120 GB. I wonder how long it will take before someone comes out with a tray, rack, or box designed to store them safely.

And Speaking of Photos and Video…

Film cameras have all but disappeared. Digital cameras mean we take more pictures, because we don’t have to worry about running out of film, and if they don’t come out, you can always delete them. How are your photos organized? In some cases, it might make sense to sort them by subject, but if you archive each year’s photos into a folder with the date, you’ll have a much easier time when it comes to showing your grandchildren what you looked like in high school, or embarrassing your child by showing his baby pictures to his first girlfriend.

Also, if you take a lot of photos, your hard drive starts to fill up. Keep the best ones on your hard drive and store the rest on DVDs or an external drive. Then you won’t have to look through 1000 photos to find the two you actually wanted to print.

If you use a photo-sharing service like Flickr or Photobucket, those can act as backups of the pictures you upload, as well as helping you organize them and letting you show them to other people. There are even programs to back up your Flickr photos.

E-mail and Contacts

Even if all your correspondence is personal, you might want to save it–and to save the e-mail or postal addresses of the friends and family members you write to. If you make a copy for each year, it will save you a lot of time and trouble when you decide to write your memoir or family history. Your calendar information can be useful there, too. Even if you never write a memoir, your children or grandchildren might want to know what your life was like back when. If you’re like me, you forget a lot of the details.

Your Blog

Most of the blogs I read are business blogs, but many people do use blogs to write personal journals. If you think you’re going to want to read what you wrote on LiveJournal or MySpace or Blogger, better make a copy of what you’ve posted. It’s good to back these things up regularly, but even if that’s too much trouble, save your blog onto a CD or DVD at least once a year. (Most blogs don’t take up a lot of storage space.) If you want more details about backing up your blog, see my previous posts on the subject or do a Google search for “backup <name of blogging platform>.” There are even tools like Blurb BookSmart to let you back up your blog in hardcopy format by turning it into a book, though they don’t work with all blogs.

That should be enough to keep you busy for a while. Remember to store your annual archives somewhere other than the place you keep your working files: in another room, at a friend’s house, in your safe deposit box.