NASty Surprises: FileSlinger™ Backup Reminder 02-23-07

Those of you who have been reading for a while may remember that Seagate gave me a Maxtor Shared Storage II drive a few months back, and very excited I was about it, too. It’s still working perfectly. In fact, I haven’t turned it off once since I installed it. It does a good job making scheduled backups of my computer and my housemate’s, and I’ve had good luck using it for storing Ghost backups on when I make those (which I confess is less often than it probably should be, right now).

Not everyone has had such a good experience with the Shared Storage II, however. About ten days ago I got a message from someone who’d found the FileSlinger™ Backup Blog while searching for a solution to the hair-raising problems he was having. I’ll be quoting some of the ensuing discussion in a minute, but I’ll say now that the upshot of it is that Network Attached Storage is not right for all kinds of backups.

I cut some parts of the messages for brevity and edited here and there for typos, then ran the article past both the unhappy drive owner and the Ur-Guru to make sure I hadn’t misrepresented anything.

On 2/10/07, PT wrote:

I (optimistically, it turns out) purchased two Maxtor Shared Storage II (1Tb NAS) units, but they seem I-N-C-R-E-D-I-B-L-Y S—L—O—W, to the point of being impractical (well—actually, virtually unusable).

There’s a long story behind it; I’ve had one of them replaced THREE TIMES (yes, that’s THREE [3] times!!) because of drive failure [within a period of weeks]. I am extremely disappointed with the product (and its virtual lack of documentation, or useful software). At any rate, the point of telling you all this (which—believe me!—is the very SHORT version) is to let you know that I have, in fact, been working diligently to try to figure out how to make these work.

The problem is that it is taking WEEKS (of 24-hour-a-day file copying) to get 1Tb of data copied to the drive, or (perhaps even worse yet—don’t know how to benchmark speed, and neither does {evidently} the senior tech) from one of these MSSII drives to the other. This is seemingly orders of magnitude slower than USB2.

I’m running this from a dual-processor, fairly recent (Gateway 836GM) machine. Neither the processor nor the network usage is anywhere near strained (typically ~5%, and 0.5% [gigabit-speed network, Netgear switch], respectively).

Copying more than one file at once sometimes (not always) makes throughput even worse. The Seagate tech says something about the processor on the MSSII being slow—but jeez, this is craziness.

Somehow, this just doesn’t seem right.

It didn’t seem right to me, either, so I wrote back to say so. Then, since I had no relevant experience, I passed the question on to the Ur-Guru. He, too, was puzzled, but he had more insight than I did:

That’s not how those things are supposed to work. I’d say try the drive on a completely different system if you have a chance to do so, in order to isolate whether this is a continuous problem with the drive or related to the system. Another option is to hook the device directly to the machine and cut out the Netgear switch to see if perhaps that is causing trouble.

Config of the NIC and/or switch could also perhaps be an issue, like, maybe the switch and NIC are set to use Jumbo frames and perhaps the NAS doesn’t like that (don’t know, I’d have to look it up, I have no clue about the MSSII in that area).

Based on the description I’m seriously inclined to think it is definitely network related, not MSSII related.

PT promised to check out these suggestions and get back with us. When I saw Friday looming this week, I thought I’d ask him how he’d made out. Here’s what he said:

Initially, I reset my network switch and everything went very fast for a short while. I think this was just coincidence, though. Later, I spent an HOUR on the phone with NetGear to see if we could isolate a network problem. Bottom line is: for single-file-at-a-time huge files (we tried >300Mb), it’s pretty dang fast. But for realistic use—in my case, tens of thousands of small [ca. 2k] files—the MSSII just dead-dog slow. Smaller drives could be slow with less consequence; but huge drives have GOT to be fast!

When he read this, a light bulb went on for the Ur-Guru. (Which is good, because it didn’t illuminate anything for me.)

Ohhhhhh….

Yes, that would explain a lot. Tons of tiny files are murder on any network load and disk performance. But even if you widen the network to 10Gbit, it’d still perform quite badly.

Lots of my source code trees are riddled with tons of small files and the sync/copy over the network got so slow as the amount increased that I wrote a script that would first zip all the trees and then copy the zip to the other side (zip in this case since it is faster in packing and the size of the resulting file is less of an issue). I don’t remember any statistics I ran but… 400,000 small 4K-50K files were zipped into an approx. 500MB file and that goes over the network in a few seconds. Copying those 400,000 small files, however, takes about 22 minutes. Huge difference there. And we’re talking very fast RAID systems in my case, not a drive with a slow processor like the MSSII.

The tons of small files explains the problem he’s seeing quite well. The ethernet packet wrapping overhead for that is more than the file data combined so it really starts to slow things down, and unless you have some serious heavy hitter RAID on the other end of the network, it’ll be slow as hell. And even with heavy hitter RAID on the other end, copying 1GB in 100.000 small files is easily 100 times slower than copying a 1GB file as a single file (very rough number here, non-scientific). It’s also murder on disks in terms of seek time and fragmentation and… just about everything. 🙂

So there you have it: network backups are not good for huge numbers of small files. I passed the bad news on to PT, who said:

I guess my expectations were just too high—and the disappointment (other than the multiple returns of defective drives!) was primarily due to my lack of experience with large amounts of data being transmitted over networks. (I’ve had several PCs networked for years now, but I guess I’ve only transferred large amounts of data via USB2/Firewire-type drives—and, until recently, I only had a 10/100 system anyway (so the network would max out before the drives did anyway for transfers of large files.)

The idea of zipping the files pre- (and post-) transfer makes a lot of sense. Not sure that I’ll be able to be as fancy as writing a routine to do it for me, but I’ll definitely use the (manual) technique to circumvent some of the problems I’ve had.

I pointed out that even without a custom routine it might be possible to get the zipping automated, as there are various backup programs that will zip the files before copying them to their destination.

As for the multiple defective drives…I wonder whether they burned out trying to handle those many files? It seems like resoundingly bad luck to have. Maybe I got the only reliable drive in the batch. I have to admit I’d hesitate to buy one of these after hearing that story, even though I’ve had no problems whatsoever with my own Shared Storage II.

In any case, it’s clear that size matters when it comes to network backups. The Ur-Guru thought everyone knew that, but I didn’t, and neither did PT. And I’m betting we’re not the only ones.