Thoughts on backups

Backups have weighed heavily on my mind lately. When you have 125 servers to tend to at work, chances are one of them is going to fail eventually. Really what seems to happen is they fail in bunches.

One of my clients has a problem. He’s out of capacity. And that’s gotten me thinking about backups in general.You see, my client’s golf buddies are telling him nobody backs up to tape anymore. Backing up to disk is the hot thing now. Here’s the theory. Your network is fast, right? Why make it wait on the tape drive? Back up all your servers to disk instead, and they can all back up at once, and hours-long backups take minutes instead, and restores take seconds. And no more paying $3,000 for tape drives and $6,000 for a rotation of tapes for it!

Now here’s the problem. A CIO hears "disk" and he thinks of that 400-gigabyte IDE drive he saw in the Sunday paper sales ad for $129 with a $60 mail-in rebate. (It wasn’t really quite that big, and it wasn’t really quite that cheap, but these things are always better on Monday morning than they were the day before.)

No enterprise bases something as important as backups on a single consumer-grade IDE disk. For one thing, it won’t be fast enough. For another, they’re not designed to be used that heavily, that frequently. An enterprise could get away with something like HP’s $1200 entry-level NAS boxes, which use cheap IDE drives but in a RAID configuration, so that when one of those cheap disks fails, it can limp along for the rest of the night until you swap out the failed drive. The chances of one drive failing are small but too large for comfort; the chances of two drives failing at once are only slightly better than Ronald Reagan winning the Republican primary this year. With Abraham Lincoln as his running mate.

One can set up some very nice backups on a Gigabit Ethernet setup. Since Gigabit’s theoretical bandwidth is about 3 time that of Ultra320 SCSI’s theoretical bandwidth, you can back up three servers at once at full speed. Drop in a second NIC, and you can back up six. In reality, the disks in the NAS box can’t come close to keeping up with that rate, but the disk can still back up everything much faster than tape will. Even a lightning-fast state of the art 200/400 GB LTO drive.

Frankly, with such a setup it becomes practical to back up your most important servers over the lunch hour, to avoid losing half a day’s work.

But you don’t get it for $129.

And in reality, no enterprise in its right mind is throwing out tapes either. If they back up to disk, they spool that backup to tapes the next day, so they can store the tapes offsite for archival and/or disaster recovery purposes.

How important is this? I remember about a year ago getting a request for a file that was changed in the middle of a week, and the person wanted that copy from the middle of the week, not from our Friday backups that are archived longer. Even with a tape rotation of 40 tapes, I couldn’t get the file. The tape had been overwritten in the rotation a day or two before.

While rare, these instances can happen. A 40-tape rotation might not be enough to avoid it. Let alone just a couple hundred gigs of disk space.

But what about home?

Consumer tape drives had a terrible reputation, and based on my experience it was largely deserved. The drives had a terrible tendency to break down, and the failure rate of the tapes themselves was high too. The lack of comfort with enterprise-grade tape that I see in my day-to-day work may stem from this.

The last time I was in a consumer electronics store, I don’t think I saw any tape drives.

I suspect most people back their stuff up onto optical disks of some sort, be it CD-R or RW, or some form of writable DVD. The disks are cheap, drives that can read them are plentiful, and if floppies are any indication, the formats ought to still be readable in 20 years. My main concern is that the discs themselves may not be. Cheap optical discs tend to deterriorate rapidly. Even name-brand discs sometimes do. We’ve had great luck with TDK discs ever since Kodak took theirs off the market, but all we can say is that over the course of three years, we haven’t had one fail.

The last time my church’s IT guy called asking about backups, we happened upon a solution: a rotation of USB hard drives. Plug it in, back it up, and take the drive home with you. It’s cheap and elegant. Worried about the reliability of the drives? That’s why you use several. Three’s the minimum; five drives would be better. Use a different drive every day.

It’ll work, and it’s pretty affordable. And since the drives can be opened up and replaced with internal drives, it has the potential for cheap future upgrades.

How about the reliability of hard drives? Well, I have a box full of perfectly readable 120-meg drives in my basement. They date from 1991-1993, for the most part. I bought them off eBay in the mid 1990s, intending to put them in computers I would donate to churches. The computers never materialized, so the drives sat. I fire one up every once in a while out of curiosity. The copies of DOS, Windows 3.1, and the DOS Netware client that were on them when I got them are still there.

Some technology writers have observed that modern IDE hard drives seem to have a use-by date; they just seem to have a tendency to drop dead if they sit unused for too long. I see this tendency in a lot of devices that use inexpensive electric motors. Starting them up every once in a while and giving them a workout to keep the lubricants flowing and keep them from turning glue-like seems to be the best way to keep them working.

At this stage, I’m less worried about the long-term viability of hard drives than I am about optical discs. Ask me again in 20 years which one was the better choice, and I’ll be able to answer the question a lot better.

If you’re stuck using optical discs, the best advice I can give is to use a brand of media with a good reputation, such as TDK, make multiple copies, and store them in a cool, dark, dry place. The multiple copies should preferably be stored in different cool, dark, dry places. Light seems to break down optical discs, and cooler temperatures as a general rule slow down chemical reactions. Dryness prevents chemical reactions with water and whatever the water might manage to pick up.

If you found this post informative or helpful, please share it!