Friday, February 12, 2010

A Few Backup Rules To Save Your Butt

Here is an anonymized version of a story that was posted on one of the Yahoo groups last year:

About six months ago I setup a network for a business customer. While I was there she asked me to check that her external USB drive was backing up her data. It had been set to the original defaults which backed up her "My Documents" folder and email.

I asked her what programs she used, and specifically what data she wanted backed up. I selected the folder that contained the data for the business program she told me she used.

I remember I also talked to her about the benefits of using a drive imaging program like Acronis and also using off site backup such as Carbonite.

This week her hard drive crashed. I reinstalled Windows, etc. on a new drive and restored her data from the backup.

She is now quite upset that she does not have any data from her primary Line of Business Application. This program contained ALL of her business contacts and appointments.

This is the first time I've heard of this program and she did not mention it when she asked me to check her backup program.

In any event, the customer believes it's my fault that her data was not backed up.

I feel bad that she's unhappy with me and I'm a little worried that she might even sue me.

To Start With: Never think this can't happen to you. This is why we don't do break/fix any more. It's also why we are incorporated and have E&O insurance.

But even in a managed environment there are clients who don't communicate well. Many have unreasonable expectations. And some even go out of their way to look like they know what they're doing when you talk to them. This gives you a false sense that they will take some responsibility for their own systems.

Here are a few things to know about backups. Some of them could be put into a nice informational flyer for your client (not all, obviously).

First, NO backup system can be considered to be "working" until you restore from it. If you can't restore, you don't have a backup.

Second, NO backup system will automatically work forever. You need to verify that it's still working from time to time. You do that by performing a restore.

Third, The computer tech needs to determine what needs to be backed up -- not the client. The client wants as much as she can get for her money. And when stuff goes wrong, she's going to want to get everything back as fast as possible.

There's only one perfectly safe way to make sure that you have her data no matter where she decides to put it: Backup or image the entire system. Everything Every Night.

Backups Fail

There is no auto-backup system that will work forever. Get over it.

We've said consistently for fifteen years: half of the new clients who come to us have no working backup. Some know it, most don't. People dutifully switch tapes (or disc or pay for off site backup) in the belief that they're protected -- but they're not. In most cases, the automated backup failed at some point in the past and never restarted.

This can happen because of an update to the hardware drivers, update to software, or even a password change. The backup stopped working and no one ever did a test restore to verify that it was still working. I literally know people who switched tapes for YEARS and didn't know that the service wasn't running because a password had been changed a couple of years before.

Let me repeat rule #1:
- NO backup system can be considered to be "working" until you restore from it. If you can't restore, you don't have a backup. It's like a tree in the woods. If you back up the data but can't recover it, have you really backed it up?

How do you know when your backup has failed? Well, you either get a report about that or you check it yourself by doing a restore.

We are often asked what to back up. The answer is easy: Anything of any value. If you want it, back it up.

Assume this building ceases to exist today. Fire, flood, hazmat spill, roof collapse. Whatever. You can't get to your machines and you want to stay in business. What do you do? How much data do you NOT want to recover?

The next question is "How often should we back up?" The answer to that is, how much data do you want to re-create. If you are happy with losing one day's data, backup every other day. If you're happy losing a week's work, back up once a week. If you're happy losing a month's work, back up once a month.

Backups Cost Money

Most people in this discussion will say they don't want to lose anything. Some will even get into discussions about zero operational downtime and total system redundancy. That's all cool until they see the price tag. Everyone knows that moving from 99.99% up time to 99.999% up time is expensive. And every nine costs a lot more than the last.

99.99% up time is 53 minutes of downtime per year. That's one really fast reboot per quarter.

99.999% up time means five minutes of downtime per year.

The different in cost is dramatic. And 99.999% of small businesses will say that they can't afford a system with that kind of up time. Cool. No shame in that.

But the calculation of reliable backups is very similar. You want a cheap backup? There are services offering unlimited skydrive space for $5/month. I don't know where your data is, who has access to it, how fast you can retrieve it, or how you even test that for disaster recovery.

The average $1 Million business should be willing to invest in a $1,000 or $2,000 backup system. Yeah, I know that's a lot of money. What's the alternative?

I had a client whine to me about the cost of Microsoft Office the other day. Well, *I* didn't set those prices. I can't make Office cheaper. You can hack together some illegal solution you bought off ebay or got from a friend. I can't be involved in that or know about it.

Similarly with backups: Do you want a fake backup solution that won't work when you need it but saves you $1,000 today . . . or do you want to stay in business when your system fails?

It costs money to be in business. It costs more money for half-baked, unproven, untested backup systems than for proven, known technology that is tested on a regular basis. Sorry. I didn't make the real world. I just try to spend time there.

. . .

In the end, I would tell you that NO Backup System is complete without regular maintenance and testing. No automated system works forever. And you do your clients a disservice by letting them believe they are safe when they are not.

If a client hires you to create a backup system and something goes wrong some day (which it will), they have every right to come to you and ask for 100% of their data back. Your only defense for failing them is that 1) You told them this would happen, and 2) You offered to maintain their systems so you could always restore after the disaster hits.

If they turn you down, then it's all on them. If they accept, you sign a managed service agreement.


Now Available:
Introduction to Zero Downtime Migrations
Seminar on MP3 Download


  1. Love your story at the beginning--so common.

    We love the Zenith BDR for all of those reasons. 1. It's being checked 24/7, 2. It's an image backup so everything is "in there", 3. It's storing data (optionally) offsite with a recovery plan, and 4. We quarterly virtualization test it ourselves to be sure it will work when needed.

    How do you get your Acronis backups offsite automatically? And how do you monitor their operation and backup integrity?

    Thanks again for a great article.



Feedback Welcome

Please note, however, that spam will be deleted, as will abusive posts.

Disagreements welcome!