Monday, October 13, 2008

Tough Love on Backups - Part II

I started this thread yesterday.

- - - - -

The Great Fallacy of backup and disaster recovery is that your goal is to get the server back to where it was when it failed or ceased to exist.

That fallacy leads people to believe that mirrored drives, RAID arrays, and disc-to-disc backups are good enough.

99% of the time, these local disc-to-disc backups are good enough, because 99% of the time you're recoverying a file that was deleted last week or restoring a database from yesterday.

But your client is not just paying for a safety net when the pesky little stuff happens. Truth is, they can figure that 99% out by themselves without help from you: Go to Walmart. Buy a big USB drive. Plug it in. Run the software.

If you don't do anything more than that, then you shouldn't be in this business!

I know that's harsh, but if your entire backup strategy consists of something you heard about on Leo LaPorte, it's pretty hard to justify paying you for your "disaster recovery planning."

You client is paying you to step up and get their business back when that 1% crisis happens.

If you have two USB drives, and each holds four copies of the entire server, AND you are religious about switching them everyday, then you have eight data restore points. How far back does that go? How many months can you recover?

Just because you can't foresee needing to go back more than a few days doesn't mean you won't ever need to.

Backup to hard drive is easy to understand. That doesn't make it good.

Why is it a fallacy? Because the real goal is to protect the business from going out of business. That might mean producing financial records going back several years. It might mean recovering to new hardware after a disaster. It might mean all kinds of things above and beyond simple file recovery.

Being able to restore a handful of files from an onsite or internet device is NOT a disaster recovery plan.

The Most Common Disaster Recovery Scenario
Truth is, everyone will have a hard drive fail if they live long enough. That's pretty foreseeable, and reasonably recoverable.

But what's the most common true disaster? For some people, it's a water disaster (flood, hurricane, busted water pipe,etc.). For everyone else, it's a fire.

So let's assume you have a real disaster. Whatever backup system you had connected to the server in the office that burned down is gone. Hard drive? Gone. D-D-T? Gone. Tape? Gone. Mirrored Server? Gone.

You get the point.

The Most Common Absolute Truth of Disaster Recovery is that the first medium you try to restore from will fail. Doesn't matter whether it's VSS, tape, hard drive, mirror, or whatever.

So, if one USB drive is in a building that burned down, and the other one is 2,000 miles away, you can look forward to a failure of the first image you try to restore on that drive when it gets here.

Now, assuming you had four complete images per drive, you have three images that may be successful.

The entire recovery of this business now relies on you being supremely clear about what's on that disc and how you're going to get it onto the new server.

You've lost a bunch of data (fire, etc.). Now your skill level and your backup plan will determine how much more will get recovered.

If you don't really understand how the data are arranged on the drive, or you don't have the software needed to recover it, then you're just scratching around in a panic while the client's business is down.

Excuses, Excuses, Excuses

We've been in this business a long time. We've set up hundreds of systems. So far, we've never had a critical backup failure. We design, build, and support backup systems that work for hundreds of computers every day. That makes it very frustrating for us when we hear computer technicians or clients say that tape backup doesn't work.

Good backup systems work if you know what you're doing.

The proof -- The only proof that matters -- is whether you can get the client's business back up, in operation, with zero or near-zero data loss.

Having something that's "easy" is fine, but it's not your job to just do what's easy.



  1. Anonymous5:46 AM

    In your first post you said you like Backup Exec. Do you insist the client takes annual maintenance? In my experience Backup Exec needs a lot of TLC to keep it running sweetly with tape or not but especially with tape. Maybe I'm one of those that 'doesn't understand it' properly ...
    Ian B

  2. We only quote it with maintenance. The client doesn't know the difference.

    Also: Part of how we save money with managed service is that everything has to have maintenance. That way we can pick up the phone and get the problem solved.

    As for backup exec being tricky: yes, it can be. But we find that we have fewer problems when our technicians have actually read the 1" thick manual.

    Symantec software is notorious for requiring a complete uninstall/reinstall to apply updates. We're not quite there with BUE, but it's moving in that direction.

    The most important rule for success with BUE is to use their drivers and not manufacturer drivers whenever possible.

  3. Anonymous1:05 AM

    Thanks Karl. You must be doing something right to be able to bill the 20% maintenance without the client caring and also have time to get all your techs to read 1" of manual for only one of your products ... ;o)

    Ian B

  4. On price: Generally speaking, clients don't shop around unless you've trained them to do so.

    We only quote what the client really needs. We never deliver the wrong product, or an older version when a newer one has been released.

    In other words, when they order from us they get the right thing and there's no delays or hassles with returns.

    That has value.

    Eventually, they learn to just order what we tell them. They know they could save a little here or there buying from someone else, but in the long run, we're cheaper because it's the right product the first time.

    As for training techs: Daily beatings are helpful.

    Seriously: We show them all the features we use and then take opportunities to make them step up a level from time to time. They don't sit and read the whole manual. But they do learn the more advanced features.


Feedback Welcome

Please note, however, that spam will be deleted, as will abusive posts.

Disagreements welcome!