Sunday, October 12, 2008

Tough Love on Backups - Part I

This is one of those "If all I cared about was making sales, I'd just shut up" postings.

I've had a lot of feedback from people about my little $5 white paper on backups.

They agree with everything I say EXCEPT the conclusion that tape is the only medium that fills all the requirements for a true backup and disaster recovery system.

I just spent a weekend at SMB Nation. Without naming names, let me just say that the vendor hall was filled with people who will take your money, give you peace of mind, and leave you hanging when your server goes up in flames.

Why We Do Backups

99% of the time, backups are for mundane file recovery, to create a legal point-in-time snapshot, to create a financial point-in-time snapshot, and to create an archive. Hard drives that are constantly recycled don't do most of these.

The other 1% of the time backups are vital to a true disaster recovery.

In a true, major disaster, you may not be able to get to your office. Consider the floods in the Midwest, hurricane Katrina, the five-day electrical outages from last year. Or the folks in Ohio who just went through two weeks of electrical outage in the aftermath of Hurricane Ike. And so forth.

It is quite realistic that you could be kicked out of your office and need to relocate. Yes, the chances are slim, but it's your job to take responsibility for your business, and your clients' businesses.

On more of a day-to-day basis, "stuff" happens. Someone deletes an important file. The database becomes corrupt. You discover on Thursday that you saved the wrong version of a spreadsheet last Monday. Or you discover in January that the former bookkeeper nuked a database last July.

Sometimes we can restore from the local Volume Shadow Copy. You might have a local backup archive. It is vitally important that you have multiple point-in-time backups.

One reason our company has never failed to recover data from a system we managed is that we do backups the right way and we have multiple restore points.

Here's a hard lesson that won't make me any friends: Most of the technicians we run across cannot figure out how to create a backup strategy that works.

Most technicians don't have a workable strategy for backups.

Most technicians don't understand tape, so they don't recommend it.

Most technicians don't understand SCSI, so they don't sell backups that rely on it.

Most technicians have never played with iSCSI. In fact they don't even know what it is.

Corollary: Most SMB technicians settle for a backup system that sounds simple because they can't figure out tape, can't figure out SCSI, and are willing to settle for "good enough."


Good Enough Isn't


Internet-based backups are wonderful as a gap-filler for specific circumstances. But it is irresponsible to rely on them as your only backup solution, your primary backup solution, or your disaster recovery plan.

Please, please, please talk to your clients. Do not assume that you know how much downtime they can stomach unless you've asked them.

For some companies, a day of downtime hurts, but they can stay in business. For others, an hour is unacceptable.

As we've moved all the critical systems (email, databases, sharepoint, file storage) onto a single server, we have dramtically increased the need to get that server back online ASAP when something goes wrong.

Any system that involves shipping a hard drive across the country is unacceptable as a disaster recovery option.

Period.

And USB drives? Again, wonderful as a gap-filler for specific circumstances. But it is irresponsible to rely on them as your only backup solution, your primary backup solution, or your disaster recovery plan.

Maybe I've just spent too much time fixing other peoples' messes. But disc-to-disc backup is only good for disaster recovery purposes when it is part of a disc-to-disc-to-tape backup strategy.

The Bottom Line

Local imaging, CDP, USB, and even onsite tape are all part of a strategy to have multiple restore points available. They each play a role.

But don't forget that we do backups for lots of reasons besides restoring a file or two.

Learn your job.

Create and test disaster recovery plans that assume you won't have access to your original office or equipment.

:-)

10 comments:

  1. Anonymous12:11 PM

    what backup software do you use?

    ReplyDelete
  2. Backup software depends on the client. Our preferred product for disc-disc, tape, and disc-disc-tape has been Backup Exec.

    CA/Brightstore is also a top quality product. Once in awhile we have a client using that.

    For other components in a backup strategy, we use Kaseya/Acronis, and Sonicwall CDP/Acronis.

    ReplyDelete
  3. Anonymous1:48 PM

    We use backup exec too. I used ultrabac for a while but too many issues. I find that more and more backup software is excluding the option to backup to tape.

    We use tape extensively as most of our clients use very large interbase databases. The thought of backing up 30gig databases each night online is unacceptable ;) Clients also demand the ability to take backups offsite - and from past experience USB drives are just too fragile as clients can be very rough with them.

    I have kaseya but haven't tried the backup option yet.

    ReplyDelete
  4. Good one Karl - appreciate you fostering this important community dialog on backups.

    ReplyDelete
  5. Karl,

    I'm a die-hard tape person too. All of our clients used tape until a year ago and then we started to move to disk based. I was unwilling to do this until I discovered high-rely hardware. Basically we have moved from swapping tapes to swapping drives made for the purpose.

    For bigger clients though we've found that they are only backing up data, have no disaster recovery solution and are doing a poor job on backup in general. For these we're planning to move to DPM, disk-disk-tape and introducing for the first time bare metal restore.

    All of this to say, it's not just small biz consultants with poor backup skills. It is rampant through the industry at all levels.

    ReplyDelete
  6. Good point.

    There are now disc cartridges reliable enough, and cheap enough to compete with tape for the offsite component.

    One of the important things to consider is quick access to an identical tape/disc system when the building's under water.

    ReplyDelete
  7. Anonymous7:55 AM

    Karl - 2 questions

    What do you think of SBS's native backup app vs. backup exec for SBS. You say you are using BE. For SBS though?

    and is this series of blogs duplicating / adding too / something totally different from the white paper? (like your MS in a monht series, you blogged then compiled it into a sellable product).

    thank you for all that you do for the community!

    ReplyDelete
  8. Good questions.

    Last one first: These posts are supplemental to the white paper. The white paper is a pretty detailed discussion of backup strategies, the reasons for backups, why they fail, etc.

    I don't think there's much overlap. If you buy it and think otherwise, I'll give you a refund.

    As for native backup on SBS: Ugh.

    A basic tool to keep in your toolkit. But it's strength is doing a complete backup. It's great weakness today is that your most common restore need is either difficult or impossible.

    Backup Exec excels at allowing quick and easy restore of one tiny email in one folder for one person. BUE has a lot of very cool features (tape sets, etc.) that make it extremely flexible and powerful.

    If a client can't affort $600 every three years for backup software, they're not really the clients we're going after.

    MS Backup for 2008 does not support tape. This is true inside SBS 2008 as well. You could set up a Home Server and use that to backup your SBS network, but overall, BUE is cheaper.

    I could be wrong, but I believe we have no clients using MS Backup.

    ReplyDelete
  9. Lucas Drege4:28 PM

    D2D2T? Thats taking two steps forward, then taking one back. Why on earth would you possibly want data residing on a medium with the highest propensity for failure?

    ReplyDelete
  10. Well, Lucas, the truth is that there is no more reliable medium than tape. That's why 1000 of the Futune 1000 use it.

    And while they're adding disc to the mix, they're not removing tape. They invented and perfected disc-to-disc-to-tape backup.

    Backup systems can be set up wrong. Hardware can be configured wrong. Software can be configured wrong.

    People make tapes fail.

    They misconfigure things, they don't swap tapes, and they don't take them offsite.

    If you gave me a tape from twenty years ago that had been smashed with a hammer and sat in water for a week, I could get the data off of it. I'm not saying that recovery will be cheap, but it would be complete, or nearly complete.

    At least that's true if the backup system was properly set up in the first place.

    Learning how to make tape successful is not that difficult. But most people simply don't put any real effort into learning it.

    ReplyDelete

Feedback Welcome

Please note, however, that spam will be deleted, as will abusive posts.

Disagreements welcome!