Comments on Small Biz Thoughts by Karl Palachuk: The Reality of Offsite Backup

Good comments Juan. I particularly appreciate the ...

2010-04-28T20:53:38.436-07:00

Good comments Juan. I particularly appreciate the perspective that there's more than one way to get it right. Or wrong.

Working from the wrong angle... It seems that mos...

2010-04-28T20:42:24.231-07:00

Working from the wrong angle...

It seems that most of the opinions on here are throwing technology at the backup function blindly without analyzing business risk first.

Similar to insurance, you have to find a happy medium between ultra high premiums and shoddy coverage.

All backup and DR strategies fit well for someone, somewhere.

Tape is still good for some businesses becuase their tolerance for time to recover is higher...like a higher deductible.
Also, you can have multiple independant tapes that have complete backups very inexpensively. If you backup NAS dies, then what? You have to troubleshoot it, get it replaced, reconfigured, and in the meantime, no backups. With a bad tape, use the one from the day before.
Regarding reliability, the tapes and tape systems of today are significantly better than the previous years in terms of reliability...just read the label: Replace annually...and of course, keep it off the sun baked dash of your car. :-)
As far as the comment about safes, the problem you have run into is that the tapes were stored in a fire rated safe, not a MEDIA RATED safe. If it takes more than a few hours for your fire dept to get there, move.

Don't get me wrong, I like D2D as much as anyone...that is what I recommend to most.

Again, the solution depends entirely on the cost vs the risk tolerance.

People hem and haw about security and encryption, which is important, but they forget about physical security in their own office. Most techs are too busy gushing about the features of their backup/recovery solution to concern themselves with these basics.
Most small business office servers and backup units can be stolen with a pickup truck, a rock and a sawsall.

We use Zenith's solution because:
1. the data is encrypted with a password Zenith does not have when it is written to disk. There is your security.

2. The data is stored locally providing for a quick restore of files or entire server, or using the built-in virtualization to be up and running fast in the event of a catastrophic server failure.

3. The data is sent offsite, currently to another datacenter, with the capability of overnight shipping those images pre-loaded on a replacement BDR to an alternate location in the event of a site-wide disaster.

That covers most of the bases.

We have customers that use many different solutions...Tape, Tape Library, RDX, D2D, D2D2T, and our BDR devices...and some a combination...it doesn't mean one is better than the other...it is just the challenge of making sure you provide the customer with the one that best fits THEIR business backup and recovery requirements, not YOURS.

If you got this far, thanks.

-JT

I think no backup plan is complete without have th...

2010-03-02T18:12:53.654-08:00

I think no backup plan is complete without have these minimums:

1. Internal Backup: Volume Shadow Backup (file recovery) = Quick and simple for end user or adminstrator to recover in seconds. This reuqires extra disk storage.
2. Onsite Backup: NAS or some form of attached storage where you can backup file, database and images for quick onsite recovery or ability to boot from network or attached storage image. There is no way to test and restore a tape solution without physical onsite tech assisting or investing in a mechanical tape changer (ridiculous option).
3. Offiste Backup: for file and image recovery. This can be transported offsite via internet or pickup and replace courier. If you do internet based backup then you had better have ownership or access directly to the storage server. You must gain physical access to the server for quick dump of data so you can courier it out (this is what we do). Carbonite and these other cheap webbased storage are good in theory but when it takes 6 weeks to upload your data guess how long it is before you can restore it when it is a real disaster?? You guessed it......6 weeks if your lucky!

James: I want to make clear that I'm open to a...

2009-07-12T05:52:19.428-07:00

James: I want to make clear that I'm open to all kinds of stuff. But remember that someone who can't switch a tape is also not going to understand any kind of multi-tiered DR system.

As Gilbert Arland said, "Failure to hit the bull's-eye is never the fault of the target."

It takes human beings to make all these things work.

In the end, a multi-tied system is the best. That means Internet based backups play a role, but are not (by themselves) the answer.

Thanks, TheConsultant, for the discussion group.

Thanks, Karl. For any that are interested... I st...

2009-07-11T14:19:15.534-07:00

Thanks, Karl. For any that are interested... I started a discussion of methods and technologies at Consulting Notes by Scott Cameron

I agree that most internet based backups are not g...

2009-07-10T20:31:11.806-07:00

I agree that most internet based backups are not great for a disaster recovery in a quick fashion. They should be combined with a local backup of some kind. Going to the Internet should be the last resort, not the first.

If you look at where most medium to large businesses are going, it is away from tape

They use multiple methods to backup and recover the data they have. For example, If I am smart, I turn on Shadow copy for files if it is a file server, layer 1, If I have a SAN, I can create a snapshot schedule and replicate it (Layer 2), If I have a virtual environment and it happens to be VMWARE, a product such as vision core ESX ranger allows me to replicate data to another VMWARE machine or a disk someplace (Layer 3) or use a product such as acronis and I can restore to a different hardware platform or VMWARE. Finally, if they see need, they might go to tape or a product such as Unitrends as disk based backup system that uses disk I can take offsite (Layer 4). A product such as data domain also can do this

Some people just don’t understand disk or internet based backup solutions or how to design a true solution with multiple recovery points. You can do this for small businesses and make it affordable. Tape is just a PIA, tapes go bad, the software can be problematic. Also, if you are going to do a restore to a different machine in a disaster, good luck trying to get it to boot without a bare metal backup option. Finally, how often had you have a tape go bad and not able to restore the data? I have seen this numerous times in my career. How often do you test your restores for you clients?

They idea of backing everything up is good, but in reality are you going to restore a machine that crashed and had some corrupt data, no I would rather put down a fresh OS and restore the data. It is quicker in the long run and cleans up all of the crap.

I think the other issue is when you talk DR, the discussion needs to me about more than just tape. What happens when a fire takes out your building, what is your plan? Where are your people going to meet, how are your clients and customers going to reach you. If you have multiple systems, which is the most important to bring up first? DR should start with a plan first and data recovery needs to be part of the overall plan. Most small businesses are not going to pay for a full plan, but the discussion should be brought up.

If a fire destroys you building, how long is it going to take to get hardware and a tape Drive? That is also a 24 hour proposition. What happens if tapes do not go offsite? (Who is in charge of taking them offsite, if you rely on a human the error factor is very high)

Finally, lets discuss imaging. If I get it in the cloud, it is not that hard to get people to it if I have a good plan. They key again is designing a solution and not just pushing a product or a service.

What I think the problem is with most techs and small business solution providers is not Internet based backups or tapes. It is the ability to design a solution that fits the clients needs and make sure that solution works. Most are more concerned about how much recurring revenue a solution will provide and spent less time ensuring the solution will work.

By the way, a good solution that gets rid of tape and has mulitple options is Unitrends, http://www.unitrends.com/. You can tape disks offsite, encrypt them. You can also vault data to a remote location for a total diaster. It is just one of many solutions the market has to offer.

Also, 1000 of the 1000 Fortune 1000 compaines do not use tape backup. The trend is away from tapes. See this article as it discusses how enterprises are designing backup systems these days.

http://www.datadomain.com/pdf/DataDomain-CaseStudy-byIDC-Fortune50Financial.pdf

Michael: RDX is a good option. Still a little pric...

2009-07-09T21:37:00.544-07:00

Michael: RDX is a good option. Still a little pricy but kicks butt on restore speeds.

And don't worry about clients thinking some technology is "old" if they still have a fax machine, copy machine, and a phone system on premise.

:-)

Catching up on comments here. Boogie: OUR colo is...

2009-07-09T21:34:25.992-07:00

Catching up on comments here.

Boogie: OUR colo is good for about a month on generators and has riot-proof windows. But your point is very valid. After 9/11 I fully admit that I can't foresee all the stuff that could happen. We have one client backing up over the internet to Iron Mountain. Costs a fortune.

The Consultant: Give us your blog link.

Folks in general: If you're selling body part enhancements, feel free to leave links.

Karl, there are several reasons why we don't u...

2009-07-09T21:05:06.173-07:00

Karl, there are several reasons why we don't use tapes anymore. Firstly the tape drives are only as good as the software running the backup and unfortunately I am yet to find a robust, reliable 'tape' backup software. Sure we are able to configure the likes of backup exec, ultrabac etc to work, but then for no reason they just stop with all sorts of errors.

Secondly, most of our customers are of the mindset that tapes are old technology and therefore it makes us look bad if we recommend them. Thirdly, we have found success using RDX drives.

Most clients that use tapes, don't actually take them offsite - they just put them in the safe. This is a big no no. Did you konw that tapes will not survive in a safe when there is a building fire? Most 'fire-proof' safes will protect to some extent but the majority of fire proof safes will still heat up inside, enough to melt a tape. Most people don't know this.

We use Kaseya's backup solution (acronis) and RDX drives and have had the best results we have ever had. Firstly we can recover a server to different hardware in most cases under an hour. Secondly we provide an archiving service for our clients (another billable income source) whereby we take one of their backups each month/year and keep an archived copy in our secure facilities. We have quick easy access to it and it is easy to do and provides another revenue source, plus removes the stress and responsibility for the client.

There are many variations here that will work. Ba...

2009-07-09T12:37:12.808-07:00

There are many variations here that will work. Backup & disaster recovery isn't a one-size fits all topic.

At our clients we use a blend of solutions. At one, we have Zenith Arcas replicating between 2 office locations. At some others we do local image-based backups that the client takes offsite weekly.

If cost wasn't a factor I would absolutely host a replication site for my customers. I would have servers there that could do rapid virtualization of their images and I would then provide RDP access to them. I would also pre-contract with a disaster recovery site for office space, phones, fax, etc.

Not everyone can afford this though. I can sell it easily enough, but if I walk in to an architecture firm that has been hard hit by our recent financial troubles with this they'll just laugh me out the door.

This is why local consultants acting as Trusted Advisors will never be out of a job... every client has different needs and wants.

We've all discussed at length the various options. I talked about some today in my blog. It's how we build and deliver our solutions that matters.

Karl, Putting the appliance in your colo helps bec...

2009-07-09T11:15:06.028-07:00

Karl,
Putting the appliance in your colo helps because in most cases, you're local to your client. But what about when there's an extended city wide outages that is longer than the generators at your datacenter can sustain?

Karl, It is always up to people to make any syste...

2009-07-09T05:39:34.252-07:00

Karl,

It is always up to people to make any system work properly. nuff said.

Framework IT www.frameworkitllc.com is a channel only software and methodology development firm and have created an environment that empowers our partners and provides them with the opportunity to help their SME clients develop a 90% self authored Business Continuity Plan with the resellers help. It is a web based solution that uses remote/off-site replication. No tape here.

FYI-Framework IT was recently named by CRN as an Emerging Vendor of 2009.

Thanks you for allowing me to post this, and tell Eric hello

David Dempsey asks a great question: What do I lik...

2009-07-08T19:05:33.233-07:00

David Dempsey asks a great question: What do I like?

Ugh.

We have been extremely reluctant to adopt an internet-based backup strategy for several reasons.

First, we have to have 99.999% reliability or nothing.

Second, the data has to be in our control at all time. That means off to cloud in Arizona, Canada, or the Philippines is out of the question.

Third, we need to be able to restore to a machine within a few hours and deliver a working solution to the client before four business hours have passed. That's very possible and quite reasonable to expect.

We need to have a relationship with every single person who is going to have access to this data. Period.

[Some people will say that this is unreasonable. Fine. Tell that to an accounting firm with 14 accountants who handles nothing except critically sensitive data. Tell that to the intellectual property attorney. If they really understand what you're doing with their data, clients want it under your STRICT control.]

Until very recently, there were two options that were front-runners for us.

One was to put a SonicWall CDP in our colo facility and duplicate the real-time images from client CDPs. The primary problem with this is cost. Gulp.

The other option was to set up an HP Storage server and roll our own offsite imaging system.

We did try to implement the Kaseya offsite backup. But we learned an extremely important lesson: Most clients don't have an internet connection that allows the Kaseya offsite backup to complete. Too slow. Too many resends. Too many lost packets.

We have not used the Zenith BDR solution because we could not choose the offsite replication partner.

Thankfully, Zenith Infotech has introduced a product that allows onsite backup images AND allows us to put an identical unit in our colo.

Unless new information arises, our plan is to implement the Zenith strategy with as many clients as we can.

:-)

Pete at Framework IT: Give us your URL. And pleas...

2009-07-08T18:26:03.126-07:00

Pete at Framework IT: Give us your URL.

And please describe your system in a paragraph or two.

As for your comments:

A) You can't dismiss tape out of hand except as ploy to sell a non-tape solution. 1,000 of the Forture 1,000 companies rely on tape. It's not their only backup system, but it's a critical component.

B) You're completely correct: If anyone has a self-authored DRP, they're going to be a major leg up on actually executing said DRP.

But here's where it all falls down:

- Let's say you can't figure out how to configure a tape rotation (and you're unwilling to learn).

- So you buy a CDP device and never clear the logs. So when recovery time comes around, you find you haven't been backing up for months.

The same technician who can't figure out tape will not be able to figure out SonicWall CDP.

So then you go to an off-the-shelf internet backup because someone you've never met mentioned it in a Twitter Tweet.

It's like the porn industry. You want to believe that this is a 19-year-old hottie, but it's really an escaped shemale convict who's so obese she hasn't gotten off the couch since Clinton was president.

When the stuff hits the fan, how do I know where my clients' data is, who has access to it, and what their protocols are? As I mentioned before, there will come a day when all this stuff hits the front page of the New York Times -- and I don't want my company called out by name!

To be clear: There ARE good, legitimate companies doing REAL, high quality internet-based backups. But they are few and far between. And the technicians buying and reselling these services are, for the most part, not doing enough homework on this.

From your comment, it sounds like you have a system that allows a build-your own system of some kind? (I haven't done the research.)

More info appreciated.

:-)

As I said . . . people are passionate about this s...

2009-07-08T18:10:26.672-07:00

As I said . . . people are passionate about this stuff.

I agree with most of the comments here. Offsite backup needs to be done right. And they need to be part of a much bigger picture.

For way too many people, offsite internet-based backup is 3rd rate panacea because it's being administered by a technician who clicked a button and doesn't understand anything about how to actually verify that it's working.

MShipman summed it up: For some folks, you can't convince the client to spend 3 minutes a day protecting their entire multi-million dollar infrastructure. You might be tempted to say "We will care as much about your network as you do," except that every technician I've ever met (except one) cares more about his clients' networks than the clients themselves.

This expanding farce is possible because clients want to write a check and make the problem go away.

Nothing against the clients: We as an industry have told them that we'll transfer the liability for their network from them to us for a flat monthly fee. You can't blame the clients at the end of the day. But you also can't sell smoke and mirrors and let the client think everything's fine.

When the smoke clears and the fire fighters go home, the owner will turn to you and say "When can we back up? We've got a big telepresence seminar scheduled for noon."

ElliotRoss has a good idea to store off all the crap that most people don't need all the time. I wonder if it would be better to put all that stuff on a hosted SharePoint site for $29.95/month.

:-)

The part of the issue you cover is fair. There is...

2009-07-08T14:54:59.687-07:00

The part of the issue you cover is fair. There is more to the issue however and that is where you have "thrown the baby out with the bath water". As the founder of a SaaS Business Continuity Planning methodology and application we have spent 1,000s of hours looking at this issue. Offsite is just that, offsite. It can be a SAS 70 certified data center on the other side of town. That shouldn't take 2 days to get the drives which contain ALL the firm's data with time stamps, but in a regional disaster it may take 3-4 days to get the 5-10 servers they need to start the rebuild.

Offsite backup done properly is the ONLY way to go. Tape is not a good answer at all.

The key to getting people to understand is helping them develop a self authored Business Continuity Plan. This will help them understand their vulnerability and the best way to assure their survival and not become a statistic.

I understand your point. I think that the problem...

2009-07-08T14:12:39.593-07:00

I understand your point. I think that the problem often lies with the small business's lack of interest in backup and D.R. I constantly get asked why my solution costs hundreds of dollars per month, when they could use Mozy for $100 a year or something.

I personally believe that tapes are a lousy option for small businesses. Your point that small business people don't have the discipline or interest to rotate tapes is evidence of the problem. I always preach that BU/DR is a system AND a process. You can have the best system, but the process has to be implemented flawlessly or it will fail.

Here's my questions for you: we know what you don't like, but how do you like to do it?

David Dempsey
Managed Data
http://www.managed-data.com

For those reasons I don't look at online stora...

2009-07-08T10:38:59.305-07:00

For those reasons I don't look at online storage (or 'backup') as a primary DR method.

What I am considering it for is the opposite - as an org that has hundreds of Gigs of large media files that may never be used again (but need to be kept around for 7 years) I am considering getting those damn things off of my network. So that DR plans can worry about the truly critical need it NOW data.

In the event DR plans need to be invoked - the last thing I need to worry about are those media disk wasters.

I agree with your thoughts in principle. For the ...

2009-07-08T09:25:44.842-07:00

I agree with your thoughts in principle. For the MOST part any of our customers using an off-site solution have something to complement onsite. For some of our customers an off-site only solution is actually better than the tape they had in play previously. While I always found it absurd, some customers did not have the discipline to facilitate daily backups. Yes, I mean physically take a tape out and put the next day's tape in. In those scenarios having a "File Recovery Plan" is better than a "Who knows what we will get plan". I think the cornerstone to all DR, or otherwise, plans though is being completely open and honest with the customer about expectations.

This is the kind of a thing I used to advise my pa...

2009-07-08T08:14:24.155-07:00

This is the kind of a thing I used to advise my partners through and eventually just gave the f up. People don't understand the value of offsite backup and they don't sell it as such either.

"When your office goes up in smoke, your data will be elsewhere so you're all good and you don't have to worry about taking tapes and hard drives home."

Yeah, but be prepared to be out for a week while the request is put in, drive is shipped, gets there, restore work begins and you're back up and running.

Enterprises get this, and they backup the granular pieces because offsite backups are for retention control and long term archiving. They are not very useful for DR work unless you're really, really, really savvy. Most aren't.

That said, we use our offsite backup for both local and remote and replication and granular and archiving. But we do so because we understand the product and have 2-3 other systems in play at all times.

-Vlad