Friday, July 22, 2011

SOP Friday: Responding to Viruses

Ugh. We all hate viruses. They represent that rare I.T. problem that can be challenging but not rewarding. When you conquer most problems, you emerge with a better system, a faster network, more storage, . . . or something worthwhile. When you conquer a virus, you just get to use your computer again.

Modern viruses (worms, trojans, etc.) can be almost unbelievably destructive. They can infect every pore of a system - dll's, registry, O.S. files. Everything.

And more importantly, modern viruses can can HUGE amounts of time to fix. And sometimes they can't be fixed. And that means they can be extremely unprofitable! When a new computer with a fast processor and all the software you need is less than $1,500, there's a limit to how many hours you want to spend "fixing" viruses.

A standard operating procedure is in order.

SOP Friday: Responding to Viruses

- Overview -

Unlike the SOPs we've discussed so far, this one is strictly defensive in nature. How do you restore the machine, keep the client happy, provide a timely response, and make money (or at least not lose money)?

More than anything, virus protection is most successful when you are very well prepared. That means the right hardware, the right software, the right configurations, the right customer training, and the right practices. All of that makes it possible for you to have the right response. Without adequate preparation, there may be no good response. Let divide this world of preparation so we can conquer it.

First, you need to lay the groundwork with hardware and software. If you're a managed service provider, your life gets pretty easy here. If you're not, then you just have to convince your clients.

Note: Some of these "policies" are really the essence of Standard Operating Procedures. We recommend one way of doing things. We push. We cajole. We quote the right tools, etc. We can't force a client to protect their systems. Which leads to one of my favorite sayings:

We can't care more about the client's network than they do.

Our managed services contract (see Service Agreements for SMB Consultants) specifically requires that the client have a good, working firewall that's under warranty or covered by a maintenance agreement. In other words, it's the latest and greatest, and can protect them from new attacks that show up unannounced.

It's amazing how effective hardware firewalls can be at detecting and stopping viruses - even the ones where clients are tricked into "installing" the Anti-Virus 2011 virus. Now, let's be honest. We're talking about a $750 firewall, NOT a $49 firewall. See the discussions around last week's SOP Friday discussion.

As for software . . .

This has two components. First, there's anti-virus software. This one is fairly obvious and takes very little convincing. The main decision is whether you're supplying and annual renewal or a monthly subscription. If you have annual renewals, you need ticklers to remind you to send the invoices.

The other piece of the software puzzle is Newer Programs. Old programs - like Microsoft Office 97 and Windows XP have some vulnerabilities that will never be fixed. Newer programs and operating systems are inherently more secure. Moving clients to the newer stuff is a never ending battle. We are constantly reminding clients that modern software is part of their security.

Hardware, operating systems, and software, must all be kept patched and updated. That means you need to have those processes as part of your maintenance plan, whether on managed services or not.

Imaging Machines

We do not currently image desktop machines. Our policy is that machines should be properly maintained, we limit our exposure to virus incidents, and we fix machines when a virus hits.

On related notes, it is our policy that we do NOT redirect My Documents to the server. We don't encourage clients to use My Docs. All information that's important needs to live on the server. Period. The server is on redundant drives and backed up. The desktops are essentially disposable.

Having said that, I have often heard people say that they image desktops and use them to restore from virus attacks. This sounds great as long as the desktop never changes. If you need to restore an image and then run all the updates since the image was refreshed, it may not save you any labor.

Imaging is a viable option. We just don't happen to practice it.

Note: We DO allocate space to let Windows store previous versions so we can roll back to before the Virus hit. That has saved our bacon more than one. Just remember to do it.

Client Education

There are two kinds of client education related to viruses. First, there's education on your contract/agreement and what your response will be. Second, there's training on how to avoid viruses and what to do when one hits.

Our contact is very clear on this point: All maintenance, including all software installations, must be performed by an employee of America's Tech Support. So, when a client installs a virus on their computer, it is not covered by the managed services agreement.

Now, the truth is, we're going to believe the client that it's an accident and fix the first incident for free. But we're also going to make it very clear that they next one is on THEM. They'll get a bill for X hours at $00.00/hour. But we'll also make sure they know the next one will be for $150/hour.

Client education consists of emails, memos, newsletters, harping, haranguing, and whatever else we need to get across a few simple points:

1) You already have an anti-virus program. You don't ever need to install another one, no matter what pops up in front of you.

2) Whenever you receive an email with an attachment and you did not ask that person to send you that attachment, Delete It!

Period. I don't care if it's your mother or your boss. If need be, email them back and ask if they sent it.

3) Whenever you receive email with links that look urgent, do not click on them. Go to the appropriate web site yourself by typing the regular address into your browser (e.g., your bank). Log in. If there's an urgent matter that needs your attention, it should be flashing in front of your face. Delete the email.

4) If you're browsing the web and a window opens up by itself, click the Red X in the upper right hand corner. Do not click . . .

- Yes
- No
- Accept
- Decline
- Close
- Unsubscribe

or anything else. Just click the Red X to close the window. If you feel violated, reboot your computer.

5) If you get an infection, log off of your computer. If you can't log off, restart the computer (force a power down and restart) and do NOT log on. We need the computer on to connect remotely.

The Bottom Line: Educating your client about your policies and their expected behavior will help limit your liability/exposure during a virus infestation.

Stand Firm by your processes and procedures. 99% of modern viruses are stopped by almost any anti-virus software . . . until the user clicks OK. In other words, it is almost always the user doing this to themselves. They need to understand that.

Standardized Response

So . . . when you finally get a service ticket about a virus infection, what do you do? Here's a rough outline of our process.

1. As with any ticket, determine the urgency and assign a priority level.

2. Have a discussion with the client. Remind them about the policies. Verify the maximum number of hours we will put into fixing a machine before we move to billable labor. Request how many hours of billable labor are acceptable before the client wants us to stop working on the issue and simply re-install the O.S.

It is very important that you agree on limits to your time and to what happens when you reach those limits.

3. Connect to the machine remotely and log on in safe mode.

We do this with Zenith Infotech and LogMeIn. Other programs may provide similar functionality, including machines with Intel V-Pro network cards and LogMeIn.

If you don't have such remote access, then you'll need to be onsite. In either case, log on in safe mode. This will stop user-specific viruses from continuing to cripple the machine.

4. Attempt to clean the machine with your standard company-approved tools. These may include Trend, Symantec, AVG, Hit Man Pro, or whatever your decides is the best fit for you.

5a. If that appears to work, reboot the machine, log on as the user, and attempt to verify that the virus is gone.

5b. If that appears not to work, attempt to restore the machine to an earlier version running the tools built into the operating system. If you know the day the machine was infected, you should be able to restore to a previously working version.

6. If you believe the virus has been cleaned, apply all appropriate updates, and create a new restore point.

- Implementation Notes -

Implementing this policy can be very troublesome. Many clients insist that local users have admin rights. That's now always in their best interest. If you're losing money every time they get a virus, then it's not in your best interest either.

If a client allows themselves to be infected more than once, you really need to take them out of the local administrator's group. This might mean that the client needs to pay you to install a few programs here and there, but the cost is very small compared to a four-hour bill for fixing viruses.

As I mentioned earlier, an appropriate response means the right hardware, the right software, the right configurations, the right customer training, and the right practices. That means you need to really think through these processes and push them on to employees and clients every time there's a virus.

Note on "All You Can Eat"

I have never been a fan of "All you can eat" managed services. After sixteen years in this business, I know "all" some clients can eat is my entire company! Fighting viruses is a perfect example of that. You need to limit your losses with good processes and policies.

- Forms -

There are no specific forms for implementing this SOP. You might write up a brief description of the procedure and put it into your SOP or binder.

This kind of policy requires that everyone on the team

1) Be aware of the policy

2) Practice the policy

3) Correct one another's errors

4) Support one another with reminders

Your Comments Welcome.


Want to figure out how to make money with Cloud Computing?

Join the Cloud Services Roundtable today and listen a great series of podcasts!


  1. "Stand Firm by your processes and procedures. 99% of modern viruses are stopped by almost any anti-virus software . . . until the user clicks OK. In other words, it is almost always the user doing this to themselves. They need to understand that."

    This isn't true anymore, a growing chunk of spyware will install without any user interaction by exploiting flaws in browser/flash/reader/java/quicktime - then again if your customers are on a managed service plan with automatic updating of said programs you are usually protected from a large percentage of those. But there are still plenty of 0-day flaws out there, so simply going to a site that loads a PDF crafted to exploit a flaw in reader is all it takes.

  2. Well, yes, there is some microscopic probability that an innocent user can stumble onto a site that automatically infects them.

    But with proper hardware, software, etc. this is extremely unlikely.

    This speaks to the need to keep clients up to spec with new programs and all the patches.

  3. Anonymous12:18 PM

    I agree with everything except #4. Even the Red X can be a free ticket to install. If the "popup" window is in fact part of a full screen graphic and the red X is a hotspot on the web page, it could actually be a button and that button could be the "Install me" button. I have actually seen one of these in the wild. I tried to move the popup window by dragging it and it wouldn't move since it was just a fake window being part of the whole background image of the web page. So, I usually tell people to right click the program (usually the web browser) in the status bar at the bottom and select Close from there. Hopefully THAT is a real Close since it's part of a Windows dialog.


    4) If you're browsing the web and a window opens up by itself, click the Red X in the upper right hand corner. Do not click . . .

    - Yes
    - No
    - Accept
    - Decline
    - Close
    - Unsubscribe

    or anything else. Just click the Red X to close the window. If you feel violated, reboot your computer.

  4. Unfortunately, Frank, I agree completely!

    I'm not a big fan of a hard shutdown, but I'm okay with clients who just pull the plug when this stuff happens.

    Like I said, it's challenging and not rewarding.

  5. Karl,

    I really appreciate this series. In this post, I need to agree with Gantry though. A well-prepared computer without local admin rights and with all Microsoft updates applied, behind a SonicWall with all the bells and whistles turned on can be badly infected by accessing a legitimate website with JavaScript or acrobat code injected by hackers. The code will invoke the exploit of the day with full admin privileges without needing the user's help. All they need to do is land on the wrong website while having a Java, Acrobat, or Flash that lack the latest patches. It's instantaneous.

    I never felt that I could be a competent MSP until I found a third party patch management system. Kudos to GFI Max. They have it and it works beautifully. Our managed clients no longer get infections and our unmanaged clients sign up right after we do a $300 cleanup that was 100% preventable.

    We're also believers in OpenDNS MSP edition. It gives us yet another layer of protection without sacrificing Internet speed. They're pretty quick to block infected sites at the DNS level.

  6. Thanks, Mudgie. I know it can and does happen.


Feedback Welcome

Please note, however, that spam will be deleted, as will abusive posts.

Disagreements welcome!