Patch Management Philosophy
I know some people think "philosophy" seems out of place in the world of technology. But we all have some philosophies that guide us. For example, when do you sell a new operating system? It used to be common wisdom that you don't install a new version of Windows until Service Pack 1 is released. That hasn't been true for a long time. For almost ten years now, Microsoft has had massive public beta programs that put their O.S. in the hands of millions of users for many months before the release date. So the release version is very stable.
Service packs are another story. Here, too, Microsoft has improved dramatically. But there's still good reason to wait. First, SPs are 98% just a collection of previously released fixes. So a well-patched machine will have the critical elements. As a result, modern SPs provide a certain level of consistency and stability, but their deployment is not critical.
For non-Microsoft service packs and patches, we're more leery Because we don't see them as often or as consistently, we don't know how reliable or safe they are. As a result, we are very careful with installs and very watchful when we perform them.
If you support a specific application, you may know it's quirks and patterns. If so, you may have a philosophy for that application. If not, then fall back to the more generic (careful) philosophy for lesser-know application updates.
One final example has emerged in recent years: The Zero Day Attack. These are viruses or attacks that become widely known on the same day. There is no time (zero days) to prepare for the attack. Sounds like there's nothing you can do? Not true. As long as you talk about WHAT you will do when the day comes, you can have some level of preparedness.
In this case the SOP involves having pre-defined lines of command and communication. When a zero day attack happens, who will decide how your company responds? How will you communicate with the team? How will you communicate with the clients?
As I said, we all have these philosophies. And every technician you hire has a set of philosophies as well. But each company needs to operate on one philosophy. If you're the owner, or service manager, then YOUR philosophy is the one that matters. You need to write it out and educate the rest of the team.
When major service packs or patches come out, you need to orchestrate a plan for installing them. This probably means creating one service request for each client server or one SR for each client. You might schedule these for the same time as the monthly maintenance, or maybe assign the whole lot to one or two technicians.
The key to success is that you 1) Have a philosophy, and 2) Propagate it to your staff. This is best accomplished by writing out your philosophies and putting them in your SOP document/folder. After that, you need to have a training, talk this out with your staff, and agree on how you'll proceed.
Manual Patch Management
If you do not have an RMM (remote monitoring and management) system such as Continuum, LabTech, Level Platforms, or PacketTrap RMM, then you will be installing these updates "by hand." Alternatively, one of your philosophies may be to enable automatic updates from Microsoft. But you still need to apply updates to Office applications. Many non-critical updates won't be applied by automatic updates. And, of course, you need to apply updates to all those non-Microsoft products as well.
If you don't have an RMM system, you will have to create a schedule for updates.
One possible policy is to apply updates whenever you find yourself logged into a client computer. That's not bad, but if you do this, make sure you keep a list of all updates for that client on the c:\!tech directory of their server, or in your own PSA system. That way, you can make sure that you do a thorough job each time.
The problem with this approach is that you might not touch every machine.
Therefore, you should have a policy to check updates on all machines on a regular basis. For servers, this is part of the regular monthly maintenance. Whether performed remotely or on site, all updates will be applied at least once a month. For desktops, you will need to create a service request to update all machines. I recommend one service request per client (not one per machine). You will need to have a document for keeping track of which machines have been completed.
You should go through this process at least once a year. Ideally, you will do it once per quarter. You can see why an RMM tool can save you a LOT of time. These "sweeps" can be extremely time consuming if you manage a lot of machines. If you have an RMM tool, you can apply updates weekly or monthly with almost zero additional effort.
Automated Patch Management
Automated patch management is a life saver for managed service providers. If you have a thousand desktops and 100 servers, patching them adequately would be a full time job. The more you can automate the better.
The key with automated patch management is to decide how much you want to be involved. You can simply pass things through (e.g., If Microsoft released it, install it.). Or you can test each patch for yourself. Here's the process we follow:
1. Patch is released by Microsoft (Tuesday)
2. One business day later we look to see whether the patch has been recalled
3. We deploy the patch to our internal servers/workstations
4. Three-five business days later (assuming no problems), we deploy to client servers/workstations
If your RMM company has a patch approval feature, you could simply set up groups so that patches are deployed once they have been white listed. In this way, you simply rely on your RMM vendor to do the vetting for you.
For a variety of reason, you might choose to blacklist some patches so they do not get installed. Sometimes this is applied throughout all your clients (for example, if there's a Microsoft patch that just causes problems). More commonly, you might have a specific customer's Line of Business (LOB) application that is incompatible with a specific patch or service pack.
Patches have three possible statuses: Pending Approval, Approved, and Denied.
When patches are denied, you need to be very certain that they do not get applied. That means you need to have strict policies in-house to make sure that everyone respects the "denied" status.
Interestingly enough, the "update" that is most likely to cause problems with client LOBs and web-based applications is Internet Explorer. How many times have we seen something break because a newer version of IE is installed? While I don't recommend avoiding all IE updates, please treat them with respect and be careful.
A Few Final Notes
Desktops can always be fixed, and don't normally affect the entire company. Servers are much more critical. Servers should always be backed up before patches are applied. Whether manual or automatic, stop the process until you have a good backup. I know things almost never go wrong with "simple" patches, but they sometimes DO go wrong. You need a way back without wasting a lot of (down)time.
As a general rule, the RMM company's patch approval process is probably good enough to rely on. But you know your clients and their computers better than anyone. You need to decide on a patch management level that makes sense and is comfortable for you.
You need to articulate that philosophy and make sure your staff understands it.
Your Comments Welcome.
- - - - -
About this Series
SOP Friday - or Standard Operating System Friday - is a series dedicated to helping small computer consulting firms develop the right processes and procedures to create a successful and profitable consulting business.
Find out more about the series, and view the complete "table of contents" for SOP Friday at http://www.smallbizthoughts.com/events/SOPFriday.html.
- - - - -
Next week's topic: Organizing Your Company Files and Folders