Friday, March 09, 2012

SOP Friday: Checklist for Major Scheduled Maintenance

Most of the time, our maintenance and repair work only affects one desktop, or sometimes a few desktops. But occasionally, we do some work on a server or a piece of equipment that either will or might cause an interruption of critical systems during work house.

This could could be as simple as updating the drivers on a network card. It includes adding rules or features to the firewall, upgrading the line of business application, and making major changes to the Exchange Server or a SQL server used by everyone in the office.

There are two keys to success in this scenario. The first is to take is seriously. I know that sounds obvious, but too many technicians only look at the job (which seems simple) and not at the client (who just wants uptime). The second key to success is communication. You're going to go out of your way to communicate with the client. And if there's a third party technician involved (hardware, software, network, line of business), you will carefully manage all communications with them as well.

Here's our basic Standard Operating Procedure for Major Scheduled Maintenance:

This document is intended to outline the procedure for implementing scheduled maintenance where any number of services, servers, or networks may experience an interruption that affects more than one person.

1. It is our preference that all work that may result in service interruption requires one our our technicians to be onsite during the maintenance. That means we will do the work onsite rather than remotely. If work is to be performed by a third-party technician, we would like to be there. We will follow this process whenever practical.

2. Inform the client technical contact(s) via email as far in advance as possible. Prior notice of one week is ideal. The email must include names of servers, LOBs, and services affected, as well as desired start time and expected duration of interruption.

A copy of the email should be sent to our support email address with the service ticket number in the subject line. This guarantees that the Email2AT email parser will attach the email to the appropriate service ticket.

3. Send an email to client technical contact(s) the day before the scheduled maintenance.

4. Inform client technical contact(s) via email in the morning of the day of the approved maintenance date.

5. Inform client technical contact(s) verbally 30 minutes prior to approved maintenance window.

6. Once the maintenance has been performed, verify that all affected services/servers/LOB applications have been brought back up and are running properly. Have the client contact verify that everything works.

7. After verifying success, inform client technical contact(s) that the maintenance has been completed.

This process is to be performed by the service manager, or in close association with the service manager. In most cases, the maintenance is routine. But just in case something goes wrong, the service manager needs to know what's going on and must be available to re-route technicians and manage client communications if needed.

Note that the client might be very skittish about any downtime, even if you think it's a ten-second blink. The truth is, stuff can go wrong. That's why we have a policy. Because you have heightened the client's awareness, they might request that the service be moved to another date or time.

The most likely client feedback you'll get is either "That's a good day/time" or "That's a bad day/time." You probably don't know when your clients are performing payroll processing, insurance audits, or other activities that can't give way to simple maintenance. The client will be grateful that you informed them of the procedure and gave them the opportunity to move it to another day.

Checklist for Major Scheduled Maintenance

Client: __________
Project: __________
Service Request/Ticket # __________

_ Email sent to client informing them of maintenance on date of: __________

_ Acknowledgement received for date of: __________

_ Email sent to client the day before scheduled maintenance.

_ Email sent to client on morning of: __________

_ Client informed 30 minutes prior to maintenance window.

_ Maintenance completion date and time: __________

If you have a PSA system, you can create a workflow to make sure these steps are taken. If you don't have a PSA system, you may want to have both the tech and the client rep sign a document on completion of a major project (e.g., a new router installation).

Controlling Firefighters and Heros

As you know, one of my themes is "Slow Down, Get more Done." This is a case where that's very important. There is a tendency in our business to jump right in, get to work, and start clicking things. But there are times when that's just not the right thing to do.

Most experienced technicians can look back on when they thought everything would go smoothly and it didn't. And the client turned to them and wanted to know what went wrong. Even if the client was very understanding, they still managed to comment that "It would have been nice to have some warning."

This procedure might be overkill most of the time. But on the day that two minutes of expected downtime becomes an hour, you'll be glad you coordinated with the client. This process is normally really uninteresting and just "SOP." But it can go a long way to helping with client relations.

- Implementation -

Implementing this procedure is simple. Agree to a process similar to that outlined above. Talk to your team and make sure they understand that this is important. (No firefighters needed.)

To the extent you can, build these processes into your PSA system.

This kind of policy requires that everyone on the team

1) Be aware of the policy

2) Practice the policy

3) Correct one another's errors

4) Support one another with reminders

Your Comments Welcome.

- - - - -

About this Series

SOP Friday - or Standard Operating System Friday - is a series dedicated to helping small computer consulting firms develop the right processes and procedures to create a successful and profitable consulting business.

Find out more about the series, and view the complete "table of contents" for SOP Friday at

- - - - -

Next week's topic: Phone Procedures 1 - Philosophy and General Rules


Now Available:
Introduction to Zero Downtime Migrations
Seminar on MP3 Download

Two hours of audio training - Plus two slide decks in .pdf format.
Agenda: Project Management in a Managed Service Business and Zero Downtime Migration Strategies.

No comments:

Post a Comment

Feedback Welcome

Please note, however, that spam will be deleted, as will abusive posts.

Disagreements welcome!