Sometimes, the most important policies and the little, daily routines that can save your bacon in the long run.
A great example of this is Router Configurations and Firewall Configurations. Note: I might sometimes use the term "router" by itself. Assume we're talking about routers and firewalls throughout.
SOP Friday: Router and Firewall Configurations
- Overview -
Routers and Firewalls are interesting equipment. They are critical to our success, critical to the client's network, and completely ignored. At least they're ignored when they work perfectly!
If there's any general "problem" with modern routers, it's this: They almost NEVER need attention. But when they do, it's needed urgently. As a rule, we set up routers and firewalls when they're new. They we don't touch them until a specific changed is needed (e.g., mapping at external IP to an internal IP), or the client changes ISPs.
Aside from being competent to configure these devices (not the topic here), you really only have two issues with routers and firewalls:
1) You can't get in because you don't have the logon credentials
2) The configuration gets nuked (by electronics, by accident, by a fool who works for a company other than yours)
Luckily, these can be solved with documentation and a couple of very simple SOPs - Standard Operating Procedures.
Okay, let's be honest. One of the most frustrating things about this profession is to come across a router that is totally useless because the previous technician . . .
1) Didn't change the passwords
2) Didn't write down the passwords, OR
3) Left and and won't give you the passwords.
This happens ALL the time. I don't know why these people aren't sued by business owners every day. It's amazing to me. I know they think they're somehow protecting their jobs. But the fact that you're here cracking into their router suggests that they've lost that battle.
Anyway . . . Grrr . . .
We have an easy way of never losing the passwords and logon information to the routers/firewall: Use your label maker to put the logon credentials on the bottom of the device. As long as you're at it, label the Router "Router" and label the firewall "Firewall." That will help your clients locate them when you need to walk them through a reboot over the phone.
Some people get in an uproar about putting the password on the bottom of the device. Unless this client ALSO has a web cam connected to a web site, pointed at the bottom of the router, there's no way for someone on the Internet to break into their system just because you put a label on the router. It can't happen.
But what CAN happen is that you get hit by a bus and your client (or another tech) can't get into their own firewall.
The next level of documentation is to fill out the Router or Firewall configuration form (see sample forms in The Network Documentation Workbook). Basically, this form includes logon info, IP addressing, route mapping, access rules, QOS information, etc. The Router or Firewall configuration form goes in the Network Specs section of the Network Documentation Binder.
The final step of documentation is to add this information into your PSA system. We have documentation divided into various categories. Routers and Firewalls go under "hardware" and include a summary of all relevant configuration information. The most important piece of this is the administrator user name and password.
Once you have a router/firewall set up - or any time anyone changes anything - you need to back up the configuration. Our standard procedure is this:
- Before you make any changes, back up the router to the c:\!Tech\Hardware\Network directory on the primary server. This is always accessible from the inside network, so it's a perfect place to put it. The file name should be in the format of YYYYMMDDxx, where xx is an increment for the day.
For proper date formats, see SOP Friday: Date Formats.
- After you make any changes, back up the router to the c:\!Tech\Hardware\Network directory on the primary server. Same file name format.
This before/after routine makes absolutely certain that you can go back to where you were before you touched the device. Yes, according to this procedure there should be a backup already. But you're being very cautious because it takes three minutes and you're a belt-and-suspenders kind of technician!
Note the increments: You might need to make several changes, test them from outside the network, reboot lots of equipment, etc. You may have three or four configurations if you get into a troubleshooting mode. the xx increment is very important. Don't keep backing up over the same file: If something goes wrong you could lose the new and the last version. Slow down, get more done.
Once you're done for the day, you can upload this config file to your PSA system as a document. That way, you have just one more backup in case something happens to that server.
Note on file names: You might also append a note to the file name. For example, "2011071502 firewall after new terminal server install.config"
Note on cheap firewalls (routers). If you have a firewall (router) that does not have a tool for backing up the configuration, send an email to your sales manager, asking him to send the client a quote for a real firewall (router).
A Couple More Notes
We also have a couple of minor SOPs to make life a little easier with firewalls and routers.
First, we really try to stick with two brand name vendors and only sell business-class equipment. I don't want any clients relying on $40 firewalls and wondering why they're not getting the performance they deserve. I want something with a good warranty, good documentation, and a good reputation with partners.
Second, we configure routers and firewalls so that they can only be configured from inside the network. That means we have to be on site, or we remote into the server and then open the firewall configuration page. Most ISPs require that they can get to your router from the outside, but most lock this down to access from within their network. So that's cool.
We also have policies about IP Address allocation and configuration, but that's a topic for a future SOP Friday.
- Implementation Notes -
You will need a configuration checklist for routers and one for firewalls. You will also need the configuration form discussed.
The checklist and configuration forms should be printed at the same time since you'll go through them together. The checklist will lay out all of the steps we mentioned above as well as specific procedures for your company. You may even have one for each brand (e.g., SonicWall vs. WatchGuard) so you can be very precise in you click-by-click instructions.
When you first start to deploy with this method, you might even put specific instructions in the service ticket. Call out that the tech will back up the configuration, map the IP addresses, change the access rules, save the config, and backup again.
If there has ever been a great argument for why you need a printed Network Documentation Binder, it's router configurations. When there's no Internet, or you're between ISPs, then keeping this information on your cloud drive or SharePoint is totally worthless.
Depending on your forms, you might have a little extra space. That's a good place to list out some commonly used port numbers as a reference for your technicians. This is particularly true with SBS and some newer technologies that use non-universal port numbers.
Who Needs To Know?
At the beginning of this article I mentioned that we tend to set up routers (firewalls) and then not touch them for a long time. As a result, you are NOT likely to remember every setting on every router at every client. Documentation is extremely important. Documenting the PROCESS that makes that documentation possible is also critically important.
You probably have one tech who is the "guru" and handles most router/firewall configurations. It doesn't have to be that way, especially with modern equipment, which is pretty easy to configure. Having a good SOP allows you to train other techs very easily.
Your Comments Welcome.
Introduction to Zero Downtime Migrations
Seminar on MP3 Download