We Need a Real Data Center

We Need a Real Data Center

 - Lessons Learned, episode 59

 

In 2006 I bought a friend’s company. Well, essentially, I acquired his company in exchange for him coming on my staff at a very nice salary for a couple years. His company included a few regular consulting clients and a huge number of web sites set up in a rack full of crappy old “servers” and “network devices” that ran mostly on Unix.

I acquired all that so I could get his one big client. They became our largest client at the time. We also got a couple of smaller clients that I was happy to keep. The plan for all the micro clients (1-3 users) and all the hosted web sites was to pass them off to other consultants. First, we needed to move those web sites to the cloud so I didn’t have to stay awake at night worrying that the Frankenstein rack of junk hardware would fail.

At the time, I had a mostly-full equipment rack that housed three servers for us, switches, and a couple of servers that housed products we were test-driving for vendors. It was nice and clean and organized and beautiful. We used blue network cables on this rack and it was known as the blue rack.

Then we had the other rack full of junk, housing hundreds of web sites. We used green network cables on that one so, as you guessed by now, it was called the green rack.

All of this was in a room we’d built up to be our “server” room. It was about ten feet by four feet, had a dedicated through-the-wall air conditioner, lots of 30-amp electrical outlets, and a bunch of UPS backups.

This was a decent little server room, but hardly a data center in any sense of the word. In 2006 we were committed to moving all servers and services to the cloud, both for our clients and ourselves. So this little server room would do nicely until we could move things to a real data center. We’d already started planning that move. And then . ..

The owner of the building next to ours assumed that, since his parking lot had been there for decades, it would be okay to dig a hole in it for something. The back hoe went down about two feet before it cut through a major electrical cable and blew up a transformer, forcing a power outage for blocks around that lasted more than a week! The transformer was one of those 4-foot cubes you see from time to time. I guess you don’t replace those things quickly.

Anyway, we were instantly on battery backup, but that wasn’t going to last days. I ran down to Home Depot and bought a generator, then built a cable beefy enough and long enough to go from the parking lot to our server room. I didn’t know whether the UPSs were rated for generator usage, but it didn’t really matter. I plugged them in and ordered new UPSs to be delivered as soon as possible.

And then we signed the deal to set up our racks in the real data center. Our plan hadn’t changed, really. We were just forced to speed up the execution. Unfortunately, that experience cost a good deal of money. Aside from the UPSs, the generator, and the cable, we had to pay someone to sleep in the office because we couldn’t close the doors with the big cable running through the place.

Lessons learned. Well, this is a tough one. Aside from learning that the building owner next door is a moron, the rest was just poor timing. We had a plan to move everything. We just didn’t get to do it on our schedule.

We did not have a week-long power outage in our emergency planning, but our experience dealing with disasters made this one pretty low stress under the circumstances.

Looking back, my final assessment is that we should have been done two weeks earlier.

But for context, most of the rest of the industry didn’t gets servers out of the office for at least five or ten years after us!

 

Feedback always welcome.

-- -- --

 

Episode 59

This Episode is part of the ongoing Lessons Learned series. For all the information, and an index of Lessons Learned episodes, go to the Lessons Learned Page. https://blog.smallbizthoughts.com/p/lessons-learned-blog-series.html

Leave comments and questions below. And join me next week, right here.

Subscribe to the blog so you don't miss a thing.

 

:-)

 

-----------------------------------------------------------

 

      AD = Community Red

 

-----------------------------------------------------------