Wednesday, August 17, 2016

Booting HP Proliant Server from an SD Card

I'm in the process of building a new HP MicroServer for a client. This is absolutely my favorite server for use in our "Cloud Five Pack" offering. That's a combination of the following for use by five users:

- 250 GB Storage
- Hosted Exchange mailboxes
- Hosted basic web site
- PC remote monitoring
- PC patch management
- PC virus scanning
- Spam filtering
- Microsoft Office (Outlook, Word, Excel, PowerPoint, Publisher, OneNote)

We offer an add-on to this package that provides a server on site (which we own). It is used to provide local security, Active Directory, and provide very light file serving. For example, if the client has an on-site installation of QuickBooks or Business Works, that lives on this server. All other company files are in the cloud with a mapped drive.


Server Lite


We refer to this on-site server as "server lite" to clients. We want to make sure it is NOT in the league with Small Business Server. It will not serve up in-house Exchange, SQL, web, SharePoint, CRM, and file sharing.

The reason I gave all that background is because most of my readers are nerds. We nerds have a tendency to ask questions like, "What about capacity planning, stress testing, and loading up line-of-business applications?" This server is intended to be just right - powerful enough and extremely reliable - for the specific jobs listed above. It is not intended to replace six virtual servers.

I love to run Microsoft Server 2012 Essentials on the HP MicroServer. This is truly a killer combination.

Because we want the focus to be on cloud services, it's important to downplay the on-site server. Having said that, I think the local server gives us a great deal of power and flexibility. It provides:

- Active Directory
- Backup (either to or from the cloud)
- DNS
- Local "lite" application such as QuickBooks
- Remote access if needed
- A way for me to get on the client's network without disturbing anyone. So I can test some things and access internal resources. For example, if we need to make updates to the network scanner, I can access that from the server but not the Internet. We also set up firewalls so they can only configured from the internal network.


The HP MicroServer Gen8


The server I bought comes configured with 8 GB RAM and four 1-TB hard drives. Because it has an on-board RAID controller, my plan was to create two RAID-1 mirrored drives. One is for the operating system and "tech" files. The other is for company data and backups.

The client has about 100GB of "live" data and another 60GB of old data. All of that lives in the cloud. So we back it up by bringing a copy back down from the cloud to this server. The client doesn't realize it, but their on-site server is their backup for all hosted services.

I did set up the O.S. as expected. It takes up almost exactly 24 GB of disc space. That includes a 2GB swap file, created by Windows. That seems low for a system with 8 GB RAM, but it's likely to grow over time.

That's when I got to thinking about booting from SD. I knew this option existed, but we've never done it before. So I decided to give it a try. If I'm unhappy with it for any reason, I can just use the drives I've already set up.

I won't go into too much detail on the build once the Microsoft Installation starts. Once the file copy starts, the process is identical. Loading from USB media to SD operating system is only a little faster than USB to spinning disc. I assume this is due to the process of un-compressing files and then copying them.


SD Technology for the Operating System???


I probably had the same reservations as everyone else about the SD technology. In addition to the considerations for speed, and the limited size, I was also worried that the limited number of reads/writes would severely limit the lifetime of the SD card.

HP has a great (short) document on this subject. In addition to covering the simple process of setting up the hardware, this doc addresses why SD cards are a legitimate medium for this and how to configure: http://h20195.www2.hpe.com/V2/getpdf.aspx/4AA4-9872ENW.pdf?ver=1.0. That document's a quick read and well worth the time since it provides a really good understanding of SD technology.

Basically, the message is: You want SLC - Single Layer Cell - technology for performance and endurance. But you want MLC - Multi-Layer Cell - technology for density and cost effectiveness. So of course HP came up with a combo they call SMLC technology.

The HP cards sold for this application top out at 32GB. I wanted more (of course), so I ordered a Kingston MicroSD card, class 10 for speed, and SDXC for high capacity. Of course that document told me that the non-HP product wouldn't work, but they say that about hard drives as well. So I thought I'd give it a try.

Result: The Intelligent Provisioning option for MicroSD simply doesn't show up. It's not grayed out. It's gone.

Lesson One: You must use the HP MicroSD card.

OK, lesson learned. I ordered the HP branded 32GB MicroSD "kit." It's official title is 32GB microSDHC Enterprise Mainstream Flash Media Kit and the part number is 700139-B21. It was about $130 from my distributor, including shipping.

The stats on this tiny little card are very impressive. Projected block writes = 13,107,200,000. That's 13 with a B for billion. Includes very impressive Error Correcting Code (ECC) and you can monitor wear via the HP Integrated Lights Out (ILO) card. That means you can log into the server via ILO and get underneath the operating system to verify wear on the SD card.

In addition to the obvious benefits such as cooling and space, SD-based operating system drives are very fast. That's not the old SD reputation. But the new XC class 10 cards are significantly faster than any spinning disc. Not sure how they compare to SSD cards on a SATA interface. As a rule, HP estimates that the O.S. drive will experience about 70% reads vs. 30% writes during normal operations. And since reading will be significantly faster than writing, performance should be excellent.

The biggest concern I had was the Windows Swap file on SD. My plan was to disable it and put a Swap file on the data drive. The document noted above says that you can basically disable swapping altogether if you have a solid state O.S. drive and enough system RAM.

Again, remember that most of this is just for fun since the server will have essentially near-zero load all the time.


Loading the O.S. onto MicroSD


First, let me note that I did a full, normal, regular Server 2012 R2 Essentials setup on the mirrored SATA drives. Worked perfectly. Textbook. Exactly as expected. So even though I've done this install several times, I had a fresh new experience on fresh new hardware.

The installation to SD was not as easy and trouble-free.

I added the HP branded 32 GB SD card even though it seems really small. I removed all hard drives. I booted up the machine and hit F10 to run the HP Intelligent Provisioning. See screen shots below.

There's not much to choose from on the first screen. I do note that the MicroSD card is selected as the only option for installing the operating system. Of course there were no hard drives present.




On the second screen, OS Selection, I started with the recommended selection, but on the next screen it would not install to the SD card because it wasn't large enough. So I went back. Selected "Customized" install and then was able to configure the third screen.

On the third screen, OS Information, you need to pick the specific operating system you want to install. I picked Server Essentials. When I checked the box to have no swap file, the size indicator of the card went to zero and I could not install the O.S. So I went back and refreshed the page.

When I did not check the box, I got a warning the the partition was below the 180GB recommended for the volume. I said to continue and it did.

(Note: After several installs during troubleshooting, I noted that it ALWAYS disables the swap file when you install to the MicroSD card.)

After lots of file loading and lights flashing, I was asked for the product key and then it booted into Windows.

I noted that the O.S. was taking up about 21 out of 29.5 GB usable on the MicroSD card. That's less than on the spinning hard drives.


Trouble: The Server Essentials Configuration


So far everything had gone well. But the Server Essentials "interview" configuration stopped at 25%. I retried and it stopped at 29%.

I reviewed the requirements for Server Essentials. Among other things I tried to re-run the configuration after disabling the second NIC and installing one of the 1TB drives so the system could store stuff there if need be. Of course there's a reboot after each of these.

Ran Server Manager. It was not able to add any roles or features. I verified that all "automatic" services were started.

I reviewed the few logs that were created (setupact.log, dism.log, and smh_installer.log). No errors. No help.

I thought that forcing the server into a domain would at least allow me to add some roles. So I opened PowerShell and ran

Install -ADDSForest -DomainName xxx.local

I had to create the \Windows\NTDS directory manually, but I was able to get the domain created. I verified that DNS was running, and machine was pointing to itself for DNS.

After a reboot, I was able to run Server Manager and add several roles. BUT the Server Essentials Configuration keeps running . . . and stopping at 29%!

At this point I did some research on Technet and the never-ending Microsoft boards where 99.9% of all discussion amounts to, "I'm having the same problem. Hasn't anyone fixed this yet?"

I have installed Server Essentials many times and never seen this error. So I tried the few suggestions that were given over and over. Some suggestions worked for some people, but not for others. The big recommendation from Microsoft folks is to add the user ServerAdmin$ to the "Log on as a Service" group policy. But like many people, this user did not exist for me.

One person said that all he did was double the RAM and double the hard disk space and it worked for him. That led me to believe that we really have a resource issue, not a permissions issue.

I decided to take another approach:


A Solution of Sorts: The "Essentials" Role within Server 2012 Standard


Given my problem with Roles and Features, and particularly with the Server Essentials Configuration program, I decided to attack the problem from a different angle.

First, I'll install regular Server 2012 R2. No wizards involved.
Second, I'll see if I can add the roles I need, including the Essentials Role

[ Note: The license for Server Standard is more expensive than for Server Essentials. But Server Standard is a VERY light installation with no roles assigned. One of the roles available is the Essentials Role. It allows all the cool management and features of Server Essentials without the licensing limitations.

Since I have an Action Pack subscription, I can install the full Server Standard for my testing purposes. But if it's the long-term solution, I will either have buy the full license or go back to the successful installation I created on the regular hard drives. ]


I'll skip through some troubleshooting here to bring you the answer that worked.

I physically removed the MicroSD card, put it in another machine, and formatted it. Then I put it back in the MicroServer and ran Intelligent Provisioning. I added an empty 1TB drive even though I was installing the OS to the MicroSD.

This time, the same screen I had before warned me that the 30GB partition was smaller than the 40GB recommended (last time it said 180GB recommended). Again, the total setup time was about an hour and forty-five minutes before I got a logon screen.

At this point, I had 20.7GB free on the "C" drive out of 29.5. That means a mere 8.8 GB were used for the operating system. Yay. The system had configured itself to have no swap file and no memory dump on blue screen.

Now I had a working version of Server 2012 R2 Standard.

Server Manager ran fine. I added the DNS Service without any problems. Added the Server Essentials role without any problems.

Oh No!!!

When the Server Essentials Configuration came up, it stopped at 29%. So while I had a much lighter installation, I had the same issues.

At this point, I had no ServerAdmin$ user to add to the "Log on as a Service" policy. I also had no domain and the machine name was a random self-assigned name. So the interview hadn't gotten to the rename scripts yet.

But since I had the Server Manager available and working, I added AD Services and promoted the server to a Domain Controller. I had to create a forest and local domain.

After a reboot, I had the new machine name, the new domain, and no errors in Server Manager.

The Server Essential Configuration ran automatically because it had not finished. This time it zipped passed 29% and finished very quickly. The system was now running Server 2012 R2 Standard in Server Essentials Role.

Hoorah.

(Except I don't want to spend that much on a license for seven users. So now I have to do it all over again with Server Essentials.)

Lesson Two: There appears to be an issue with lack of resources (primarily hard drive space) and a smooth installation of Server Essentials to the MicroSD card. You can bootstrap the installation by loading the OS, loading DNS, adding the AD/DS role, and then completing the Server Essentials Configuration.


One Last Time!!!


OK. So I nuked all that and started over one more time to install Server Essentials. Here are the basic steps:

1. Intelligent Provisioning. See Screen shots above.

2. Setup works fine. Server Essentials Configuration runs automatically and fails at 29%.

3. Added the DNS Role in Server Manager. Verified configuration.

4. Added Directory Services Role. Reboot.

5. Promoted the Server to Domain Controller. Renamed as needed. Reboot.

6. Ran Server Essentials Configuration. Successful.

Side note: During Directory Services setup, I created the three NTDS folders on the "D" drive (SATA) rather than let them be created on the MicroSD card ("C").


Overall, with 1.5-2 hour installations and lots of troubleshooting, the whole adventure took about twelve hours. That's a BIG example of why I use and recommend a Troubleshooting and Repair Log with every server build. Basically, you should keep really good notes and refer back to them.

The six successful steps above took about 2.5 hours total. THAT's what I want to replicate the next time I do this - not the 12 hours of troubleshooting and trial-and-error. That also gives me a very solid estimate of how much time I need to quote in order to be/stay profitable.


Performance Thoughts


As you can imagine, I'm concerned about overall performance.

As an interactive user on the server, it performs very well. Even connecting through RDP is nice and fast. Boot time is almost identical between SATA RAID-1 drives and MicroSD card. A few seconds is all the speed you get. But it's a pretty fast bootup anyway.

I ran a few performance tests on both the SATA-based installation and the Micro-SD installation. Copying about 7GB of files from one point to another on the "D" drive (where data would be stored), the MicroSD installation was a great performer.

Both installations finished the chore in about a minute, although the MicroSD was a bit faster. More importantly, the SD installation used an average of five percent processor time during the file copy operation, with a few peaks at 30%. The SATA installation used a minimum of 20% processor and had a few peaks in the 80-95% range.

Measuring Average Disk seconds per Transfer, the the SD installation was in the range of 0-5 one-hundredths of a second. The SATA installation was in the range of 5-70 one-hundreds of a second, with an average in the 20's.

Disk Queue Length for the SD installation was generally around 2-3 with eight peaks around 9-10 and eight troughs at zero. The SATA installation had an average Disk Queue Length of four with seven peaks around 9-10 and four troughs at one. It only reached zero at the end of the operation.

Both systems had very consist measures of total Available MB of Memory at about 5,000 (5GB). This was a nearly flat line for the SD installation and had one small dip in the SATA installation when the Disk Queue was high, as you would expect.

Of course I installed my standard LogicNow MAX RMM agent on the server. So it will be interesting to see how it compares to other servers in my dashboard.


Bottom Line: I am cautiously optimistic about this installation. I will be monitoring it very closely. And I've saved the two-drive mirror with Server Essentials installed, so I can literally just plug that in and go if need be.

Stay Tuned. I'm going to write another blog post about using the ILO (Integrated Lights Out) card to monitor the system performance and the SD card read/write history. If you haven't used an ILO card to monitor servers before, I think you'll find it interesting.

- - - - -

Sorry this is technical and off-topic for the business folks. But it was a fun project.

Questions and comments welcome.

15 comments:

  1. Comments / Questions:

    * It seems like a lot of labor cost involved along with added unnecessary complexity for a negligible performance gain on what was acknowledged at the outset as a very lightly used server. What's the justification?

    * With 4x1TB drives, why not RAID10 the array and partition beneath? You certainly don't need 1TB of OS and "tech" files, as the experiment of installing on a 32GB SD card shows. RAID10 would give you redundancy *and* read performance boost, with no additional write penalty. The justification I can see is the ability to simply yank the pair of data drives and drop in another box. That will come along with its own attendant permissions issues, so potentially no time savings over restoring from backup.

    Jon

    ReplyDelete
  2. Hey, Jon. Thanks for the comments. Now that I have a documented process, the install will be very easy going forward. The justification in this case was to see how well it works.

    Gotta Say: This would have been trouble free with a 128GB or 256GB MicroSD card.

    Going forward, having the OS on the SD card frees up the hard drives to do whatever we want.

    I haven't decided on the final drive configuration, but it will probably to have the backup OS on the mirrored pair along with the !Tech directory. Yes that leaves massive amounts of unused disc space, but there's no point in buying smaller hard drives just because.

    A third drive is basically stand-alone. It's used to bring a copy of the data down from the cloud. Remember, there's no "live" data on this server. It just backs up the cloud storage so we have a local copy. The fourth drive is unused and will happily wait to see if it's ever needed.

    I don't mirror those last two drives because I want to occasionally take one out of service, box it up, and put it on a shelf as permanent offsite backup. Being able to just slide drives in and out without messing with a mirror is very handy. Sometimes the low tech solution is all you need.

    ReplyDelete
  3. Are there larger Hp cards available? Maybe a 128 gb if availabke would solve the resouece problem.Or try a mirror of 64gb.

    Also, what about Microsofts advice to keep two dc's in a domain. With a microserver a customer gets just one.

    ReplyDelete
    Replies
    1. also, a decent backup is also necessary for systemstate. Azure is backuped automatically in the Azure cloud althouh i am not sure whats with the version control. Am not sure if that is available

      Delete
  4. Believe me, Andre, if there was a larger card available I would have bought it. I was surprised that my options were 8 GB and 32 GB. I *did* note that the SD card is visible from an OS on the hard drives. That leads me to believe I could do an installation to my 128GB MicroSD card if I didn't use the Intelligent Provisioning. But that would mean I'd have to go find all the drivers from HP. That might take a lot of time.

    Microsoft has been making exceptions to the multiple-DC rule since at least 1998. Small Business Server most commonly had no second DC. Same with most installations of Server Essentials, Foundation, Home Server, and Multi-Point. Multiple DCs are necessary in larger environments because machines and networks are constantly busy or in maintenance mode. In a very small office, most people can log on with stored credentials if the server's unavailable for some period of time. But these machines are so rock solid, the server's always available.

    ReplyDelete
  5. System State is stored on the SATA drive in this case and easily backed up to cloud. From there, the cloud storage is backed up on another local drive. So we're good to go.

    ReplyDelete
    Replies
    1. Perfect. Are ye using Windows server backup for the systemstate or a 3th party tool? Think the combination of a micro server and Azure is a golden shot.

      Delete
  6. Thanks for the excellent post Karl. Looking forward to hearing about your ILO experience.

    ReplyDelete
  7. Hi Karl,

    Who do you use for cloud storage?

    ReplyDelete
  8. I have used Azure, AWS, and Rackspace. Currently I have a couple clients on Rackspace attached via JungleDisk. But everyone else (including my own company) are just on JungleDisk. I like that product because I can map a drive on every machine and no matter where they are (even virtual), the drive X: is the drive X: and it looks very familiar to them.

    ReplyDelete
  9. Hallo Karl,

    What do you mean with the Essentials Role, are you talking about the Server Essentials Dashboard or re you talking about something completely different?

    ReplyDelete
  10. If you buy Server 2012 Standard, one of the "roles" you can install is the Essentials Role. It gives you the features of Server 2012 Essentials without the license limitations. Instead, you use appropriate Server Standard licensing. This includes CALs for users, CALs for remote access, etc. Server Essentials comes with 25 licenses and you can't buy more. Server Standard allows that "role" and you can buy as many licenses as you wish.

    There's some info here:
    https://blogs.technet.microsoft.com/sbs/2013/09/03/understanding-licensing-for-windows-server-2012-r2-essentials-and-the-windows-server-essentials-experience-role/

    and juicy good information here:
    http://download.microsoft.com/download/F/3/9/F39124F7-0177-463C-8A08-582463F96C9D/Windows_Server_2012_R2_Licensing_Datasheet.pdf

    ReplyDelete
  11. Have you or do plan to do a comparison using SSD cards?

    ReplyDelete
    Replies
    1. Another option to fix your capacity issues is to use multiple SD cards and raid them using either raid 0 or raid 5. I only mention those teo as it would allow you to use the least number of SD cards. Optimal configuration would be one big raid 10. (See article: https://community.spiceworks.com/topic/262196-one-big-raid-10-the-new-standard-in-server-storage?page=1)

      Delete
    2. Another option to fix your capacity issues is to use multiple SD cards and raid them using either raid 0 or raid 5. I only mention those teo as it would allow you to use the least number of SD cards. Optimal configuration would be one big raid 10. (See article: https://community.spiceworks.com/topic/262196-one-big-raid-10-the-new-standard-in-server-storage?page=1)

      Delete

Feedback Welcome

Please note, however, that spam will be deleted, as will abusive posts.

Disagreements welcome!