W32Time

Started a new job in Minneapolis about a month ago, and things are finally starting to get exciting. Found that there are a ton of issues for Windows Time here and spent a good deal of effort investigating them. This company does a lot of work that depends on accurate timestamps, so I had little trouble getting management to let me dig into the issues. This company is PCI certified, so there are tons of domains, and the time hierarchy goes a bit like this:

Internet time server

Infoblox DNS servers

Domain controllers in master domain (let's call this D1)

PDC in subdomain (SD2, SD3, etc)

DC in SD2

Member server in SD2

One issue I found right away was that the PDC in SFD2 was getting time from ntp.D1.com. This DNS entry only pointed at the PDC in D1, resulting in a non-redundant time stream, which caused issues when we took the D1 PDC down. I enabled round robin DNS for this by adding a second DNS record with the same name (ntp.D1.com) pointing at the DC's IP.

The next issue was that the D1 DC was not pointed at the D1 PDC (domain-based hierarchy). It instead was pointing at the infoblox server. I decided to ignore this for now.

Next, I put together a list of all the PDC's in every domain and verified NTP access for each to the DC's in D1. I also audited the registry on each and found another issue. The ",0x1" flag for special polling intervals that should be added to "ntp.D1.com" in the ntpserver reg entry was missing in many domains. I'm working through getting this added, but it's not a top priority.

Next, I did audits of the event logs of several domains (~5 servers, a PDC and a DC each) and determined that there was some correlation between reboots and the time failing. I ruled out patch issues or change issues since there was nothing in common. There also seemed to be a strange inverse correlation between the PDC not receiving time properly and the member servers receiving time properly. One worked, the other didn't, and visa versa.

Last, I built an entire test domain mimicking a normal domain, turned on logging, and started testing. I found that things got screwy quick, and eventually determined by parsing the packet reception in the logs that my SD2 PDC had stratum 15 somehow. I got logging turned on in D1and found that the infoblox servers had stratum 13!!! If you count the list above, the highest stratum we should be at is 6, and MS limits stratum to 15 exactly. Stratum 15 also is an indicator flag that the stratum count may not be correct, but in this case it was incremented correctly (after the infoblox, which is the root of our issues).

We're opening a case with infoblox to figure out what's going on with them. After we fix that, I'll know if I can consider this case closed.

EVAPerf Statistical Limitations

Here's a big issue I wrestled with over the last few weeks:

EVAPerf occassionally hiccups, kicking out a single data point that claims a 2Gb host port has 30Gb/s throughput, or that a single diskgroup has 200GB/s being written to it. Sometimes these are clear overflow numbers (214748.3647 showed up repeatedly) and sometimes they are just absurdly high, though unique and precise. HP recommends using 95th percentile to statistically analyze the performance of your EVA's, and these super-high numbers skew our statistics to the point of being worthless.

My solution: create duplicate, empty SQL tables and daily screen the data to move any data points over thresholds I set to those duplicate tables, where they'd be out of the scope of my automated reporting. The trouble is, where's the threshold?

After a few weeks of emailing back and forth with HP, it became clear that they weren't interested in giving any definite answer or OK'ing my calculations. Part of the reason for that is that they have no real insight into how much actual IO CAN occur in a disk group because of all the behind-the-scenes leveling, parity calculation, and striping that occurs on top of the server-generated IO.

On top of that, block size varies so widely that the IO capacity calculations that they ARE able to do give you no real concept of the throughput capacity of your hardware. For example, let's say their PerfMonkey tool said your disk configuration allowed for 6000 read and 4000 write IO/sec in your disk group. Theoretically, with block sizes of up to 64MB apiece, this means your throughput could be over 380GB/s. So we're without a solid mathematical recourse.

I settled on 50GB/s for disk groups and LUNs, and 10Gb/s for host ports (even though they're only 4Gb ports) after careful analysis of how that affected the data - ends up looking like about 40 data points per month would be moved using those as thresholds.

Working pretty nicely so far.

HP EVA fnames.conf

Our SAN environment has 120TB spread over 4 HP EVA's (3 8100's and a 4400). We've worked through numerous difficulties with these, not the least of which was the dreaded "Saturday morning slowness." Part of our efforts to combat this was to attempt to gain greater insight into where the IO was actually coming from - at the time, we simply didn't have the system in place to do this.

Our friends at HP provide EVAPerf, which kicks out CSV files with a deluge of data (200MB+ per day per array). In order to make sense of this, a good friend on the software engineering side was added as a resource: he did a great job of writing a loading program that took those CSV files and kicking them in SQL. Our company is looking into IP rights, it's that awesome.

Meanwhile, we discovered a few months ago that our CSV's were filled with WWN's, which are pretty cumbersome to work with. HP's solution is the friendly names file fnames.conf, which just allows the EVAPerf task on your server to replace the WWN's with readable, English names. So a fnames.conf file was set up to automatically recreate itself once a day.

Well, as time went on and we changed more and more, the data I was working with was increasingly filled with WWN's. Investigation Monday morning turned up this little gem:
"The fnames.conf file must reside in the directory in which HP Command View EVAPerf was installed. "

Well, whoever set this up had it creating the updated file in a c:\utilities subdirectory - we were working with a months-old fnames file, since EVAPerf was installed under c:\programs\hp... After updating this, we went from 15/150 correctly represented LUNS in one EVA's csv file to 135/150 LUNS showing up with real names, greatly simplifying our vdisk I/O statistics. Woot!

Extremely useful information on decoding EVAPerf data:
http://www.fcoe.ru/index.php?option=com_content&task=view&id=257&Itemid=46#addcomments

See here for more official info on the fnames/evaperf integration
http://h10032.www1.hp.com/ctg/Manual/c00605846.pdf

Resurrection!

I had hoped I'd be able to continue to post solutions to tough problems and general summaries of things I'd learned, but only got 5 posts in before other priorities took hold. A lighter school load this quarter will hopefully give me a better chance to keep this updated!

Group Policy and the Registry

Interesting - this job really keeps you on your toes. I thought GPOs weren't applying correctly - turns out I had the wrong definition of 'correctly'. I changed this:

Policies\Administrative Templates: Policy Definitions\Windows Components\Terminal Services\Remote Desktop Connection Client\Do not allow LPT port redirection

and expected that policy to set this:

HKLM\SYSTEM\CurrentControlSet\Control\Terminal Server\Winstations\RDP-TCP\fDisableLPT

But it didn't. It changed this:

HKLM\SOFTWARE\POLICIES\MICROSOFT\Windows NT\Terminal Services\fDisableLPT

which disagreed with the first one. Very confusing. Also, I noticed that the only GPO's not taking effect were under policies\administrative templates. But, according to page 524 of the GPO guide (which a very smart client admin friend of mine pointed me towards), all group policy registry changes are changed in only two registry paths:

HKLM\SOFTWARE\POLICIES
HKLM\SOFTWARE\MICROSOFT\WINDOWS NT\POLICIES

And these settings are POLICIES, which take priority over PREFERENCES. Apparently, changes made by a user on a per-machine basis are considered PREFERENCES.

The sum of all this is that GPO's take effect by creating a duplicate registry value, and that is the setting that is effected onto the machine.

Additional important notes:
1. GPResult and gpotool are cool tools. Work on 2008. Come with the 2003 resource kit.
2. If you go to the local group policy object editor mmc, you will find that your GPO's are not represented there. This is because the GPOEDIT does not dig into settings to check what is currently there - rather, it's a mask that is applied to the computer. Interesting, eh?

VM NIC Issue

One more thing - Had a 2008 vm that I created, installed the OS, but for some reason wouldn't connect to anything. Device Manager showed Ethernet Controller as being unrecognized, and said that it needed drivers. There were no connections showing in connection manager. Installing vmware tools did not solve it.

I shut down the server, downloaded the .vmx file to my laptop, edited it in wordpad, added this line:

ethernet0.virtualDev = "e1000"

above the other references to ethernet0.

Saved another copy of the old .vmx to be safe, uploaded the new copy, and everything was golden. My resource was this: http://communities.vmware.com/message/1078349

Time for jiu jitsu.

Carry on!

ESX Server Craziness

Had two server 2008 research VM's (which were deployed from a template) that wouldn't power on. After the initial attempt to power on, they both would not allow any other changes to be performed on them, because there was already a "task in progress," "fault null" or something to that end. Couldn't remove from inventory, couldn't delete, was able to migrate sometimes, couldn't power on.

Now, does VMware 2.5 support server 2008 templates? Nope. I knew that going into this. But somehow, I've gotten away with it before on other test servers, so it was worth a shot. Research suggested a solution would be to kill the process on the esx server that is hung trying to reboot the server. I tried all sorts of things, including stuff like

ps -ef grep
Kill -9 <>
rm -rdf machine_name

etc. What it came down to was that our vmware virtual center thought a task was being performed on these screwed up VMs, although the breakdown could have been with either the communication between the vcenter and the esx server, or the esx server with the guest OS.

Anyway, this was resolved not by leet linux commands, but by a good old restart, which was what I had been trying to avoid the whole time. First I tried a "service mgmt-vmware restart," which made all the guest OS's appear offline in vcenter. I have since learned that you should accompany this command with a "service vmware-vpxa restart."

The two services are tied together in some way - I didn't have a chance to research this today. When the servers did not come back online after 10+ minutes, I did a restart of the entire machine using

/sbin/reboot

You should preferably put the esx server in maintenance mode before doing this.
vimsh -n -e /hostsvc/maintenance_mode_enterbounce
vimsh -n -e /hostsvc/maintenance_mode_exit

I satisfied my engineering curiosity of "what if" by just bouncing it :-) Not to worry, it only hosts research vms, so this was as risk-free as you get. There are two esx servers in this cluster, and I rebooted B. Interestingly, A eventually went down (red mark on the esx server in vcenter). Then came back up. Then went down again. B stayed down. After 10+ minutes, both esx servers came up in very close chronological proximity, happy and refreshed.

And I was able to remove those two VM's from inventory, delete them, and free up space to continue my GPO testing :-D

Speaking of which, found today two issues with my GPO admx testing.
1. "Policy Definitions" is not the same thing as "PolicyDefinitions" in the sysvol. Only the second will be recognized by the DC.
2. You need SOME adml files to support the ADMX files. ADML's are only optional after you implement the first set. Else you'll get swarmed with errors when you open GPMC.

There's one of the problems I solved today.

Carry on!