Nehalem Performance Optimization (BIOS Edition)

Long-awaited recommendations on the BIOS options for IBM's Nehalem offerings.  I can't speak to their Westmere technology, having been unable to get my hands on it yet.  I did a considerable amount of research on this - at the end of the day, this has been an effort to aggregate IBM's standards, rather than me attempting to prove/disprove their statements.  I'm not sure I can take on IBM like that :-)


You can find the details in the link below, not a virus I promise:
(Update: MediaFire's link expired somehow.  Working with their techies to resolve)
Nehalem Recommendations (excel)


Official IBM notes on Performance:
http://www.sbsmn.org/Optimizing%20Nehalem%20server%20memory%20XSW03025USEN.pdf
ftp://ftp.software.ibm.com/systems/support/system_x/dx360m2_dx360m3-cmos-settings-v1.2.txt


VMware notes on Turbo:
http://communities.vmware.com/thread/261540

Optimizing uEFI boot speed (recommendations forthcoming):

ftp://ftp.software.ibm.com/systems/support/system_x/dx360m2_dx360m3-cmos-settings-v1.2.txt

http://download.intel.com/design/intarch/papers/322253.pdf

Daily VMware Slowness

I was in a meeting Friday when someone mentioned that our VM's (of which we have hundreds spread across 4 datacenters and 12 ESX servers) were experiencing degraded performance every day around lunchtime.  I set up PerfMon tasks to watch the CPU utilization and I/O writes and reads of the various processes on  two different VM's.  What I found was: nothing.  There was no data to support the slowness. We then started experiencing slowness at 5pm: I analyzed that data, and found this:

Virtual Machine 1, 4:50pm to 6pm: 

Virtual Machine 2, 4:50pm to 6pm:

If I were a detective, I would probably call that a clue.  Turns out our engineer who owns McAfee for us had recently made some changes, which I haven't been able to dig into, that made all of our VM's update their local virus definitions simultaneously.  When there was conversation about that possibly causing performance issues (a theory that was disparaged by some), he moved it to 5pm.  Mystery solved.  We're going to work to stagger the DAT updates, so not all our VMware CPU and LUN I/O resources are pegged simultaneously.

Couple notes on perfmon:

- Kick logs out to CSV or TSV.  Either can be imported into excel, which is much easier and more versatile to work with than PerfMon's own data analyzer.

- Make sure to select "All instances" so that each process gets its own set of data points.

- I took readings every 10 seconds and saw no real performance impact over the course of the day. 

- I highly recommend deleting "Idle" and "Total" processes first thing when analyzing the data you've collected, or better yet not recording data from them at all: they're just noise.  

- I recommend setting a 1-4 hour time frame for the data collection so the data is broken out in multiple files: nothing is more hassle than trying to work with at 200MB text file.  Trust me, learned that lesson a year ago (thanks HP EVAStats!).  

Duplicate SID's

Interesting challenge to conventional wisdom..."I became convinced that machine SID duplication – having multiple computers with the same machine SID – doesn't pose any problem, security or otherwise. " - Mark Russinovich, of SysInternals fame.  In case you haven't heard of Mark, he's pretty much a legend.  Against what I've been taught, MS has concluded that duplicate local SID's within a domain is perfectly OK.  Domain SID's, on the other hand, need to be unique.


Gotta constantly re-evaluate commonly held truths I guess!


However, I have seen an issue: if the server you're joining to the domain has the same local SID as the DC, you will see some funky results.  The domain trust will not function correctly and you won't be able to log onto the member using domain accounts.


http://blogs.technet.com/b/markrussinovich/archive/2009/11/03/3291024.aspx
http://technet.microsoft.com/en-us/sysinternals/bb897418.aspx

Working with IBM Scaled x460's

IBM offers the ability to stack two or more x460's together to make the resources of all the servers (RAM, CPU, etc) available to the Primary server.  This is some pretty old hardware we're talking about here though, since then much better solutions have been developed.

But if you find yourself having to deal with one of these, here's some information I learned recently while troubleshooting this dinosaur of a machine:

1.       IBM says average cost to repair at this time is $2661. 

2.       These servers are not clustered (implies redundancy): they are scaled (implies increased performance capacity).

3.       When the two servers are properly working together, the state is called “merged.”  The primary OS will show the total RAM of both boxes, minus overhead. 

4.       When the two servers are not merged properly, the primary OS will show less than half the total RAM of the two boxes.  This is a good test for whether they are merged or not.

5.       When the secondary is merged properly, it will display a black screen to the effect of “This server is merged, please view primary server.”

6.       Each server has an independent BIOS.

7.       Both RSA’s should be available.  The RSA’s are also merged in some sense, I wasn’t able to look into this very much. 

8.       You obviously want to avoid powering down the Secondary before the Primary – this would be akin to yanking half the sticks of RAM out of a live machine.

9.       Removing the power supplies and putting them back in helps clear spurious errors and reset the machine.

10.    I recommend against ever touching, looking at, or thinking about this equipment.

To shut down these servers:

1.       Shutting down the Primary via the OS will automatically shut down the secondary.  Wait for them both to blink power lights.

2.       I have seen this hang before, at a blank grey screen.  If it does: manually power down the Primary, and then the Secondary. 

To start up these servers:

1.       There is a “latching” mechanism here:

a.       KVM the Secondary.

b.      Start up the Secondary first.

c.       When the Secondary says “Waiting for primary” at the blue IBM screen (1 minute or so), start up the Primary.  If you wait too long, the Secondary will give up and begin a “failure to boot” loop.

d.      The Primary will display a "initializing system memory, please wait" screen for several minutes.

e.      The Primary will then display “Initializing PCI devices.”

f.       The Primary will then display “Searching for Secondary server.”

                                                               i.      If it is not able to find the Secondary server, the Primary may automatically shut down.  If you start it again and it is still unable to find the Secondary, it will boot to the OS unmerged.

                                                             ii.      If you are attempting to get into the primary’s BIOS, press F1 shortly after the Primary displays that it was unable to merge.  It will acknowledge your input and boot to setup.

                                                            iii.      If it is able to find the Secondary, it will indicate its merging attempt was successful and then boot properly.

Nehalem Performance Optimization

Found a really well put together IBM document on Nehalem performance configuration*, highlights and link below:

RAM Configuration
1. Identical configuration for each memory channel (3 channels per Proc), and same speed RAM across the board. Within a memory channel you can mix sizes, but each memory channel must have an identical configuration. For best performance, each channel would have a single DIMM.

2. If a dual Proc machine only has RAM in one bank, there is a significant performance hit for the second Proc to access the “Remote RAM.”

3. The optimal configurations for dual Proc Nehalem servers are

a. 6GB (6x1GB)
b. 12GB (6x2GB)
c. 18GB (6x2GB, 6x1GB)
d. 24GB (6x4GB or 12x2GB)
e. 48GB (6x8GB or 12x4GB)
f. 72GB (6x4GB, 6x8GB)
g. 96GB (6x16GB or 12x8GB)

4. For single Proc Database servers, the best configs would be
a. 3GB ( 3x1GB sticks)
b. 6GB (3x2GB sticks)
c. 9GB (3x2GB, 3x1GB sticks)
d. 12GB (3x4GB sticks)
e. 24GB (6x4GB sticks)
f. 36GB (3x8GB, 3x4GB sticks)
g. 48GB (6x8GB sticks)

5. In general, avoid populating DIMM slots 1 (the first slot), 4, 9, or 12. Doing so unbalances the memory channels and decreases performance, so it is preferable to increase the DIMM size across the board rather than add more sticks.

6. Populate the furthest slots first, in this order: (3, 6, 8) and then (2, 5, 7)

7. Always use dual rank memory if available (e.g. 2Rx4, 2Rx8, etc).

BIOS Settings (more analysis to come):

Setting	Maximum Performance Setting per IBM
Memory Speed	Auto
Memory Channel Mode	Independent
Socket Interleaving	NUMA
Patrol Scrubbing	Disabled
Demand Scrubbing	Enabled
C-States	Enabled
Turbo Mode	Enabled
Thermal Mode	Performance
Hyper Threading	Dependent upon App

*This is for x3650 M2/x3550 M2&M3/dx360 M2. HS22’s have other requirements.

Optimizing Nehalem Performance (Dead link?)

http://www.sbsmn.org/Optimizing%20Nehalem%20server%20memory%20XSW03025USEN.pdf

IBM ASU

In my research into the uEFI settings on IBM's Nehalem offerings (info soon), I ran into a bit of a gem: IBM Advanced Settings Utility. It's a command line utility that can script setting BIOS/RSA settings via the RSA. Trust me, IBM's not paying me for this publicity to my legions of readers, but I'm a pretty big fan of this thing so far. I sifted through IBM's typically lacking documentation* and translated it into usable English for you.

1. ASU can edit select settings on the uEFI/BIOS and IMM/RSA. Surprisingly robust options on a x3650m2, I'm not sure about RSA's or RSA2's yet.
2. Some settings are no reboot required. I didn't have time to dig into this.
2. Search IBM.com for ASU to download it, their links change too frequently to post an URL here. The architecture (64 vs 32 bit) of the exe refers to the server you are using this tool on as opposed to the target server being configured. These two are not always the same thing, as it works over the network too.

4. Double clicking the downloaded exe extracts ASU.exe and supporting files.
5. Two methods of configuring a server using ASU:
1. Run on local server. It will connect to the IMM via OS integrated drivers: "USB in-band interface." Basically a virtual NIC in Windows.
1. CMD: Asu.exe batch c:\admin\uEFI.log
2. Run on a hop server. It will connect to the IMM over network at the IP you specify (!!!!).
1. CMD: Asu.exe batch c:\admin\uEFI.log options --host
2. e.g. Asu.exe batch c:\admin\uEFI.log options --host 10.10.10.10
3. If you go over ethernet, you will need to supply credentials. Do so using the following syntax:

CMD: Asu.exe batch c:\admin\uEFI.log options --host --user --password

e.g. Asu.exe batch c:\admin\uEFI.log options --host 10.10.10.10 --user batman --password robin

6. Developing your batch file
1. Cmd: Asu.exe Show > c:\admin\uEFI.log
2. Edit
1. Remove top lines
2. Syntax: set "setting value"
1. e.g. set IMM.IMMInfo_Location "1234 Batcave Drive"
2. Use this line to see your options:

asu.exe showvalues C:\admin\Values.log

You may need enable the "Allow commands on USB interface" setting if your script is connecting to the IMM, and then failing to properly execute.

"Note: The ASU works with a disabled USB in-band interface if an IPMI device driver is installed."

ASU Guide

http://publib.boulder.ibm.com/infocenter/toolsctr/v1r0/topic/asu/asu_guide.pdf

Notes on the "Allow commands on USB interface" option.

http://publib.boulder.ibm.com/infocenter/bladectr/documentation/index.jsp?topic=/com.ibm.bladecenter.hs22.doc/dw1iu_t_lan_over_usb_resolving_conflicts.html

*Most tech writers are far too focused on being complete and accurate to remember that documentation is supposed to be helpful. The efficiency of the reader seems to rarely be taken into consideration, so most of it ends up looking like a yahoo search: tons of data you don't need and one thing you do need, written in a way that will make perfect sense once you already know it. And that's why I use Google :-)

Visual Studio 2010 Silent Install

Needed to script out installing VS2010, wanted to throw a couple of the gotchas out into the interwebs for others. Basically, I wrote three batch files that used .reg files to queue each other up and used the setup's native unattend capability to install on the D:\ drive and only install selected components (we didn't install SQL or Sharepoint components).

Notes on my implementation:
- The setup run via silent install and automatic reboots.

- It uses the registry to queue up more installs after each reboot, so you can just sit back.

- The setup uses an unattend setup.ini file to only install the components required by our development teams per their recommendation.

- The install.bat warns that several reboots are required and pauses to let the user quit, as well as sets a timer on the first reboot. After that, the script just runs.

- The c:\admin\ path is vital, so be sure to extract it there.

- About a 35 minute install, it puts 2GB on the D:\ drive and 4GB on the C:\ drive.

- The initial zip file is 2.4GB, and unzipped it’s still 2.4GB.

Make sure you have 10GB or so free on C:\ before you start this process.

You can get away with 8GB free on C:\ if you delete the .zip file after you extract it.

- Security scanning only picked up 1-2 vulnerabilities after install, but that may change over time so I recommend scanning it post-install.

- This DOES install .NET 4.0.


Scripts below:

Install.bat
echo "This install requires a reboot. Please press enter if you'd like to continue, else quit."
pause
regedit /s install.reg
VS2010setup.exe /q /norestart /unattendfile setup.ini
echo "Complete. If you don't want to reboot the server, use shutdown -a"
pause
shutdown -r -t 60 -c "The server is restarting per Visual Studio 2010 Install requirements."

Install.reg
Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\RunOnce\]
"VS2010" = "C:\\admin\\vs_2010\\Setup\\install_stage_2.bat"

Install_Stage_2.bat
regedit /s install2.reg
c:\admin\vs_2010\setup\VS2010setup.exe /q /norestart /unattendfile c:\admin\vs_2010\setup\setup.ini
echo "Complete. Rebooting"
shutdown -r -t 10 -c "The server is restarting a second time per Visual Studio 2010 Install requirements."


Install2.reg
Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\RunOnce\]
"VS2010" = "C:\\admin\\vs_2010\\Setup\\install_stage_3.bat"

Install_Stage_3.bat
c:\admin\vs_2010\setup\VS2010setup.exe /q /norestart /unattendfile c:\admin\vs_2010\setup\setup.ini
echo "Complete. Rebooting"
shutdown -r -t 10 -c "The server is restarting the last time per Visual Studio 2010 Install requirements."


Gotchas:
- The slash is an escape character when you are working in quotes, so double slash it.
- Watch out using setup.exe as the runonce target. I renamed the setup file to avoid a possible issue with a native Windows file named setup.exe.
- The setup.exe in the \setup\ folder is the only one with unattend functionality. Don't confuse it with the setup.exe in the root folder.
- Whatever happens during manual install is not necessarily what you'll see once you automate it. I was able to install in it only one reboot manually, but was unable to automate it without two reboots. Three is ideal.

- SP2 is required for server 2003 installs.



Sources:
http://aka-community.symantec.com/connect/forums/installingscripting-out-visual-studio-pro-2008-visual-studio-pro-2010 - Other people's scripting


http://techsupt.winbatch.com/TS/T000001029F22.html - Runonce discussion


http://msdn.microsoft.com/en-us/library/aa376977(VS.85).aspx - Official MS run once documentation