Performance Case

Just a performance case I worked recently.  These kinds of things can be instructive for people with similar problems, or those who learn by playing along at home.  Here's my email, sorry for the copy paste without context.

 Please take a look at the email chain below as the starting point for this conversation.  At this point, we’ve reviewed four sets of performance data gathered over the last two months and have closely correlated a spike in large-IOP-size traffic to our latency spikes.  This spike is both in number and size of IOPS, exceeding 32,000 IOPS for 15+ minutes at a time.  There is no single volume driving the traffic, as it appears to be increasing dramatically across the board.  

Here is a summary of the performance data from Thursday 4/3, please note the IOP ramp up and associated latency:

Start Time	CPU Busy	NFS Op/s	Read Op/s	Read Lat (ms)	Write Op/s	Write Lat (ms)	Net Sent (MB/s)	Net Recv (MB/s)
9:53p	52	7,127	1,256	2.32	5,731	0.63	27	50
9:57p	67	13,637	5,396	19.96	8,076	110.09	145	81
10:05p	99	31,119	17,272	22.91	13,697	341.18	371	205
10:13p	99	32,311	22,519	8.4	9,739	229.45	621	200
10:24p	99	23183	12,819	7.21	10,261	143.16	348	260

And here is the data from Thursday 2/13.

Period	CPU Busy	NFS Op/s	Read Op/s	Read Lat (ms)	Write Op/s	Write Lat (ms)	Net Sent (MB/s)	Net Recv (MB/s)
9:09p	78	12,574	5,562	3.19	6,926	1.05	200	141
9:18p	98	27,771	17,291	4.7	10,347	24.29	571	312
9:29p	98	33,050	21,460	9.11	11,507	125.85	650	352
9:38p	98	34,149	22,813	11.28	11,216	530.9	647	345

One thing that stands out in the data is a large, sudden increase in 64k+ IOPS.  I’ve adjusted the table to include a row for 64k IOPS and have highlighted the relevant statistic.

FAS6280 Maximum IOPS
	Read/Write Mix
Avg IO Size	100/0	75/25	50/50	25/75	0/100
64k	61,000	43,000	32,000	26,000	22,000
32k	68,000	48,000	36,500	30,000	25,000
24k	74,000	51,000	39,500	31,500	27,000
16k	80,000	56,500	43,000	36,500	30,500
8k	85,000	63,000	50,000	41,500	36,000
4k	90,000	66,000	54,000	45,000	40,000

  The workload mix appears fine for most of the day but experiences large-IOP-size peaks that are outside our guidelines and cause some pain (38,000 IOPS 1pm 4/5,  45,000 IOPS 11pm 4/4, 37,000 IOPS 11am 4/3) .   I’d also make mention that ~10% of IO to this system is misaligned, which hinders us from achieving maximum performance ROI.  Lastly, this system is achieving 65-85% dedupe ratios, which is fantastic space conservation but adds to the overall workload.

  As discussed yesterday, here are our options:

·         Short term steps:

o   Stagger workloads (Symantec, et al)

o   Disable aggr snapshots (done)

o   Stagger dedupe

o   Case open on daytime snapshot correlated latency (done)

o   Update Data ONTAP

·         Long term solutions:

o   Add new disk to passive controller and balance workload or

o   Shift workload to a different or new HA pair

Reading LUN Stats -o

netapp1> lun stats -o
    /vol/UCvol/UClun  (98 days, 0 hours, 46 minutes, 51 seconds)

Read(kbytes)	Write(kbytes)	ReadOps	WriteOps	OtherOps	QFulls	PartnerOps	PartnerKBytes
15940415	640772	30487379	8734	9767	0	30442488	15496708

A few things about this are not immediately intuitive.  First, statistics labeled "Partner" are indirect IO, passed through the cluster interconnect.  This means both heads have to process the information, which is adds up to considerable overhead in terms of work and latency.  

Second, the statistics not labeled  "Partner" are totals: they include both the direct and indirect IOPS.  At first glance you'd see that ReadOps is a very close number (.1% off) to PartnerOps, which would indicate that the traffic is being load balanced across indirect paths.  

But that would be counting the same Ops twice.  If you add up ReadOps, WriteOps, and OtherOps, that's the total number of direct and indirect Ops.  If you then subtract PartnerOps, that resulting number (63392 Ops) is the number of direct Ops.  That's .2% direct IOPS, which indicates a serious pathing configuration issue.

NetApp Training Brain Dump: Problem and Solution

In SAN management, you're always working to meet several requirements depending on the nature of the data.  You may need to just add more space, in which case throwing more disks at the problem might be the best solution.  Other issues can be more complex to address, such as increasing performance working under a strict budget.  Below is a detailed progression of options as we search for the best solution.

Potential Options:
1) If you add more disks, the workload might be spread over more spindles and performance may increase.
2) What speed/size disk are you using?  Do you have well tiered data management solutions?
3) It might be smart to upgrade the RAM/Cache/Controllers in your SAN storage unit.
4) Utilization management: what applications are driving this change in performance requirements?  Can your SQL queries be more efficiently written?  Or maybe backup processes are poorly designed?

5) If you have all your high-transaction data on the same disks, you may want to spread that more evenly with your low-IO data.  This may require better software to be able to manage this solution (NetApp V-Series).

6) If a small amount of your data is responsibly for a disproportionate amount of writes, you may want to implement a Storage Acceleration solution in front of slower disks, giving you the performance you need at a lower cost.
7) If high read access is required, NetApp offers cache upgrades of high performance Flash memory to reduce the impact on your disks.


NetApp V-Series: Somewhat similar to HP SVSP, you place these devices between the clients of storage (servers, etc) and providers of storage (IBM XIV, etc).  This allows you to utilize NetApp's awesome software w/ your existing SAN technology.

NetApp SA (Storage Acceleration): Again, place these between your clients and providers of storage.  The device determines what data to hold in local, very fast memory, and what to relegate back to the actual SAN disk.  Pretty much the same relationship as your laptop's memory vs hard drive.  Increases performance.

Flash Cache: Implement this in your SAN storage unit for an inexpensive solution to drastically boost read operations!  Marketing says the performance of this technology is "comparable to SSD."

EVAPerf Statistical Limitations

Here's a big issue I wrestled with over the last few weeks:

EVAPerf occassionally hiccups, kicking out a single data point that claims a 2Gb host port has 30Gb/s throughput, or that a single diskgroup has 200GB/s being written to it. Sometimes these are clear overflow numbers (214748.3647 showed up repeatedly) and sometimes they are just absurdly high, though unique and precise. HP recommends using 95th percentile to statistically analyze the performance of your EVA's, and these super-high numbers skew our statistics to the point of being worthless.

My solution: create duplicate, empty SQL tables and daily screen the data to move any data points over thresholds I set to those duplicate tables, where they'd be out of the scope of my automated reporting. The trouble is, where's the threshold?

After a few weeks of emailing back and forth with HP, it became clear that they weren't interested in giving any definite answer or OK'ing my calculations. Part of the reason for that is that they have no real insight into how much actual IO CAN occur in a disk group because of all the behind-the-scenes leveling, parity calculation, and striping that occurs on top of the server-generated IO.

On top of that, block size varies so widely that the IO capacity calculations that they ARE able to do give you no real concept of the throughput capacity of your hardware. For example, let's say their PerfMonkey tool said your disk configuration allowed for 6000 read and 4000 write IO/sec in your disk group. Theoretically, with block sizes of up to 64MB apiece, this means your throughput could be over 380GB/s. So we're without a solid mathematical recourse.

I settled on 50GB/s for disk groups and LUNs, and 10Gb/s for host ports (even though they're only 4Gb ports) after careful analysis of how that affected the data - ends up looking like about 40 data points per month would be moved using those as thresholds.

Working pretty nicely so far.

HP EVA fnames.conf

Our SAN environment has 120TB spread over 4 HP EVA's (3 8100's and a 4400). We've worked through numerous difficulties with these, not the least of which was the dreaded "Saturday morning slowness." Part of our efforts to combat this was to attempt to gain greater insight into where the IO was actually coming from - at the time, we simply didn't have the system in place to do this.

Our friends at HP provide EVAPerf, which kicks out CSV files with a deluge of data (200MB+ per day per array). In order to make sense of this, a good friend on the software engineering side was added as a resource: he did a great job of writing a loading program that took those CSV files and kicking them in SQL. Our company is looking into IP rights, it's that awesome.

Meanwhile, we discovered a few months ago that our CSV's were filled with WWN's, which are pretty cumbersome to work with. HP's solution is the friendly names file fnames.conf, which just allows the EVAPerf task on your server to replace the WWN's with readable, English names. So a fnames.conf file was set up to automatically recreate itself once a day.

Well, as time went on and we changed more and more, the data I was working with was increasingly filled with WWN's. Investigation Monday morning turned up this little gem:
"The fnames.conf file must reside in the directory in which HP Command View EVAPerf was installed. "

Well, whoever set this up had it creating the updated file in a c:\utilities subdirectory - we were working with a months-old fnames file, since EVAPerf was installed under c:\programs\hp... After updating this, we went from 15/150 correctly represented LUNS in one EVA's csv file to 135/150 LUNS showing up with real names, greatly simplifying our vdisk I/O statistics. Woot!

Extremely useful information on decoding EVAPerf data:
http://www.fcoe.ru/index.php?option=com_content&task=view&id=257&Itemid=46#addcomments

See here for more official info on the fnames/evaperf integration
http://h10032.www1.hp.com/ctg/Manual/c00605846.pdf