Just a performance case I worked recently. These kinds of things can be instructive for people with similar problems, or those who learn by playing along at home. Here's my email, sorry for the copy paste without context.
Please take a look at the
email chain below as the starting point for this conversation. At this
point, we’ve reviewed four sets of performance data gathered over the last two
months and have closely correlated a spike in large-IOP-size traffic to our latency
spikes. This spike is both in number and size of IOPS, exceeding 32,000
IOPS for 15+ minutes at a time. There is no single volume driving the
traffic, as it appears to be increasing dramatically across the board.
Here is a summary of the
performance data from Thursday 4/3, please note the IOP ramp up and associated
latency:
|
Start Time
|
CPU Busy
|
NFS Op/s
|
Read Op/s
|
Read Lat (ms)
|
Write Op/s
|
Write Lat (ms)
|
Net Sent (MB/s)
|
Net Recv (MB/s)
|
|
9:53p
|
52
|
7,127
|
1,256
|
2.32
|
5,731
|
0.63
|
27
|
50
|
|
9:57p
|
67
|
13,637
|
5,396
|
19.96
|
8,076
|
110.09
|
145
|
81
|
|
10:05p
|
99
|
31,119
|
17,272
|
22.91
|
13,697
|
341.18
|
371
|
205
|
|
10:13p
|
99
|
32,311
|
22,519
|
8.4
|
9,739
|
229.45
|
621
|
200
|
|
10:24p
|
99
|
23183
|
12,819
|
7.21
|
10,261
|
143.16
|
348
|
260
|
And here is the data from
Thursday 2/13.
|
Period
|
CPU Busy
|
NFS Op/s
|
Read Op/s
|
Read Lat (ms)
|
Write Op/s
|
Write Lat (ms)
|
Net Sent (MB/s)
|
Net Recv (MB/s)
|
|
9:09p
|
78
|
12,574
|
5,562
|
3.19
|
6,926
|
1.05
|
200
|
141
|
|
9:18p
|
98
|
27,771
|
17,291
|
4.7
|
10,347
|
24.29
|
571
|
312
|
|
9:29p
|
98
|
33,050
|
21,460
|
9.11
|
11,507
|
125.85
|
650
|
352
|
|
9:38p
|
98
|
34,149
|
22,813
|
11.28
|
11,216
|
530.9
|
647
|
345
|
One thing that stands out in the
data is a large, sudden increase in 64k+ IOPS. I’ve adjusted the table to include a row for 64k IOPS and have highlighted the
relevant statistic.
|
FAS6280 Maximum IOPS
|
|
|
Read/Write Mix
|
|
Avg IO Size
|
100/0
|
75/25
|
50/50
|
25/75
|
0/100
|
|
64k
|
61,000
|
43,000
|
32,000
|
26,000
|
22,000
|
|
32k
|
68,000
|
48,000
|
36,500
|
30,000
|
25,000
|
|
24k
|
74,000
|
51,000
|
39,500
|
31,500
|
27,000
|
|
16k
|
80,000
|
56,500
|
43,000
|
36,500
|
30,500
|
|
8k
|
85,000
|
63,000
|
50,000
|
41,500
|
36,000
|
|
4k
|
90,000
|
66,000
|
54,000
|
45,000
|
40,000
|
The workload mix appears
fine for most of the day but experiences large-IOP-size peaks that are outside
our guidelines and cause some pain (38,000 IOPS 1pm 4/5, 45,000 IOPS 11pm 4/4, 37,000 IOPS 11am 4/3) . I’d
also make mention that ~10% of IO to this system is misaligned, which hinders
us from achieving maximum performance ROI. Lastly, this system is
achieving 65-85% dedupe ratios, which is fantastic space conservation but adds
to the overall workload.
As discussed yesterday,
here are our options:
·
Short term steps:
o Stagger workloads (Symantec, et al)
o Disable aggr snapshots (done)
o Stagger dedupe
o Case open on daytime snapshot correlated latency (done)
o Update Data ONTAP
·
Long term solutions:
o Add new disk to passive controller and balance workload or
o Shift workload to a different or new HA pair
