Understanding SolidFire Capacity

To calculate the effective capacity available, follow this formula:

• Error Threshold (#3) minus Used Capacity (#1) = Available physical space
• In the example below, 86.24TB - 48.41TB = 37.83TB
• Multiply Redupe Ratio * Compression Ratio, then divide by 2 (for Double Helix)
• In the example below, (1.81 * 1.79)/2 = 1.62:1
• Multiply the results of (a) and (b)
• 37.83TB * 1.62 = 61.29TB

*This calculation assumes the current rate of dedupe and compression will continue
*This calculates capacity until the Error Threshold is reached, not the Total Capacity.

From ActiveIQ, our cloud monitoring tool:

1. Used Capacity.  This is the capacity physically taken up on disk by data.  After dedupe, compression, and double helix occur, this number is the end result.
2. Warning Threshold.  This is an adjustable alert threshold to alert that you’re approaching the Error Threshold.
3. Error Threshold.  This is the point after which the system cannot rebuild the second copy of data after a node loss.  This is calculated by subtracting one node’s physical block capacity from the Total Capacity.
4. Total Capacity.  This is the raw physical space on disk.  In this example, 1.92TB * 9 SSD’s * 5 nodes = 86.42TB.

From the SolidFire GUI:

      5. Block Remaining.  This is calculated subtracting the Used Capacity (#1) from Total Capacity (#4).

      6. Block Capacity until Warning.  This is calculated subtracting the Used Capacity (#1) from Warning Threshold (#2).

***Bonus***

       7. In ActiveIQ, under Reporting | Cluster Efficiency, hover over the graph to view the dedupe and compression ratios.  In this lab system, 1.81 * 1.79 = 3.24:1

Always On Dedupe

We're finding more and more clients with transient data who want to dedupe all the time.  With more powerful systems and especially the advent of flash, you can set dedupe to run all the time with little to no performance impact.  Here's a primer:

You can set it to always on like this:

Create a schedule object:

cluster:>> job schedule cron create –name per_minute –dayofweek * –hour * –minute 1

Create a dedupe policy linked to that schedule:

cluster:>> volume efficiency policy create –vserver -policy “Always_On_Dedupe” –type schedule –schedule per_minute –qos-policy background –enabled true

Assign that dedupe policy to your volume:

cluster::> volume efficiency on -volume -vserver -qos-policy Always_On_Dedupe

Those commands might be a bit off, I’m writing free-hand here.  It would be easier to use system manager to do it via GUI:

And here are some of the published results:

Design, testing, and results document: 

http://www.netapp.com/us/media/tr-4335.pdf

Command Reference Link:

https://library.netapp.com/ecmdocs/ECMP1141760/html/GUID-2593A6B9-C800-4ABE-8033-DC82F12B4027.html

Performance Case

Just a performance case I worked recently.  These kinds of things can be instructive for people with similar problems, or those who learn by playing along at home.  Here's my email, sorry for the copy paste without context.

 Please take a look at the email chain below as the starting point for this conversation.  At this point, we’ve reviewed four sets of performance data gathered over the last two months and have closely correlated a spike in large-IOP-size traffic to our latency spikes.  This spike is both in number and size of IOPS, exceeding 32,000 IOPS for 15+ minutes at a time.  There is no single volume driving the traffic, as it appears to be increasing dramatically across the board.  

Here is a summary of the performance data from Thursday 4/3, please note the IOP ramp up and associated latency:

Start Time	CPU Busy	NFS Op/s	Read Op/s	Read Lat (ms)	Write Op/s	Write Lat (ms)	Net Sent (MB/s)	Net Recv (MB/s)
9:53p	52	7,127	1,256	2.32	5,731	0.63	27	50
9:57p	67	13,637	5,396	19.96	8,076	110.09	145	81
10:05p	99	31,119	17,272	22.91	13,697	341.18	371	205
10:13p	99	32,311	22,519	8.4	9,739	229.45	621	200
10:24p	99	23183	12,819	7.21	10,261	143.16	348	260

And here is the data from Thursday 2/13.

Period	CPU Busy	NFS Op/s	Read Op/s	Read Lat (ms)	Write Op/s	Write Lat (ms)	Net Sent (MB/s)	Net Recv (MB/s)
9:09p	78	12,574	5,562	3.19	6,926	1.05	200	141
9:18p	98	27,771	17,291	4.7	10,347	24.29	571	312
9:29p	98	33,050	21,460	9.11	11,507	125.85	650	352
9:38p	98	34,149	22,813	11.28	11,216	530.9	647	345

One thing that stands out in the data is a large, sudden increase in 64k+ IOPS.  I’ve adjusted the table to include a row for 64k IOPS and have highlighted the relevant statistic.

FAS6280 Maximum IOPS
	Read/Write Mix
Avg IO Size	100/0	75/25	50/50	25/75	0/100
64k	61,000	43,000	32,000	26,000	22,000
32k	68,000	48,000	36,500	30,000	25,000
24k	74,000	51,000	39,500	31,500	27,000
16k	80,000	56,500	43,000	36,500	30,500
8k	85,000	63,000	50,000	41,500	36,000
4k	90,000	66,000	54,000	45,000	40,000

  The workload mix appears fine for most of the day but experiences large-IOP-size peaks that are outside our guidelines and cause some pain (38,000 IOPS 1pm 4/5,  45,000 IOPS 11pm 4/4, 37,000 IOPS 11am 4/3) .   I’d also make mention that ~10% of IO to this system is misaligned, which hinders us from achieving maximum performance ROI.  Lastly, this system is achieving 65-85% dedupe ratios, which is fantastic space conservation but adds to the overall workload.

  As discussed yesterday, here are our options:

·         Short term steps:

o   Stagger workloads (Symantec, et al)

o   Disable aggr snapshots (done)

o   Stagger dedupe

o   Case open on daytime snapshot correlated latency (done)

o   Update Data ONTAP

·         Long term solutions:

o   Add new disk to passive controller and balance workload or

o   Shift workload to a different or new HA pair

Dedupe

Couple quick points on dedupe:

1. sis start /vol/volname is a "from this point forward" command.  It does automatically go back to any existing data and dedupe it.
2. Volume snapmirror is dedupe aware
3. Qtree snapmirror isn't dedupe aware.  You can enable dedupe independently on the source and destination.
4. It's good to run sis start -s on older volumes.  Cleans up stale fingerprints, improves efficiency, and does dedupe the existing data in the volume.  If you run this on the source vol that is snapmirrored, it also affects the dest vol.
5. If you're enabling dedupe on an existing volume, run sis start -s during off hours to tackle existing data.

https://communities.netapp.com/community/netapp-blogs/drdedupe/blog/2010/06/17/snapmirror-and-deduplication--dedupe-here-there-or-everywhere