NetApp Training Brain Dump: RAID Groups, Disks, and Shelves

A brain dump of fundamental NetApp concepts and terms with regards to RAID Groups (RG), Disks, and Shelves.

Note: You can use this calculator to play with the numbers within NetApp's best practices and find the optimal solution for your environment.

RAID Group Best Practices*1:

• An aggregate is made up of RAID Groups.  You cannot split a RAID Group between aggregates, but you can have multiple RAID Groups that make up a single aggregate. 
• Always use RAID-DP, which is an implementation of RAID-6 that uses striped data drives and 2 disks reserved solely for parity and diagonal parity.  This allows you to lose two disks per RAID Group without losing data.  As the ratio of data disks to parity disks goes up, your space efficiency goes up, but also the risk of losing 3 disks in a RG increases.  There are also performance implications for a high data disk to parity disk ratio.  
• Same size/speed/type disks in a RG.  No mix and match here.  Any imbalance will result in over-utilization of some disks compared to others (resulting in increased likelihood of failure), as well as sub-par performance and uneven utilization of capacity.
• Max RAID group size is 28 disks (for SAS/FC), best practice being 16 disks per RG, for a 14:2 data:parity ratio.  This ratio is the balance between data protection and storage utilization: you can lose 2 disks out of 16 and still be up and running.  As mentioned before, higher ratios mean less protection but more efficient space utilization.
• Interesting note: HP's EVA line uses RAID 5 groups of 8 disks, with the capability of losing 1 disk per RAID group.  
• Minimum best practice RG is 7 disks. 
• SATA RG max size is 20 disks as of ONTAP 8.0.1, it was 16 disks before.
• Lastly, the performance increase for adding more spindles drops dramatically to a flat line after 14 data drives, partially for the reasons above.
• Make RAID groups all the same size, for the same reason that you use disks of the same speed and size: homogeneity creates the most efficient systems.
• Remember to take into account the number of disks you have and their size when determining your RAID group layout.  ONTAP 8.0 supports aggregates of up to 100TB (depending on the system), but any pre-ONTAP 8.0 systems support only 16TB aggregates.  Spares and parity disks are not included in this limit.
• Remember, shelves are not owned.  Ownership by CPU Modules is disk-level.

Aggregate Limits by System*2

• FAS6080 = 100TB
• FAS6070 = 100TB
• FAS6040 = 70TB
• FAS6030 = 70TB
• FAS3170 = 70TB
• FAS3160 = 50TB
• FAS3140 = 40TB
• FAS3070 = 50TB (requires a PVR)
• FAS3050 = 40TB (requires a PVR)
• FAS3040 = 40TB (requires a PVR)

Disks:

• When a disk fails, the system "bypasses" the disk and recreates the data on the fly from parity calculations to serve requests.  It also pulls any unused disks available into the RAID Group and rebuilds to that disk using the parity calculations.
• The term "spare" is nuanced, as only a disk assigned to a controller (but not part of a RG) is considered a spare in proper terminology.  This is because only a disk assigned to a controller will be automatically pulled into a RG and built as a data disk.  Assigned disks also have a consistent stream of traffic to them, checking it for validity.  
• In choosing the number of spares for a system, use this document: http://partners.netapp.com/go/techontap/matl/storage_resiliency.html
• If 2 disks fail and there are no spares available, you have 24 hours to replace a failed disk before the system halts to protect the data (I would love to find the override for this).
• When a "disk show" displays (xx) for a specific disk, that indicates the system can't read that disk at all.  Usually this occurs when someone pulls a disk and replaces it before the system is able to recognize the old one was pulled.  Typically, you want to give the system 60s after you pull the disk before you put in a new one.

Shelves:

• Let shelves spin for at least 2 minutes before powering on the controllers per NetApp best practice. 
• SFP (Small Form Factor Pluggable): Standard size of a plug-in.  Smaller than QSFP.
• OSFP (Optical Small Form Factor Pluggable):  Fibre connection for FCP, same size as regular SFP.  One use for this is from the shelf IOM to the FAS system for SAS. AKA GBIC, Optical SFP Transciever
• QSFP (Quad Small Form Factor Pluggable): Copper connection plug-in.  Bigger plug-in than SFP.
• DS14mX (Disk Shelf 14-Drive generation X).  14 drives, dual shelf modules. 
• Nomenclature: a group of DS14 shelves linked together is called a loop.
• 6 shelves/loop max. 
• DS14mX-FC is a Fibre Channel shelf supported by FAS systems. 
• ESH/ESH2/ESH4: Shelf modules with connections back to the FAS system or other shelves.  These can use SFP for copper interconnect or OSFPs for FCP.
• DS14mX-AT is a SATA shelf supported by FAS systems.
• AT-FCX: Shelf module for a SATA shelf to communicate over FCP to other shelves/FAS system.
• DSXXXX (DS4243, 2246, etc): SAS/SATA/SSD shelves supported by FAS systems. 24 drives, dual modules (called IOMs). 
• Naming convention: DS +  #U + #drives + throughput per port in Gb. DS4243 is a 4U 24 disk 3Gb shelf.
• Nomenclature: a group of linked SAS/SATA DS4243's is called a stack rather than loop.
• IOM3/IOM6: Shelf modules with connections back to the FAS system or other shelves.  These use QSFP copper from the IOM to the FAS system/other shelves. 
• SAS backplanes can take SATA drives, but not visa versa. 
• 10 shelves/stack max.

Credit: Me!

Sources:
*1 http://media.netapp.com/documents/tr-3437.pdf
*2 http://lockienotlucky.wordpress.com/2011/01/28/notes-for-netapp-basics/


Great table for Aggregate/Disk size/RAID Group optimization.
http://now.netapp.com/NOW/knowledge/docs/ontap/rel73/html/ontap/rnote/rel_notes/reference/r_oc_rn_feat73_aggr-size-max-drives.html