Aggregate Snapshot autodelete_base

Interesting aggregate snapshot name: autodelete_base

The autodelete_base snapshot was created as a fix for BUG 263339: Aggregate snapshot autodelete is leaving me with no snapshots. The filer will automatically create a snapshot called autodelete_base at the aggregate level when a snapshot is removed as a result of autodeletion. This ensures that there is always a new snapshot in the aggregate after a snapshot is deleted.

https://kb.netapp.com/support/index?page=content&id=2011516&locale=en_US

Different Sized Disks in a RAID Group

We encountered an interesting situation up here recently, and it generated a lot of debate between the PSC’s.  We finally locked down the answer, and I figured I would write it up and spread the knowledge! 

 The issue was that a RAID DP group on a system had 13 450GB data drives, 1 450GB Parity drive, 1 600GB data drive, and 1 600GB DP drive.  The data drives were all right-sized to the same “effective used” size, but the 600GB DP drive was showing as not right-sized.  This was in debate because it was commonly believed that all disks within a single RAID Group were required to be the same effective size.

  In an amazing FAQ document (please find it attached), we found the following:

  The document is written by Jay White, a Technical Marketing Engineer widely regarded as the authority on NetApp storage subsystems.  It’s a fascinating series of scenario studies that comes highly recommended.

NetApp Insights: Usable Capacity

I saw some documentation given to a customer that estimated that for 144 2TB SATA disks (294.9TB), the customer could expect 170TB usable. It also said for 288 450GB SAS disks (126.5TB) they should expect 91TB usable. That's a big loss from a client's perspective.

I've previously developed a calculator to make it easy to plan your RAID Groups and aggregates, but now I want to use that to take a closer look at where all that space actually goes. A NetApp PSC expert told me the general rule is for FC disks 70% of raw is usable, and for SAS/SATA you take off another 10-12%. But let's see if we can dig into that.

• Computers measure base 2, but drive manufacturers measure base 10.  This means if your drive is labeled 1GB, it's actually 1000MB, not 1024MB.  
• The fuzziest part: drive manufacturers reserve between 7% and 15% on each disk.  Some of this is for parity, a lot of this is to account for failed sectors.  I've observed 2TB SATA drives reporting 1.69TB or less for a loss of 13.5%, I'll use that for these calculations.
• You lose some space due to WAFL/disk asymmetry.  The basic idea is that a 4KB block doesn't fit neatly into the disk sectors, so there's some waste.  Some of this is taken into account by the manufacturer's reserve, so I can't quantify this in our calculations.
• You lose some space to right-sizing.  Since each drive manufacturer's 2TB disk is a slightly different size, ONTAP right-sizes all disks to the lowest common denominator to avoid incompatibly sized disks later.  I can't find any data on how much space you lose to this process.
• For every RAID Group, you lose 2 disks to parity/double parity.
• You need to account for spares obviously.
• WAFL requires 10% of the usable space to run the file system.

So for our two scenarios above, here's what we find:
Scenario 1
288 450GB SAS Drives
Spare drives: 8
Parity drives: 28

Credit: me!

Scenario 2
144 2TB SATA Drives
Spare drives: 6
Parity drives: 20

Credit: me!

Analysis:

• You lose a consistent 15% because of the drive manufacturer whether you use EMC or NetApp or any other vendor.
• To accomplish NetApp's goal of data protection (spares, parity, WAFL striping), you lose another 18-23%.
• When you factor in backups and snapshots, you'll lose even more space.
• One bright side is that using NetApp's dedup and efficient snapshot technologies, you can end up regaining this lost space.

Notice I'm still a considerable way away from the estimates given to the customer: 7.8% low for the 450GB system and 5% high for the 2TB system.  There's still some gaps in my numbers here, I would definitely appreciate any tips!

NetApp Training Brain Dump: Aggregate/RAID Group/Shelf Planning Calculator

Capacity Planning Calculator
 Part of understanding how to implement a NetApp system is figuring out the layout of your disks. Doing the math a couple of times definitely helps, but at some point it's nice to have a Capacity Planning Calculator. So here it is, a quick and easy calculator in excel format to help you figure out the relationship between your RAID Group size and quantity, shelves, drive sizes, and spare requirements.

Please note that although this table is true to NetApp's documentation, there's no way for this to be 100% accurate in this without using NetApp's actual ONTAP code, which I don't have access to :-) This calculator is meant for planning in a simple, understandable format. Technical notes below, but I highly recommend reading this before working with this tool.

Capacity Planning Calculator Link (not a virus I promise): http://www.box.net/shared/l8v66b8jzx
I password protected portions of the calculations to make it clear what you can edit and keep life simple.  If you wish to improve upon or edit this sheet, the password is netapp

Enjoy!

Courtesy: me!

Notes (You're gonna want to read these):
-  Disk manufacturers reserve 7% of space to account for failed sectors.
-  WAFL reserves 10%
-  Fields you may alter are marked white.  Do not change fields marked grey.
-  Two drives per RG are reserved for RAID DP.  You may edit the number of spares you wish to keep.
-  All numbers are in TB.  Convert your drive size to TB (e.g. 300GB = 300GB/1024GB = .293TB).
-  The largest 15k SAS disk available as of 5.13.2011 is 600GB
-  The largest FC disk available as of 5.13.2011 is 750GB
-  Aggregates size limits do not count space lost to parity, spares, or disk reserve.  Click here for NetApp documentation (NOW login required)
-  Assumes full shelves.
-  If you want a super deep dive into space reservation with ONTAP, try here: http://rogerluethy.wordpress.com/2011/01/14/play-with-netapp-numbers/

NetApp Training Brain Dump: RAID Groups, Disks, and Shelves

A brain dump of fundamental NetApp concepts and terms with regards to RAID Groups (RG), Disks, and Shelves.

Note: You can use this calculator to play with the numbers within NetApp's best practices and find the optimal solution for your environment.

RAID Group Best Practices*1:

• An aggregate is made up of RAID Groups.  You cannot split a RAID Group between aggregates, but you can have multiple RAID Groups that make up a single aggregate. 
• Always use RAID-DP, which is an implementation of RAID-6 that uses striped data drives and 2 disks reserved solely for parity and diagonal parity.  This allows you to lose two disks per RAID Group without losing data.  As the ratio of data disks to parity disks goes up, your space efficiency goes up, but also the risk of losing 3 disks in a RG increases.  There are also performance implications for a high data disk to parity disk ratio.  
• Same size/speed/type disks in a RG.  No mix and match here.  Any imbalance will result in over-utilization of some disks compared to others (resulting in increased likelihood of failure), as well as sub-par performance and uneven utilization of capacity.
• Max RAID group size is 28 disks (for SAS/FC), best practice being 16 disks per RG, for a 14:2 data:parity ratio.  This ratio is the balance between data protection and storage utilization: you can lose 2 disks out of 16 and still be up and running.  As mentioned before, higher ratios mean less protection but more efficient space utilization.
• Interesting note: HP's EVA line uses RAID 5 groups of 8 disks, with the capability of losing 1 disk per RAID group.  
• Minimum best practice RG is 7 disks. 
• SATA RG max size is 20 disks as of ONTAP 8.0.1, it was 16 disks before.
• Lastly, the performance increase for adding more spindles drops dramatically to a flat line after 14 data drives, partially for the reasons above.
• Make RAID groups all the same size, for the same reason that you use disks of the same speed and size: homogeneity creates the most efficient systems.
• Remember to take into account the number of disks you have and their size when determining your RAID group layout.  ONTAP 8.0 supports aggregates of up to 100TB (depending on the system), but any pre-ONTAP 8.0 systems support only 16TB aggregates.  Spares and parity disks are not included in this limit.
• Remember, shelves are not owned.  Ownership by CPU Modules is disk-level.

Aggregate Limits by System*2

• FAS6080 = 100TB
• FAS6070 = 100TB
• FAS6040 = 70TB
• FAS6030 = 70TB
• FAS3170 = 70TB
• FAS3160 = 50TB
• FAS3140 = 40TB
• FAS3070 = 50TB (requires a PVR)
• FAS3050 = 40TB (requires a PVR)
• FAS3040 = 40TB (requires a PVR)

Disks:

• When a disk fails, the system "bypasses" the disk and recreates the data on the fly from parity calculations to serve requests.  It also pulls any unused disks available into the RAID Group and rebuilds to that disk using the parity calculations.
• The term "spare" is nuanced, as only a disk assigned to a controller (but not part of a RG) is considered a spare in proper terminology.  This is because only a disk assigned to a controller will be automatically pulled into a RG and built as a data disk.  Assigned disks also have a consistent stream of traffic to them, checking it for validity.  
• In choosing the number of spares for a system, use this document: http://partners.netapp.com/go/techontap/matl/storage_resiliency.html
• If 2 disks fail and there are no spares available, you have 24 hours to replace a failed disk before the system halts to protect the data (I would love to find the override for this).
• When a "disk show" displays (xx) for a specific disk, that indicates the system can't read that disk at all.  Usually this occurs when someone pulls a disk and replaces it before the system is able to recognize the old one was pulled.  Typically, you want to give the system 60s after you pull the disk before you put in a new one.

Shelves:

• Let shelves spin for at least 2 minutes before powering on the controllers per NetApp best practice. 
• SFP (Small Form Factor Pluggable): Standard size of a plug-in.  Smaller than QSFP.
• OSFP (Optical Small Form Factor Pluggable):  Fibre connection for FCP, same size as regular SFP.  One use for this is from the shelf IOM to the FAS system for SAS. AKA GBIC, Optical SFP Transciever
• QSFP (Quad Small Form Factor Pluggable): Copper connection plug-in.  Bigger plug-in than SFP.
• DS14mX (Disk Shelf 14-Drive generation X).  14 drives, dual shelf modules. 
• Nomenclature: a group of DS14 shelves linked together is called a loop.
• 6 shelves/loop max. 
• DS14mX-FC is a Fibre Channel shelf supported by FAS systems. 
• ESH/ESH2/ESH4: Shelf modules with connections back to the FAS system or other shelves.  These can use SFP for copper interconnect or OSFPs for FCP.
• DS14mX-AT is a SATA shelf supported by FAS systems.
• AT-FCX: Shelf module for a SATA shelf to communicate over FCP to other shelves/FAS system.
• DSXXXX (DS4243, 2246, etc): SAS/SATA/SSD shelves supported by FAS systems. 24 drives, dual modules (called IOMs). 
• Naming convention: DS +  #U + #drives + throughput per port in Gb. DS4243 is a 4U 24 disk 3Gb shelf.
• Nomenclature: a group of linked SAS/SATA DS4243's is called a stack rather than loop.
• IOM3/IOM6: Shelf modules with connections back to the FAS system or other shelves.  These use QSFP copper from the IOM to the FAS system/other shelves. 
• SAS backplanes can take SATA drives, but not visa versa. 
• 10 shelves/stack max.

Credit: Me!

Sources:
*1 http://media.netapp.com/documents/tr-3437.pdf
*2 http://lockienotlucky.wordpress.com/2011/01/28/notes-for-netapp-basics/


Great table for Aggregate/Disk size/RAID Group optimization.
http://now.netapp.com/NOW/knowledge/docs/ontap/rel73/html/ontap/rnote/rel_notes/reference/r_oc_rn_feat73_aggr-size-max-drives.html

NetApp Training Brain Dump: Bird's Eye View

Preparing for a deep dive into NetApp technology! In an intelligence report to King George in 1776, England's spies wrote about John Adam's strength being that he "sees large things largely."  I try to take that approach of not getting caught in minutia when approaching a new technology, to better grasp the big picture.  The next few posts will be my journey into that, and I'm sure that in trying to encapsulate complex ideas I will be slightly incorrect in some of these statements.  Nuance comes with time!  So here we go, basic terms, spelled out in English:

Product Definitions:

- FAS system (aka filer): NetApp's term for the custom machine that manages the storage. Roughly equivalent in purpose to HP EVA, IBM XIV, etc. Capable of serving storage over ethernet NAS (file based protocols like HTTP, FTP, CIFS, etc) or SAN block based protocols (FCoE, iSCSI, or FC).  FAS (Fabric Attached Storage) designates that the filer is operating on FCoE, iSCSI, or FC rather than simply as a NAS device.

- SnapVault (OSSV): NetApp's backup solution.  Allows full or incremental backups to be transfered from a server directly to a NetApp storage system.

- SnapMirror: Real time replication.  Effectively creates software layer RAID 1 by creating exact clones of volumes or qtrees (can't mirror an aggregate from what I've read).  This enables NetApp's Metrocluster.

- Metrocluster: their version of DR implementation.  Two options: stretch (both controllers in one datacenter) or fabric attached (replication across an ISL (inter-site link) with one controller in each datacenter).

- SyncMirror:

- SnapDrive:

- FlexShare: Allows you to set processing priority for volumes within an aggregate.

- iGroup: Initiator group.  All LUN's are mapped to an iGroup, which handle LUN masking based upon the client system.  The iGroups basically contain the specifications for the OS-App combo etc to communicate to the LUN.  Typically, each server (or cluster) should have its own iGroup based upon the OS, Application (SQL, VMware, etc), and SAN protocol.

Break it down: There are a few layers where the building blocks of storage are combined to form higher level concepts for easier management, each with NetApp-specific jargon.  No worries, I'm here to translate and simplify:

- Layer 1: Disk drives.  duh.

- Layer 2: RAID Group.  This is a group of up to 28 disks operating as a pool of storage, 16 best practice.  You want all the RG's in a specific aggregate to be the same size.  Two parity disks per RG.

- Layer 2.5: Plex. A plex is a physical copy of the WAFL storage within the aggregate. A mirrored aggregate consists of two plexes; unmirrored aggregates contain a single plex.  Take 11 players from the Chicago Bears and NE Patriots, and they're a football team.  Move them around a bit, and you can put them in shotgun formation.  You can say that they're a set of players (aggregate), and they're distinctly from the Bears and the Patriots (volumes in the aggregate), and that they're a formation (plex)...there are many ways to view the organization of data.

- Layer 3: Aggregate.  This is a group of RAID Groups.  A RAID group can not be assigned to more than one Aggregate.

- Layer 4: Volume. This is space carved out inside an aggregate.  Typically this is space for 1 LUN + reserve space.
- Layer 5: LUN.  This is space carved out inside a volume.  There can be multiple LUNs per volume, but that can be inadvisable.  The LUN is the actual virtual disk being presented to the server.

- Layer 6: QTree. Essentially, this is space carved out inside a LUN for a particular directory, sometimes with a hard limit.

I'll keep these definitions updated as I learn the nuances or need to make corrections.