Reading LUN Stats -o

netapp1> lun stats -o
    /vol/UCvol/UClun  (98 days, 0 hours, 46 minutes, 51 seconds)

Read(kbytes)	Write(kbytes)	ReadOps	WriteOps	OtherOps	QFulls	PartnerOps	PartnerKBytes
15940415	640772	30487379	8734	9767	0	30442488	15496708

A few things about this are not immediately intuitive.  First, statistics labeled "Partner" are indirect IO, passed through the cluster interconnect.  This means both heads have to process the information, which is adds up to considerable overhead in terms of work and latency.  

Second, the statistics not labeled  "Partner" are totals: they include both the direct and indirect IOPS.  At first glance you'd see that ReadOps is a very close number (.1% off) to PartnerOps, which would indicate that the traffic is being load balanced across indirect paths.  

But that would be counting the same Ops twice.  If you add up ReadOps, WriteOps, and OtherOps, that's the total number of direct and indirect Ops.  If you then subtract PartnerOps, that resulting number (63392 Ops) is the number of direct Ops.  That's .2% direct IOPS, which indicates a serious pathing configuration issue.

De-Mystifying Obscure NetApp Warnings

SAN volume with a non-UNIX security style:  "The recommendation has to do with avoid auditing that is usually performed by ONTAP on files (NAS) which could add unneeded IOPS to the system." 

Security style means the method the filer uses to determine whether a user has access to a file.

unix

The user’s UID and GID, and the UNIX-style permission bits of the file or directory determine user access. The filer uses the same method for determining access for both NFS and CIFS requests. If you change the security style of a qtree or a volume from ntfs to unix, the filer disregards the Windows NT permissions that were established when the qtree or volume used the ntfs security style.

ntfs

For CIFS requests, Windows NT permissions determine user access. For NFS requests, the filer generates and stores a set of UNIX-style permission bits that are at least as restrictive as the Windows NT permissions. The filer grants NFS access only if the UNIX-style permission bits allow the user access. If you change the security style of a qtree or a volume from unix to ntfs, files created before the change do not have Windows NT permissions. For these files, the filer uses only the UNIX-style permission bits to determine access.

mixed

Some files in the qtree or volume have the unix security style, and some have the ntfs security style. A file’s security style depends on whether the permission was last set from CIFS or NFS. For example, if a file currently uses the unix security style and a CIFS user sends a setACL request to the file, the file’s security style is changed to ntfs. If a file currently uses the ntfs style and an NFS user sends a setpermission request to the file, the file’s security style is changed to unix.

Volume option create_ucode set to off: "A non-unicode directory will be converted to Unicode the first time it is touched by any protocol that requires Unicode directories. It is recommended that create_ucode is set on every volume (NAS and SAN) as converting large directories can delay a takeover operation for longer than the takeover time limit"

"Setting this option to on forces UNICODE format directories to be created by default, both from NFS and CIFS.  By default, it is set to off, in which case, all directories are created in pre-4.0 format, and this first CIFS access will convert it to UNICODE format."

Volume option fs_size_fixed is set to on for a non-mirrored volume - the source side locks the volume size to prevent the source becoming larger than the destination.  If this is on for non replicated volumes, this can result in a full volume by preventing auto grow.    Related: “You can increase the size of the source volume for a FlexVol volume. After you increase the size of a source volume and perform a SnapMirror transfer, the size of the destination volume is automatically increased to the same size as that of the source volume, provided the destination aggregate has sufficient space to contain the resized volume.”  https://library.netapp.com/ecmdocs/ECMP1196991/html/GUID-90D4160B-691B-405E-A26A-D592539555B0.html

Exposure to bug 488909 - Xcopy memory corruption: XCOPY is a vmware VAAI feature that lets the storage system do the copying rather than a host reading and then writing a bunch of data.  We have a bug associated with this, so they need to upgrade ontap.  http://support.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=488909

Reader Question 1: CIFS Changes

Question from a reader!

"Quick question for you. If I move a system to a different subnet and change the IP and rename it, will CIFS shares be affected in any way? Isn’t there an SID that is generated when CIFS is setup. Would that need to change?"

The Active Directory GUID  doesn’t change with the IP address, so if you're only changing the IP you'll just need to update DNS.

But when you change the system name you’ll want to remove the system from the domain, change the IP and/or name, then rejoin the domain under the new name.  

Anyone using the cifs shares will of course be affected by this, since shares are in the format \\\share or \\\share.  Everyone would have to update their shares, their mapped drives, their scripts, saved paths, etc.  

CDOT Raid Groups

A cool bit of trivia about expanding RAID Group sizes.  

For the command storage aggregate modify

For the command storage aggregate add-disks(both quotes from 8.2 command manual)

Which means if you need to balance an unusual number of disks today but will add more disks at some future date, if you want to have small RG’s now and expand them in the future you have to be careful.  For example, if you have 45 disks and want a RG size today of 15 but 20 later, you’d do this:

Storage aggr create –aggregate aggr1 -diskcount 15 –disktype SAS –maxraidsize –s 20

Storage aggr add-disks –aggregate aggr1  -diskcount 15 -disktype SAS –raidgroup –g new

Storage aggr add-disks –aggregate aggr1 -diskcount 15 -disktype SAS –raidgroup –g new

But then we found this:

(Clustered Data ONTAP 82 Physical Storage.pdf)

While this excerpt doesn’t explicitly state you can, I tested in the lab and confirmed: you can manually expand RG sizes of all the RG’s in the aggregate by modifying the max RG size then manually adding disks to each RAID Group.

Storage aggregate add-disks –aggregate aggr1 -diskcount 5 -disktype SAS –raidgroup rg0

Dedupe

Couple quick points on dedupe:

1. sis start /vol/volname is a "from this point forward" command.  It does automatically go back to any existing data and dedupe it.
2. Volume snapmirror is dedupe aware
3. Qtree snapmirror isn't dedupe aware.  You can enable dedupe independently on the source and destination.
4. It's good to run sis start -s on older volumes.  Cleans up stale fingerprints, improves efficiency, and does dedupe the existing data in the volume.  If you run this on the source vol that is snapmirrored, it also affects the dest vol.
5. If you're enabling dedupe on an existing volume, run sis start -s during off hours to tackle existing data.

https://communities.netapp.com/community/netapp-blogs/drdedupe/blog/2010/06/17/snapmirror-and-deduplication--dedupe-here-there-or-everywhere

Adventures with CDOT

Had an interesting situation after an option 4 on a new CDOT system (3250’s), this is a bit long but I wanted to get it all onto paper and into our tribal knowledge.  While one controller was still clearing, I started configuring the other one (set HA true, set to switchless, update from 82p3 to 82p5).  Then I updated the other controller.  When the other system tried to join the cluster, I saw this:

Error: Node "cluster-04" on ring "Management" is offline. Check the health of the cluster using the "cluster show" command. For further assistance, contact support personnel.

Well, ok.  A little checking:

cluster::> cluster show

Node                  Health  Eligibility

--------------------- ------- ------------

cluster-01        true    true

cluster-04        true    true

Warning: Cluster HA has not been configured. Cluster HA must be configured on a

         two-node cluster to ensure data access availability in the event of

         storage failover. Use the "cluster ha modify -configured true" command

         to configure cluster HA.

2 entries were displayed.

cluster::> node show

Node      Health Eligibility Uptime        Model       Owner    Location 

--------- ------ ----------- ------------- ----------- -------- ---------------

cluster-01

          false  true         00:26:23.001 FAS3250              Minneapolis DR Site

cluster-04

          false  true         00:09:39.043 FAS3250

Warning: Cluster HA has not been configured. Cluster HA must be configured on a

         two-node cluster to ensure data access availability in the event of

         storage failover. Use the "cluster ha modify -configured true" command

         to configure cluster HA.

2 entries were displayed.

Well, then let’s modify cluster HA.

cluster::> cluster ha modify -configured true

Warning: High Availability (HA) configuration for cluster services requires

         that both SFO storage failover and SFO auto-giveback be enabled. These

         actions will be performed if necessary.

Do you want to continue? {y|n}: y

Error: command failed: Not enough online nodes in the cluster:

       SL_REMOVE_EPSILON_OOQ_ERROR (code 129)

       There are too few healthy nodes in the cluster to allow join of

       additional nodes. Ensure that the nodes are operational and re-issue the

       command. Use the "cluster show" command on a node in the target cluster

       to view the state of the cluster.

Well sheesh.  So I reboot and what happens?  A takeover.

Jan 22 14:53:28 [msp-cluster-04:callhome.sfo.takeover:CRITICAL]: Call home for CONTROLLER TAKEOVER COMPLETE AUTOMATIC

Jan 22 14:53:28 [msp-cluster-04:callhome.reboot.takeover:error]: Call home for PARTNER REBOOT (CONTROLLER TAKEOVER)

But the system doesn’t think it was taken over, or that it’s in HA mode.

cluster::> storage failover show-giveback

               Partner

Node           Aggregate         Giveback Status

-------------- ----------------- ---------------------------------------------

Warning: Unable to list entries on node cluster-01. RPC: Port mapper

         failure - RPC: Timed out

cluster::cluster ha> modify  -configured true

Warning: High Availability (HA) configuration for cluster services requires

         that both SFO storage failover and SFO auto-giveback be enabled. These

         actions will be performed if necessary.

Do you want to continue? {y|n}: y

Error: command failed: Could not enable auto-sendhome on partner node: Failed

       to set option cf.giveback.auto.enable. Reason: 169.254.97.26 is not

       healthy.

After another reboot I turned off and on HA, then everything cleared up and TO/GB’s were working perfectly.

cluster::> cluster ha modify -configured true

Warning: High Availability (HA) configuration for cluster services requires

         that both SFO storage failover and SFO auto-giveback be enabled. These

         actions will be performed if necessary.

Do you want to continue? {y|n}: y

Notice: HA is configured in management.

So on to the next problem: one of the vol0’s isn’t being recognized (click for larger image).

After a lot of searching, I found this magical solution. 

Curiously,  the vol0 is referred to in the excerpt above as a “7-Mode volume.”   But both vol0’s are, and there’s no way to change it.  The word from other engineers is that this is correct.

cluster::> vol show -is-cluster-volume -is-cluster-volume false

  (volume show)

Vserver   Volume       Aggregate    State      Type       Size  Available Used%

--------- ------------ ------------ ---------- ---- ---------- ---------- -----

cluster-01           vol0         aggr0_msp_cluster_01                                     online     RW        330GB    310.8GB    5%

cluster-02           vol0         aggr0_msp_cluster_02                                     online     RW        330GB    310.8GB    5%

2 entries were displayed.

Lastly, I needed to move one of the vol0’s.  I used this link to move the vol0 over to a new aggregate:

https://kb.netapp.com/support/index?page=content&id=1013762&actp=search&viewlocale=en_US&searchid=1390428133178

Thanks for making it all the way to the end with me!

Vol Options/Thin Provisioning

Once volume guarantee is set to none the volume is considered thin provisioned.  If thin provisioning is being used, I recommend these for most data sets:

Block:

·         Turn on vol autosize

·         Turn off snapshot reserve

·         Turn off fractional reserve (depends on the workload)

·         Turn on snap autodelete (based on the size taken up by snapshots or their age)

o    commitment=try

o    trigger=volume

o    target_free_space=20%

o    delete_order=oldest_first

o    defer_delete=user_created

o    try_first=volume_grow

Turn off LUN reserve

Turn off LUN Space guarantee 

File:

·         Turn on vol autosize

·         Turn snapshot reserve=20%

·         Turn on snap autodelete (based on the size taken up by snapshots or their age)

o    commitment=try

o    trigger=snap_reserve

o    target_free_space=20%

o    delete_order=oldest_first

o    defer_delete=user_created

o    try_first=volume_grow