CDOT Tips: InterCluster LIFs

When you see ICL or “InterCluster LIF,” think SnapMirror/SnapVault.  ICL’s are interfaces totally dedicated to replication between clusters, that’s all they can do.  Here’s a couple important things to think about:

1.       There are two kinds of replication in CDOT:

a.       IntraCluster: this means volume move from one node to the other within the cluster.  This occurs using SnapMirror via the private 10Gb cluster network.

b.      InterCluster: This means SnapMirror or SnapVault to another cluster, which occurs over TR’s shared network.

2.     All of a cluster’s SnapMirror and SnapVault to other clusters only occur via the ICL’s. 

3.       ICLs work on two kinds of ports: Role=Data or Role=InterCluster.  Remember, you can assign a role to a port so that only LIFs with compatible roles will ever land there.

4.       Create at least 1 ICL per node.  ICL’s can’t be failed over between nodes, they are dedicated to that node.

5.       In 8.2, a node’s ICL’s can only be in a single network.   In 8.3 IPSpaces are re-introduced and allow multiple ICL’s in multiple IPSpaces per node.

CDOT Tips: Network Troubleshooting

Here are some essential commands for digging into your CDOT network configuration:

How to show the status of your physical and virtual ports:

>net port show –node node1

Node   Port   Role         Link   MTU Admin/Oper  Admin/Oper Admin/Oper
------ ------ ------------ ---- ----- ----------- ---------- ------------
eg-si-clsn-e01-h02
       a0a    data         up    1500  true/-     auto/full   auto/10000
       a0a-2003
              data         up    1500  true/-     auto/full   auto/10000
       a0b    data         down  9000  true/-     auto/-      auto/-
       e0M    node-mgmt    up    1500  true/true  full/full   auto/1000
       e0a    cluster      up    9000  true/true  full/full   auto/10000
       e0b    data         up    1500  true/true  full/full   auto/10000
       e0c    cluster      up    9000  true/true  full/full   auto/10000
       e0d    data         up    1500  true/true  full/full   auto/10000

How to show detailed info of a specific port:

>net port show –node node1 –port a0b

                         Port: a0b
                           Role: data
                           Link: down
                            MTU: 9000
Auto-Negotiation Administrative: true
   Auto-Negotiation Operational: -
     Duplex Mode Administrative: auto
        Duplex Mode Operational: -
           Speed Administrative: auto
              Speed Operational: -
    Flow Control Administrative: full
       Flow Control Operational: -
                    MAC Address: 02:a0:98:5b:40:17
              Up Administrative: true
                      Port Type: if-group
    Interface Group Parent Node: -
    Interface Group Parent Port: -
          Distribution Function: ip
                  Create Policy: multimode_lacp
               Parent VLAN Node: -
               Parent VLAN Port: -
                       VLAN Tag: -
               Remote Device ID: -

How to show the MAC addresses of all your ports:

>port show -node  node1 -fields mac

node               port mac
------------------ ---- -----------------
eg-si-clsn-e01-h02 a0a  02:a0:98:5b:40:16
eg-si-clsn-e01-h02 a0a-2003 02:a0:98:5b:40:16
eg-si-clsn-e01-h02 a0b  02:a0:98:5b:40:17
eg-si-clsn-e01-h02 e0M  00:a0:98:5b:40:2a

Note that when you create an ifgrp in 8.2.x, the member ports will inherit the MAC address of the ifgrp.  E.g., a0a will have the same MAC as e0a and e0c. 

How to ping from a specific LIF:

  If you use the ping command, there’s no obvious way to see which port or LIF it’s emitting from.  You can use this command to specify which node or LIF you want to test:

>Network ping –node node1 192.168.10.10

>Network ping –lif-owner vserver1 –lif nfs_data_lif

There are also some cool options like number of packets, allow fragmentation, etc.

https://library.netapp.com/ecmdocs/ECMP1366832/html/network/ping.html

How to turn up/down a port:

>set advanced

> network port modify –node node1 –port e0a –up-admin true

How to list who your port is connected to:

>node run -node node1 -command cdpd show-neighbors

CDOT Tips: Replication

Following the “Data has mass” train of thought, here’s how we move and protect data in CDOT.  We have three types of replication:

•           SyncMirror: Synchronous replication.  For redundancy.
•           SnapMirror (DP): Replication to a 1-minute granularity.  For disaster recovery.
•           SnapVault (XDP): Replication to a 1-hour granularity.  Think backups.

In CDOT we’ve simplified the commands and protocol: to create a SnapVault relationship, you run “SnapMirror create –type XDP.”  To create a SnapMirror relationship, you run “SnapMirror create –type DP.”  Here’s an example of setting up SnapMirror via CLI.

vs2::> snapmirror create -destination-path vs2:dept_eng_dp_mirror2 -source-path vs1:dept_eng -type DP

The primary distinction between DP SnapMirror and XDP SnapMirror is that SnapVault allows you to keep more snapshots on the destination.  Essentially, XDP SnapMirror is for long-term backups.  Other differences:

•           DP SnapMirror relationships can be reversed (swap destination and source)
•           DP SnapMirror can replicate every minutes, XDP SnapMirror once per hour.
•           DP SnapMirror destination volumes can be made read/write.

Do you have some datasets that would benefit from a quicker time to recovery or have stricter SLA’s?  If so, DP SnapMirror is the best choice.    Now, with many systems and many volumes being replicated, how do you keep track of it all?  In our no-cost tool OCUM 6.0, there is a “Protection” tab that allows you to setup, change, remove, restore, and monitor all your replication relationships. 

Because ONTAP is our single-platform operating system, this also means you can replicate to the cloud (Cloud ONTAP in AWS or Azure), to an all-SATA NetApp “backup” system, or any other NetApp you have.  This “ONTAP everywhere” ubiquity is part of why ONTAP manages more exabytes than any other storage operating system in the world.

CDOT Tip #11: Performance on the Fly (QOS)

QOS is the performance management system within ONTAP.  It doesn’t just stop a volume from going above a certain level of performance, it also monitors and manages performance.  So it’s your one-stop shop for most perf questions!

You can use ‘qos statistics latency show’  to see volume-level latency numbers.   

-          The left most column ‘Latency’ is total latency from request received until response to the client

-           ‘Network’ is the time it takes to traverse the network stack inside ONTAP

-          ‘Cluster’ will show latency due to indirect IO

-          ‘Data’ is ONTAP itself

-          ‘Disk’ is time to leave the controller and hit the disk

cluster1::> qos statistics latency show -iterations 100 -rows 3

Policy Group           Latency    Network    Cluster    Data       Disk       QoS

-------------------- ---------- ---------- ---------- ---------- ---------- ----------

-total-                110.35ms   110.02ms        0ms   327.00us        0ms        0ms

vs1vol0                167.82ms   167.22ms        0ms   603.00us        0ms        0ms

vol1                   117.76ms   117.56ms        0ms   191.00us        0ms        0ms

vol2                    44.24ms    44.05ms        0ms   190.00us        0ms        0ms

-total-                 38.89ms    38.63ms        0ms   256.00us        0ms        0ms

vol2                    64.47ms    64.20ms        0ms   266.00us        0ms        0ms

vol1                    27.28ms    27.03ms        0ms   253.00us        0ms        0ms

vs1vol0                 23.72ms    23.47ms        0ms   249.00us        0ms        0ms

-total-                409.81ms   409.65ms        0ms   169.00us        0ms        0ms

A simpler view with IOPS and throughput can be found using ‘qos statistics performance show.’

    cluster1::> qos statistics performance show -iterations 100 -rows 4

Policy Group           IOPS      Throughput   Latency

-------------------- -------- --------------- ----------

-total-                    79     1296.00KB/s   337.41ms

_System-Best-Effort        25           0KB/s        0ms

vol1                       24       96.00KB/s   193.72ms

vol2                       18     1152.00KB/s   750.98ms

vs1vol0                    12       48.00KB/s   707.38ms

-total-                   109        1.99MB/s   133.27ms

https://library.netapp.com/ecmdocs/ECMP1196817/html/qos/statistics/performance/show.html

Please note that the –total- line indicates the system-wide performance.  Very useful!

CDOT FCP

I spent some time putting information together on SAN on CDOT.  

-          Basic Architecture

o   CDOT uses NPIV to virtualize WWPNs.  This means any node can take traffic for any vserver, and ALUA optimizes the paths.

o   When zoning, utilize these vWWPNs and not the physical WWPNs.

o   Each SVM gets its own IQN (iSCSI) or WWNN (FC).

o   8-node cluster limit today

o   LUNs can be moved to new volumes non-disruptively inside a cluster

o   Limits

§  CDOT 8.2: 8,192 LUNS per node | 49,152 LUNS per cluster | 2,048 iSCSI sessions per node

§  CDOT 8.3: 12,288 LUNS per node | 98,304 LUNS per cluster | 8,192 iSCSI sessions per node

§  Linux host (8.2): 2,048 devices (# LUNS * # paths) | 16TB LUNs

§  Windows (8.2): 255 LUNs per host, 2TB LUNs (MBR) or 16TB (GPT)

o   FAS80X0 support 4-port 8Gb cards or 2-port 16Gb cards.

-          Useful commands:

o   Network interface show (will show FCP lifs)

o   Fcp show

o   Igroup show

o   Lun show

o   system node run -node cluster1-01 fcp topology show

o   Useful SAN Setup how-to (8.1 but very applicable): https://kb.netapp.com/support/index?page=content&id=1013341

-          Foreign LUN Import: 8.3 feature, no license required.  8.3.1 enables the online import using redirect.  Cutover happens first: pause IO to foreign SAN, present LUN to NetApp.  Foreign LUN shows up as a disk in ONTAP (think V-series) and then map this disk to the original host to serve the data.  Bring everything online, data is copied to NetApp while serving data.  Once it’s all moved, shut down foreign SAN. 

You can find the 8.2 SAN config guide here:

https://library.netapp.com/ecmdocs/ECMP1196793/html/index.html

And 8.2.1 SAN Config Guide updates:

https://library.netapp.com/ecmdocs/ECMP1368692/html/accessible_toc.html

8.3 SAN config Guide:

https://library.netapp.com/ecmdocs/ECMP12458217/html/index.html

NetApp IT: Why Clustered Data ONTAP is the Key to Unlocking the Hybrid Cloud

"Business has changed for NetApp IT with the adoption of the hybrid cloud model. The key to our strategy is NetApp® clustered Data ONTAP®, the foundation for our on-premises storage environment and an integral part of our hybrid cloud model. Clustered Data ONTAP enables our hybrid cloud model because of its ability to enhance the scale-out architecture and non-disruptive operations of clustered Data ONTAP."

http://ow.ly/PZEp7

Blog Views

My blog just crossed 60,000 views.  How cool is that?  Sure, that's small potatoes today with billions of people online, and I'm sure a blog with cats or bikinis would do that many views in a day...but in the absence of clickbait on my site, you can be assured that everyone reaching my blog is an engineer of some kind, looking for technical answers.

It's a very small world of data storage engineers and they make huge purchasing decisions.  That so many in this small market are landing on my blog and learning NetApp's technology here is pretty awesome.