CDOT Tips: Network Troubleshooting

Here are some essential commands for digging into your CDOT network configuration:

How to show the status of your physical and virtual ports:

>net port show –node node1

Node   Port   Role         Link   MTU Admin/Oper  Admin/Oper Admin/Oper
------ ------ ------------ ---- ----- ----------- ---------- ------------
eg-si-clsn-e01-h02
       a0a    data         up    1500  true/-     auto/full   auto/10000
       a0a-2003
              data         up    1500  true/-     auto/full   auto/10000
       a0b    data         down  9000  true/-     auto/-      auto/-
       e0M    node-mgmt    up    1500  true/true  full/full   auto/1000
       e0a    cluster      up    9000  true/true  full/full   auto/10000
       e0b    data         up    1500  true/true  full/full   auto/10000
       e0c    cluster      up    9000  true/true  full/full   auto/10000
       e0d    data         up    1500  true/true  full/full   auto/10000

How to show detailed info of a specific port:

>net port show –node node1 –port a0b

                         Port: a0b
                           Role: data
                           Link: down
                            MTU: 9000
Auto-Negotiation Administrative: true
   Auto-Negotiation Operational: -
     Duplex Mode Administrative: auto
        Duplex Mode Operational: -
           Speed Administrative: auto
              Speed Operational: -
    Flow Control Administrative: full
       Flow Control Operational: -
                    MAC Address: 02:a0:98:5b:40:17
              Up Administrative: true
                      Port Type: if-group
    Interface Group Parent Node: -
    Interface Group Parent Port: -
          Distribution Function: ip
                  Create Policy: multimode_lacp
               Parent VLAN Node: -
               Parent VLAN Port: -
                       VLAN Tag: -
               Remote Device ID: -

How to show the MAC addresses of all your ports:

>port show -node  node1 -fields mac

node               port mac
------------------ ---- -----------------
eg-si-clsn-e01-h02 a0a  02:a0:98:5b:40:16
eg-si-clsn-e01-h02 a0a-2003 02:a0:98:5b:40:16
eg-si-clsn-e01-h02 a0b  02:a0:98:5b:40:17
eg-si-clsn-e01-h02 e0M  00:a0:98:5b:40:2a

Note that when you create an ifgrp in 8.2.x, the member ports will inherit the MAC address of the ifgrp.  E.g., a0a will have the same MAC as e0a and e0c. 

How to ping from a specific LIF:

  If you use the ping command, there’s no obvious way to see which port or LIF it’s emitting from.  You can use this command to specify which node or LIF you want to test:

>Network ping –node node1 192.168.10.10

>Network ping –lif-owner vserver1 –lif nfs_data_lif

There are also some cool options like number of packets, allow fragmentation, etc.

https://library.netapp.com/ecmdocs/ECMP1366832/html/network/ping.html

How to turn up/down a port:

>set advanced

> network port modify –node node1 –port e0a –up-admin true

How to list who your port is connected to:

>node run -node node1 -command cdpd show-neighbors

CDOT Tip #3

Let’s tackle some networking!  Networking is admittedly not my strong suit, so please pepper me with questions if you see something amiss.  Some of these are general recommendations that may not be applicable to you, but they’re good to at least have reference to.

• Remember, each lif type needs a routing group (mgmt, data, etc)
• If you create a temporary IP address it may create a temporary routing group.  Make sure you go back and clean it up.
• Remember to create the ifgrp before your lifs.  It’s a pain to go back!
• If the switch port is type access, our ifgrps can’t have vlans. We recommend using switch port type trunk even if there’s only 1 vlan to allow for future flexibility
• switchport trunk encapsulation dot1q
• Portfast on
• Disable IP fastpath
• ::> node run -node * -command "options nodescope.reenabledoptions ip.fastpath"
• ::> node run -node * -command options ip.fastpath.enable off
• Disable flow control on all non-Unified Target Adapter (UTA) network interfaces and their associated switch ports
• ::> net port modify -node -port   -flowcontrol-admin none
• Create per-network/VLAN failover groups and modify network interface failover-group setting accordingly
• ::> failover-groups create -failover-group -node -port [-vlan_id]
• ::> network interface modify -vserver -lif -failover-group
• You can do a net int show and use –fields pick field names (like routing-group) that aren’t showed by default.  Very useful
• You can choose which lif to use when pinging.  This is a fantastic testing tool!  net ping -lif-owner svm -lif smlif -destination gwaddress
• You can’t create a 2-node cluster unless the cluster network is up.  So if you’re setting up a switchless cluster, make sure you connect from each 10Gb port directly to the other node’s IC port.
• If you’re setting up a switchless cluster, you need to follow these instruction on both nodes.
• ::> set advanced      (y) 
• ::*> network options switchless-cluster modify -enabled true

Bonus tip!

If you need to halt one controller without impacting the HA partner, there is no longer a cf disable option.  Use halt with an inhibit takeover switch (there’s also a restart inhibit takeover switch).

Clustered ONTAP Networking

Basic

• ifgrps are made up of physical ports.  ifgrps are to be named "a," e.g. a0a, a0b.
• All members of a ifgrp must have the same role set (data, mgmt)
• Cluster ports cannot be in ifgrps
• A port with its own failover designation cannot be added to a ifgrp
• All ports in an ifgrpmust be on the same physical node
• A lif (logical interface group) is basically the IP addresses assigned to an ifgrp.  That IP address entity is given other properties like "home port," VLAN, failover group, etc.  Each IP needs to be able to have these properties independent of other IP's.
• Spanning tree should not be enabled on the switch for any ports connecting to the controllers except the management ports.
• It is a best practice to set flowcontrol to none for all ports except UTA ports.
• UTA ports set to full.
• You can of course add VLAN tagging and multiple IP's per LIF, named "-," e.g. a0b-441
• Each LIF gets a routing group that you may or may not want to alter.  
• You'll need to create a default routing group (default per vserver) for LIF's to default to.

Once you have those concepts figured out, you can move on to the advanced.

• For snapmirror/snapvault, each node needs its own "intercluster LIF" to identify which ports and IP to be used for replication.
• You'll need inter-cluster routing groups on each node to set up replication.
• You have to create a cluster peer to establish a relationship for replication.
• Network Interface Failover Groups are simply the cluster-wide list of ifgrps whose LIFs can migrate over to each other.  For example, you don't want your 10Gb NFS LIF failing over to another controller's 1Gb port ifgrp.
• Your node management LIF can't fail over to other nodes (obviously, it's the portal to your physical node).  So it should have its own failover group that contains only local ports.
• You want to be cognizant of not including e0M on any failover groups with data ports.

Some examples of real life setups:

• A vserver (virtual storage machine) that was strictly SAN would need only one IP address, assigned to a physical port for management.  

NetApp Insights: Ifgrp

Few cool points on ifgrps:

-  Can you add a new port to an existing ifgrp?  Yes!  If the new port is not connected to something though, the it must be configured down at first.  It will come up in the ifgroup when you've plugged it in.  ifgrp add   .  
-  Can you remove a port from an existing ifgrp? Yes!  Not live though.  You must first configure the ifgrp down, and then you can ifgrp delete .  
-  How many switches can a ifgrp span?  I don't know.
-  Can you add a second IP to an existing vFiler?  Yes!  But each vFiler can live in only one IPspace.