NetApp Insights: Ifgrp

Few cool points on ifgrps:

-  Can you add a new port to an existing ifgrp?  Yes!  If the new port is not connected to something though, the it must be configured down at first.  It will come up in the ifgroup when you've plugged it in.  ifgrp add   .  
-  Can you remove a port from an existing ifgrp? Yes!  Not live though.  You must first configure the ifgrp down, and then you can ifgrp delete .  
-  How many switches can a ifgrp span?  I don't know.
-  Can you add a second IP to an existing vFiler?  Yes!  But each vFiler can live in only one IPspace.

NetApp Experience: Networking

For reasons unknown, a disproportionate amount of the trouble tech people run into is network.  I encountered some of these issues recently that forced me to take a close look at the specifics of ONTAP networking.  Here are some important details:

• Make sure the switch ports are hard-coded for LACP if you are creating LACP interface groups (vif's) in ONTAP.
• These are the options for status:
• Up: the link is sending and receiving data.
• Down: the link is down but believed to be operational.
• Broken: the link is inactive and believed to be non-operational.

Here's a problem I ran into, see if you can spot the issue.  Upon boot, ONTAP presents these messages:

vif: Cannot create a multi-level 802.3ad compliant vif: vif1
vif: Cannot create a multi-level 802.3ad compliant vif: vif2
vif: vif1 cannot create multi level 802.3ad vif
vif: Failure adding vif1. Continuing with other interfaces
vif: vif2 cannot create multi level 802.3ad vif
vif: Failure adding vif2. Continuing with other interfaces

Here's some more hints:

rdfile /etc/rc
#Auto-generated by setup Thu Jul 14 19:22:30 GMT 2011
hostname NAME
vif create lacp vif1 -b ip e0a e0b
vif create lacp vif2 -b ip e0c e0d
vif create lacp supervif1 -b ip vif1 vif2
ifconfig supervif1 `hostname`-supervif1 mediatype auto
netmask 255.255.255.0
route add default 10.18.33.1 1
routed on
options dns.domainname acme.corp.com
options dns.enable on
options nis.enable off
savecore

ifconfig -a
e0a: flags=0xa508866 mtu 1500

        ether 00:00:00:00:00:00 (auto-100tx-fd-cfg_down) flowcontrol full
        trunked vif1
e0b: flags=0xa508866 mtu 1500
        ether 00:00:00:00:00:00 (auto-unknown-cfg_down) flowcontrol full
        trunked vif1
e0c: flags=0xa508866 mtu 1500
        ether 00:00:00:00:00:00 (auto-unknown-cfg_down) flowcontrol full
        trunked vif2
e0d: flags=0xa508866 mtu 1500
        ether 00:00:00:00:00:00 (auto-unknown-cfg_down) flowcontrol full
        trunked vif2
lo: flags=0x1948049 mtu 9188
        inet 127.0.0.1 netmask-or-prefix 0xff000000 broadcast 127.0.0.1
vif1: flags=0x22408862 mtu 1500
        ether 00:00:00:00:00:00 (Disabled virtual interface)
vif2: flags=0x22408862 mtu 1500
        ether 00:00:00:00:00:00 (Disabled virtual interface)
supervif1: flags=0x2354b863 mtu 1500
        inet 10.18.33.33 netmask-or-prefix 0xffffff00 broadcast 10.18.33.255
        ether 02:a0:98:2c:01:9c (Disabled virtual interface)

Solution: You can't create multi-level multi-mode or LACP vifs.  Essentially, if you have two vifs that are load balanced, you can't create a load balanced vif out of those two vifs.  This is documented as an unsupported configuration by netapp.

https://kb.netapp.com/support/index?page=content&id=3011251 (requires NetApp login)

So what to do?  Simple!  Just create one big multi or LACP vif out of the ports you wanted to use in the first place.

NetApp Training Brain Dump: Useful Links

Netboot basics:
http://www.wafl.co.uk/netboot-a-netapp/

Max drives for the 16TB aggregate limit:
http://now.netapp.com/NOW/knowledge/docs/ontap/rel73/html/ontap/rnote/rel_notes/reference/r_oc_rn_feat73_aggr-size-max-drives.html

VIF's Explained:
http://netappsky.com/netapp/netapp-vif-survival/

Fibre Cable Labels explained (translate to english at top right):
http://www.fcoe.ru/index.php?option=com_content&task=view&id=312&Itemid=46#addcomments

NetApp Experience: Think on your feet (1)

Volatile /etc/rc file (CL)

This was an interesting one.  When you make a change to the configuration on a NetApp system, it will take effect immediately.  The important thing to realize is that it won’t be permanent unless you save this configuration (by making an identical change to the /etc/rc file), which effects the change to the /etc/rc file.  Unsaved changes are reverted back to the pre-change state any time the memory is cleared, e.g. reboot, power off, etc.  For this customer, we were hot adding*1 expansion FC PCI cards to add a couple stacks of shelves, and literally walked onto a landmine.  

What had happened is the customer had made significant changes to the network settings on the system but not saved them.  When we brought the first CPU module back up, the customer found that it was unresponsive although we could find no problem with it.  The customer and tech lead made the decision to move forward with the change to the second CPU module, at which point the entire system became unresponsive.  This is because the changes the customer had made to the network settings were completely reverted upon reboot, causing a 15 minute outage while we tracked down the problem.

This problem was particularly tough to decipher because there was nothing wrong with the actual system – the issue was invisible to anyone but the admin who had made the changes, who was not on site.  

Take aways: 

- Definitely take a look at the /etc/rc file and make sure it lines up with the current settings.

- Possibly start off by saving the current configuration and backing it up.  I’ll have to look into the pros and cons on this – anyone with thoughts feel free to add in the comments.

*1 A hot add of PCI cards isn’t really a hot add, since it requires you to fail over and shut down one of the CPU modules at a time.  This does require a small outage (30-120s) for the fail back – the failover is just a blip.

NetApp Training Brain Dump: Terms and Acronyms

Here is a list of useful acronyms and definitions.  The list is seriously incomplete, I'll keep it updated as I learn.  These definitions are not meant to be exhaustive, but are meant to be concise and accurate to give you the general idea, in plain english, quickly.

Terms:

1. Anodefile: Haven't defined this yet.
2. The RLM/BMC/SP have essentially the same functionality.  They are control modules for management of the device, giving you remote console access in case other connections go down.  RLM is the oldest, SP the newest version of this module.
3. FlexClone: Copy of an existing volume.  Looks like a volume, acts like a volume, takes up no space until you change something from the original.  R/W.  Basically a writable snapshot.
4. Snapshot: Point in time copy of an existing volume.  Cannot be changed.  Snapshots take up no space until data is changed on the  original, because it's really just a bunch of pointers that are still pointing toward the original blocks.
5. Fingerprint database:
6. Deswizzling: background WAFL scanner establishing the relationship between the PVBNs and VVBNs.  Only impacts destination, only impacts volumes.

Acronyms:

1. Product designations*2
1. Denotes single CPU Module (Controller):
1. FAS CI (Controller/IOXM): One controller, one IOXM, one chassis.  IOXM adds PCI slots to support additional ports.
2. FAS CB (Controller/Blank): Self explanatory.
3. FAS E (Expansion): One controller, one IOXM, one chassis. 
2. Denotes only HA (dual) CPU Modules: 
1. FAS CC (Controller/Controller): Two controllers, one chassis
2. FAS c (Controller/Controller): Two controllers, one chassis
3. FAS AE (Active/Expansion): Two chassis set up for an HA pair.  One controller, one IOXM per chassis.
4. FAS A (Active/Active): Indicates the same thing as CC.
2. VIF (Virtual Interface):  virtual NIC, known as Trunked or Teamed NIC.  They come in single mode (redundancy) or multi mode (load balanced).  There are two types of Multi-Mode VIFs, Static Etherchannel and LACP.*1
3. Static Etherchannel: older protocol for combining  NIC's into a single virtual NIC.  Load balances just as well as LACP.
4. LACP (Link Aggregation Control Protocol): Newer protocol for combining  NIC's into a single virtual NIC.     This is an enhanced option over Static Etherchannel because of better error detection and handling.  Pick LACP when possible.
5. ACP (Alternate Control Path): Backup path for the CPU modules to control the shelf modules.
6. Wiregauge: software that tests whether a FAS system is correctly wired for MPIO/HA.
7. NGS: NetApp Global Support.
8. NRD (non-return disks): Client has paid for the right to keep disks after they've failed.  Never take these offsite.
9. IOXM (Input/Output Expansion Module): Module that goes in the place of a CPU module in a FAS system.  Provides more PCI slots for network connectivity.
10. IOM3/IOM6 (Input/Output module): Redundant shelf module for DS42XX series.  
11. FC-AL (Fibre Channel-Arbitrated Loop):
12. FC-VI (Fibre Channel-Virtual Interface):
13. TOI (Transfer of information).  
14. IMAC (Install, Add, Move, Change).
15. NDMP: Network Data Management Protocol.  This protocol sets up communication between the NAS device (e.g. Filer) and your backup device (e.g. tape library), bypassing the backup server.  Typically the backup server is running enterprise backup software to facilitate the exchange, but doesn't want to be the middleman in the flow of data.  You can think of the backup server as the witness in a duel, and the code of honor each dueler follows as NDMP.  When the witness says go, take 10 steps, turn and shoot - very important, but the witness doesn't want to stand in between the two and pass on the bullets :-)
16. WWPN or WWN (World Wide Port Name): In a SAN, these unique names are used at a software level to route data to and from the correct ports.  Each port has a unique name, ostensibly unique in the world
17. NPIV (N_Port ID Virtualization): How multiple 
1. For more info on this, check out http://blog.scottlowe.org/2009/11/27/understanding-npiv-and-npv/
18. RDM (Raw  Device Mapping): A term for presenting the LUN to the server via SCSI/FCoE/FCP.
19. VMFS (Virtual Machine File System): VMware's cluster file system.
20. NDU (Non-Disruptive Upgrades): refers to whether a firmware upgrade takes down the service.
21. PVBN (Physical Volume Block Numbers): how WAFL identifies the blocks of data.  Essentially an address.
22. VVBN (Virtual Volume Block Numbers): how WAFL identifies the blocks to mirror changes.  Each block has the same VVBN on the source and dest volume. 
23. FUD (Fear, Uncertainty, Doubt): how small minded people justify avoiding work.
24. QSM (Qtree SnapMirror)
25. RTO (Recovery Time Objective): How long it takes to failover and be up from a disaster.
26. RPO (Recovery Point Objective): How much data the system will lose in the event of a disaster.  Zero RPO means no data lost.
27. SDS (Storage Design Studio): Software within NetApp Dynamics that is used to design a SAN, including network, disk, and volume levels.
28. ALUA (Asymmetric Logical Unit Access): Since only one controller at a time owns and writes to each hard drive, ALUA software uses awareness of this to send traffic to the owning controller.  This optimizes performance.
29. LREP (Logical Replication): Used for the first, full transfer of data for either replication of backup.  
30. ASIS (Advanced Single Instance Storage)*3: NetApp's old name for data dedupe.
31. RBAC (Role Based Access Control)
32. RAS (Reliability, Availability, and Serviceability)

Positions (loose definitions):

- TPM (Third Party Maintainer): 3rd party NetApp contracted break/fix/install.

- ASE (Accredited Services Engineer):  Somewhat analogous to TPM.

- FSE (Field Services Engineer): Third party firefighter.  Expert level ASE.

- PSE (Professional Services Engineer): Consultant expert for implementations and base software like Operations Manager.
- TSE (Technical Support Engineer): The guys at NGS.  

- PSC (Professional Services Consultant): Consultant guru, highly specialized expert.  Architect.

Sources

*1.  http://communities.netapp.com/message/8185?tstart=898
*2.  http://www.netapp.com/us/communities/tech-ontap/tot-fas3200-boost-1101.html
*3.  http://www.backupcentral.com/mr-backup-blog-mainmenu-47/13-mr-backup-blog/250-netapp-asis.html

NetApp Training Brain Dump: ONTAP Commands and System Manager Setup

ONTAP syntax explained:
Symbols:
| means "Or"
[] means "Command."  These are optional.
{x | y} means pick one argument, x or y (required).
< > indicates input argument required  here.

Simple Example:  
ONTAP> command1 [ -a | command2 <address> | command3 {optionA | optionB}]

This would mean command1 is required, and from there you have 3 options: -a, command2, or command3.
If you go with -a, that's all you can type.
Example: ONTAP> command1 -a

If you go with command2, you need to input an address in the correct format or else the command fails.
Example: ONTAP> command1 command2 10.10.10.192

If you go with command3, you need to select from your preset options.
Example: ONTAP> command1 command3 optionB

Advanced Example:
(HINT: a new line does not necessarily indicate any syntax.  Watch the brackets.)
ONTAP> command1 [-a | command2 [ 
    [command3 {optionA | optionB}] [command4 <input>]
    [command5 {optionC | optionD}]]
    [command6 {optionE | optionF}]] 


 In this case, note that command 5 is the last command you can add to command2.
Example of a valid command: ONTAP> command1 command2 thisobject command6 15

To use command6, you need to add it to command1.  You can follow this based upon the red brackets []
TIP: When trying to zero out a setting, try using a space where the argument should be.


System Manager Setup: 
Basic notes here:
1. Don't use e0p for this, as it is generally reserved for ACP.  Pick an open data ethernet port.
2. Configure a NIC or VIF
3.  Enable SNMP:
  a. options snmp.enable on
  b. Verify community
4. Enable SSL:
   a. options ssl.enable on
   b. secureadmin setup ssl
5.  Install System Manager on machine.
6.  Add storage system from System Manager.

 If you receive "API invoke failed" messages, you likely did not enable SSL correctly.