SAP Project

A lot of lessons learned in a SAP project I've lead over the past few months.  Here's the environment:

• 7-Mode 8.1.3 FAS6220's with 2TB Flash Cache and SAS disks
• DB2 on AIX 7.1 on IBM P-Series
• SAP on SUSE Linux
• Migrating from HDS FCP to NetApp NFS

A really cool part of this: In their QA environment, they had 6 full copies of every database, taking a 30TB production environment to 200TB used in QA.  This also means that refreshes required a full re-copy of the database, which had performance impact.  Using FlexClones (thin, writable snapshots), we dropped QA capacity to 100TB used and negated the performance impact entirely.

Now that we have the basics laid out, here are a few important things we figured out.  About AIX:

1. Turn on Selective ACKS in AIX.  This was a huge performance improvement for us.
2. We saw a 40% throughput improvement upgrading from AIX 6.1 to AIX 7.1  Highly recommend it.
3. Be careful with AIX LACP etherchannels.  We saw some very strange throughput drops, almost like port flapping, on a LPAR using a LACP etherchannel.  The client saw some errors related to it, recreated it, and we saw significant performance improvements.
4. We didn't see any improvement using mount options like CIO or RBR.
5. We settled on these mount options:  bg,hard,intr,rsize=65536,wsize=65536,timeo=600,vers=3,proto=tcp,rw

About DB2:

1. Our migration plan (from HDS to NetApp) was using DB2 backup/restore.  This avoids slow log rolling and is the safest route for data integrity. 
2. Make sure you distribute your DB2 datafiles onto multiple controllers and multiple volumes.  Think of each volume as a thread: the more threads, the better CPU parallel-ization, the better your performance.
3. When performing a DB2 backup and restore, make sure you backup and restore to multiple controllers and multiple volumes: same reason as above.
4. In order to reduce the number of SnapMirror relationships and improve the RPO, we combined all Transaction Logs and Archive Logs into 1 volume each per controller.  We put 

About FlexClones:

1. When you FlexClone from a SnapVault destination snapshot, it inherits the SnapVault relationship.  You resolve this by breaking that relationship and restoring the qtree.
2. In 7-mode, you can't FlexClone from a SnapMirror destination snapshot.

About SnapMirror:

1. Single-threaded SnapMirror relationships can have a window size of 7MB, 14MB for multi-threaded.  Very good for long distance Snapmirror.
2. You can multi-thread a SnapMirror relationship.
3. Keep an eye on the SnapMirror Maximums.
4. With a logs change rate of ~10MB/s per controller, we were able to accomplish a 1-minute SnapMirror interval over a 38ms RTT WAN.

More to come!

NetApp Experience: Networking

For reasons unknown, a disproportionate amount of the trouble tech people run into is network.  I encountered some of these issues recently that forced me to take a close look at the specifics of ONTAP networking.  Here are some important details:

• Make sure the switch ports are hard-coded for LACP if you are creating LACP interface groups (vif's) in ONTAP.
• These are the options for status:
• Up: the link is sending and receiving data.
• Down: the link is down but believed to be operational.
• Broken: the link is inactive and believed to be non-operational.

Here's a problem I ran into, see if you can spot the issue.  Upon boot, ONTAP presents these messages:

vif: Cannot create a multi-level 802.3ad compliant vif: vif1
vif: Cannot create a multi-level 802.3ad compliant vif: vif2
vif: vif1 cannot create multi level 802.3ad vif
vif: Failure adding vif1. Continuing with other interfaces
vif: vif2 cannot create multi level 802.3ad vif
vif: Failure adding vif2. Continuing with other interfaces

Here's some more hints:

rdfile /etc/rc
#Auto-generated by setup Thu Jul 14 19:22:30 GMT 2011
hostname NAME
vif create lacp vif1 -b ip e0a e0b
vif create lacp vif2 -b ip e0c e0d
vif create lacp supervif1 -b ip vif1 vif2
ifconfig supervif1 `hostname`-supervif1 mediatype auto
netmask 255.255.255.0
route add default 10.18.33.1 1
routed on
options dns.domainname acme.corp.com
options dns.enable on
options nis.enable off
savecore

ifconfig -a
e0a: flags=0xa508866 mtu 1500

        ether 00:00:00:00:00:00 (auto-100tx-fd-cfg_down) flowcontrol full
        trunked vif1
e0b: flags=0xa508866 mtu 1500
        ether 00:00:00:00:00:00 (auto-unknown-cfg_down) flowcontrol full
        trunked vif1
e0c: flags=0xa508866 mtu 1500
        ether 00:00:00:00:00:00 (auto-unknown-cfg_down) flowcontrol full
        trunked vif2
e0d: flags=0xa508866 mtu 1500
        ether 00:00:00:00:00:00 (auto-unknown-cfg_down) flowcontrol full
        trunked vif2
lo: flags=0x1948049 mtu 9188
        inet 127.0.0.1 netmask-or-prefix 0xff000000 broadcast 127.0.0.1
vif1: flags=0x22408862 mtu 1500
        ether 00:00:00:00:00:00 (Disabled virtual interface)
vif2: flags=0x22408862 mtu 1500
        ether 00:00:00:00:00:00 (Disabled virtual interface)
supervif1: flags=0x2354b863 mtu 1500
        inet 10.18.33.33 netmask-or-prefix 0xffffff00 broadcast 10.18.33.255
        ether 02:a0:98:2c:01:9c (Disabled virtual interface)

Solution: You can't create multi-level multi-mode or LACP vifs.  Essentially, if you have two vifs that are load balanced, you can't create a load balanced vif out of those two vifs.  This is documented as an unsupported configuration by netapp.

https://kb.netapp.com/support/index?page=content&id=3011251 (requires NetApp login)

So what to do?  Simple!  Just create one big multi or LACP vif out of the ports you wanted to use in the first place.

NetApp Training Brain Dump: Terms and Acronyms

Here is a list of useful acronyms and definitions.  The list is seriously incomplete, I'll keep it updated as I learn.  These definitions are not meant to be exhaustive, but are meant to be concise and accurate to give you the general idea, in plain english, quickly.

Terms:

1. Anodefile: Haven't defined this yet.
2. The RLM/BMC/SP have essentially the same functionality.  They are control modules for management of the device, giving you remote console access in case other connections go down.  RLM is the oldest, SP the newest version of this module.
3. FlexClone: Copy of an existing volume.  Looks like a volume, acts like a volume, takes up no space until you change something from the original.  R/W.  Basically a writable snapshot.
4. Snapshot: Point in time copy of an existing volume.  Cannot be changed.  Snapshots take up no space until data is changed on the  original, because it's really just a bunch of pointers that are still pointing toward the original blocks.
5. Fingerprint database:
6. Deswizzling: background WAFL scanner establishing the relationship between the PVBNs and VVBNs.  Only impacts destination, only impacts volumes.

Acronyms:

1. Product designations*2
1. Denotes single CPU Module (Controller):
1. FAS CI (Controller/IOXM): One controller, one IOXM, one chassis.  IOXM adds PCI slots to support additional ports.
2. FAS CB (Controller/Blank): Self explanatory.
3. FAS E (Expansion): One controller, one IOXM, one chassis. 
2. Denotes only HA (dual) CPU Modules: 
1. FAS CC (Controller/Controller): Two controllers, one chassis
2. FAS c (Controller/Controller): Two controllers, one chassis
3. FAS AE (Active/Expansion): Two chassis set up for an HA pair.  One controller, one IOXM per chassis.
4. FAS A (Active/Active): Indicates the same thing as CC.
2. VIF (Virtual Interface):  virtual NIC, known as Trunked or Teamed NIC.  They come in single mode (redundancy) or multi mode (load balanced).  There are two types of Multi-Mode VIFs, Static Etherchannel and LACP.*1
3. Static Etherchannel: older protocol for combining  NIC's into a single virtual NIC.  Load balances just as well as LACP.
4. LACP (Link Aggregation Control Protocol): Newer protocol for combining  NIC's into a single virtual NIC.     This is an enhanced option over Static Etherchannel because of better error detection and handling.  Pick LACP when possible.
5. ACP (Alternate Control Path): Backup path for the CPU modules to control the shelf modules.
6. Wiregauge: software that tests whether a FAS system is correctly wired for MPIO/HA.
7. NGS: NetApp Global Support.
8. NRD (non-return disks): Client has paid for the right to keep disks after they've failed.  Never take these offsite.
9. IOXM (Input/Output Expansion Module): Module that goes in the place of a CPU module in a FAS system.  Provides more PCI slots for network connectivity.
10. IOM3/IOM6 (Input/Output module): Redundant shelf module for DS42XX series.  
11. FC-AL (Fibre Channel-Arbitrated Loop):
12. FC-VI (Fibre Channel-Virtual Interface):
13. TOI (Transfer of information).  
14. IMAC (Install, Add, Move, Change).
15. NDMP: Network Data Management Protocol.  This protocol sets up communication between the NAS device (e.g. Filer) and your backup device (e.g. tape library), bypassing the backup server.  Typically the backup server is running enterprise backup software to facilitate the exchange, but doesn't want to be the middleman in the flow of data.  You can think of the backup server as the witness in a duel, and the code of honor each dueler follows as NDMP.  When the witness says go, take 10 steps, turn and shoot - very important, but the witness doesn't want to stand in between the two and pass on the bullets :-)
16. WWPN or WWN (World Wide Port Name): In a SAN, these unique names are used at a software level to route data to and from the correct ports.  Each port has a unique name, ostensibly unique in the world
17. NPIV (N_Port ID Virtualization): How multiple 
1. For more info on this, check out http://blog.scottlowe.org/2009/11/27/understanding-npiv-and-npv/
18. RDM (Raw  Device Mapping): A term for presenting the LUN to the server via SCSI/FCoE/FCP.
19. VMFS (Virtual Machine File System): VMware's cluster file system.
20. NDU (Non-Disruptive Upgrades): refers to whether a firmware upgrade takes down the service.
21. PVBN (Physical Volume Block Numbers): how WAFL identifies the blocks of data.  Essentially an address.
22. VVBN (Virtual Volume Block Numbers): how WAFL identifies the blocks to mirror changes.  Each block has the same VVBN on the source and dest volume. 
23. FUD (Fear, Uncertainty, Doubt): how small minded people justify avoiding work.
24. QSM (Qtree SnapMirror)
25. RTO (Recovery Time Objective): How long it takes to failover and be up from a disaster.
26. RPO (Recovery Point Objective): How much data the system will lose in the event of a disaster.  Zero RPO means no data lost.
27. SDS (Storage Design Studio): Software within NetApp Dynamics that is used to design a SAN, including network, disk, and volume levels.
28. ALUA (Asymmetric Logical Unit Access): Since only one controller at a time owns and writes to each hard drive, ALUA software uses awareness of this to send traffic to the owning controller.  This optimizes performance.
29. LREP (Logical Replication): Used for the first, full transfer of data for either replication of backup.  
30. ASIS (Advanced Single Instance Storage)*3: NetApp's old name for data dedupe.
31. RBAC (Role Based Access Control)
32. RAS (Reliability, Availability, and Serviceability)

Positions (loose definitions):

- TPM (Third Party Maintainer): 3rd party NetApp contracted break/fix/install.

- ASE (Accredited Services Engineer):  Somewhat analogous to TPM.

- FSE (Field Services Engineer): Third party firefighter.  Expert level ASE.

- PSE (Professional Services Engineer): Consultant expert for implementations and base software like Operations Manager.
- TSE (Technical Support Engineer): The guys at NGS.  

- PSC (Professional Services Consultant): Consultant guru, highly specialized expert.  Architect.

Sources

*1.  http://communities.netapp.com/message/8185?tstart=898
*2.  http://www.netapp.com/us/communities/tech-ontap/tot-fas3200-boost-1101.html
*3.  http://www.backupcentral.com/mr-backup-blog-mainmenu-47/13-mr-backup-blog/250-netapp-asis.html