SAP Project

A lot of lessons learned in a SAP project I've lead over the past few months.  Here's the environment:

• 7-Mode 8.1.3 FAS6220's with 2TB Flash Cache and SAS disks
• DB2 on AIX 7.1 on IBM P-Series
• SAP on SUSE Linux
• Migrating from HDS FCP to NetApp NFS

A really cool part of this: In their QA environment, they had 6 full copies of every database, taking a 30TB production environment to 200TB used in QA.  This also means that refreshes required a full re-copy of the database, which had performance impact.  Using FlexClones (thin, writable snapshots), we dropped QA capacity to 100TB used and negated the performance impact entirely.

Now that we have the basics laid out, here are a few important things we figured out.  About AIX:

1. Turn on Selective ACKS in AIX.  This was a huge performance improvement for us.
2. We saw a 40% throughput improvement upgrading from AIX 6.1 to AIX 7.1  Highly recommend it.
3. Be careful with AIX LACP etherchannels.  We saw some very strange throughput drops, almost like port flapping, on a LPAR using a LACP etherchannel.  The client saw some errors related to it, recreated it, and we saw significant performance improvements.
4. We didn't see any improvement using mount options like CIO or RBR.
5. We settled on these mount options:  bg,hard,intr,rsize=65536,wsize=65536,timeo=600,vers=3,proto=tcp,rw

About DB2:

1. Our migration plan (from HDS to NetApp) was using DB2 backup/restore.  This avoids slow log rolling and is the safest route for data integrity. 
2. Make sure you distribute your DB2 datafiles onto multiple controllers and multiple volumes.  Think of each volume as a thread: the more threads, the better CPU parallel-ization, the better your performance.
3. When performing a DB2 backup and restore, make sure you backup and restore to multiple controllers and multiple volumes: same reason as above.
4. In order to reduce the number of SnapMirror relationships and improve the RPO, we combined all Transaction Logs and Archive Logs into 1 volume each per controller.  We put 

About FlexClones:

1. When you FlexClone from a SnapVault destination snapshot, it inherits the SnapVault relationship.  You resolve this by breaking that relationship and restoring the qtree.
2. In 7-mode, you can't FlexClone from a SnapMirror destination snapshot.

About SnapMirror:

1. Single-threaded SnapMirror relationships can have a window size of 7MB, 14MB for multi-threaded.  Very good for long distance Snapmirror.
2. You can multi-thread a SnapMirror relationship.
3. Keep an eye on the SnapMirror Maximums.
4. With a logs change rate of ~10MB/s per controller, we were able to accomplish a 1-minute SnapMirror interval over a 38ms RTT WAN.

More to come!

SnapMirror vs SnapVault in CDOT

 In CDOT we’ve simplified the commands and protocol: to create a snapvault relationship, you run “SnapMirror create –type XDP.”  To create a SnapMirror relationship, you run “SnapMirror create –type DP.”  In fact, for CDOT we no longer call it snapvault, it is called XDP SnapMirror. 

e.g., vs2::> snapmirror create -destination-path vs2:dept_eng_dp_mirror2 -source-path vs1:dept_eng -type DP

The primary distinction between DP snapmirror and XDP SnapMirror is that snapvault allows you to keep more snapshots on the destination.  Essentially, XDP snapmirror is for long-term backups.  Other differences:

·         DP SnapMirror relationships can be reversed (swap destination and source)

·         DP SnapMirror can replicate every minutes, XDP SnapMirror once per hour.

·         DP SnapMirror destination volumes can be made read/write.

Do you have some datasets that would benefit from a quicker time to recovery or have stricter SLA’s?  If so, DP SnapMirror is the best choice.  

NetApp All Flash FAS!

Some great notes on designing an AFF:

1. Don't mix any other disk type with an AFF (hence, all flash) in the HA Pair.  Feel free to mix within the cluster though!
2. Don't use Flash Cache with AFF
3. It's perfectly OK to put the root aggregate on SAS drives.
4. Flash disks have the same MTBF as SAS disks, and expected life of 5 years. 

And of course, some performance data:

Documents: TR-4289, "All Flash FAS Technical FAQ"

Datacenter Migration Project: Lessons Learned

Some note I wanted to save for posterity on a datacenter migration project I lead several years ago:

What we did well:

• Everything was set into the racks correctly on the first try (pretty incredible if you think about it).
• Everyone arrived to the pre-meeting and customer sites on time (started at midnight).
• Communication with the customer was consistent.
• We got the backup filer up before 7am!
• We got the equipment all up and handed over by 4:30pm  (our expected was 4pm and deadline was 7pm)
• We handled several mini-crises in stride:
• Motherboard death
• Loop “login delay” issue
• FC cable shortage
• Loop combination

What we need to remember next time:

• Starting at midnight and going all day is very, very different from starting at 8am and going all day.
• It’s important for a team to be familiar with one another and how the work tasks will flow.
• It’s important to have a good blend of experienced and inexperienced team members.
• Fly the team into town the day before – You can’t expect someone to travel all day and then work all night.
• Make sure the team understands how to read the documentation before you get onsite.
• Check to see if the team is experienced with the specific technologies relevant.
• Have a pre-job meeting (preferably with pizza) to explain expectations and game plan.
• Make sure everyone is in the same hotel, close to the jobsite.
• Make sure there is food/drink arriving regularly for the guys who are working.
• Add an extra guy to do physical work if you have someone supervising/project managing.
• Drills are important.
• Get all of the rails into the racks beforehand if possible.
• Plan to combine loops/stacks if the system is complex and spread out.  Don’t underestimate how long it takes to cable per loop!
• Cabling loops between racks is significantly more time consuming.
• Don’t expect to be able to salvage any/all cables that run under/over racks.  Was a total rat’s nest.
• Have a rested, standby team member to handle hardware failures/support issues 
• Make sure you’re aware of the plan for switches/patch panels that are in-rack, and for disconnecting the PDU’s.

Performance Case

Just a performance case I worked recently.  These kinds of things can be instructive for people with similar problems, or those who learn by playing along at home.  Here's my email, sorry for the copy paste without context.

 Please take a look at the email chain below as the starting point for this conversation.  At this point, we’ve reviewed four sets of performance data gathered over the last two months and have closely correlated a spike in large-IOP-size traffic to our latency spikes.  This spike is both in number and size of IOPS, exceeding 32,000 IOPS for 15+ minutes at a time.  There is no single volume driving the traffic, as it appears to be increasing dramatically across the board.  

Here is a summary of the performance data from Thursday 4/3, please note the IOP ramp up and associated latency:

Start Time	CPU Busy	NFS Op/s	Read Op/s	Read Lat (ms)	Write Op/s	Write Lat (ms)	Net Sent (MB/s)	Net Recv (MB/s)
9:53p	52	7,127	1,256	2.32	5,731	0.63	27	50
9:57p	67	13,637	5,396	19.96	8,076	110.09	145	81
10:05p	99	31,119	17,272	22.91	13,697	341.18	371	205
10:13p	99	32,311	22,519	8.4	9,739	229.45	621	200
10:24p	99	23183	12,819	7.21	10,261	143.16	348	260

And here is the data from Thursday 2/13.

Period	CPU Busy	NFS Op/s	Read Op/s	Read Lat (ms)	Write Op/s	Write Lat (ms)	Net Sent (MB/s)	Net Recv (MB/s)
9:09p	78	12,574	5,562	3.19	6,926	1.05	200	141
9:18p	98	27,771	17,291	4.7	10,347	24.29	571	312
9:29p	98	33,050	21,460	9.11	11,507	125.85	650	352
9:38p	98	34,149	22,813	11.28	11,216	530.9	647	345

One thing that stands out in the data is a large, sudden increase in 64k+ IOPS.  I’ve adjusted the table to include a row for 64k IOPS and have highlighted the relevant statistic.

FAS6280 Maximum IOPS
	Read/Write Mix
Avg IO Size	100/0	75/25	50/50	25/75	0/100
64k	61,000	43,000	32,000	26,000	22,000
32k	68,000	48,000	36,500	30,000	25,000
24k	74,000	51,000	39,500	31,500	27,000
16k	80,000	56,500	43,000	36,500	30,500
8k	85,000	63,000	50,000	41,500	36,000
4k	90,000	66,000	54,000	45,000	40,000

  The workload mix appears fine for most of the day but experiences large-IOP-size peaks that are outside our guidelines and cause some pain (38,000 IOPS 1pm 4/5,  45,000 IOPS 11pm 4/4, 37,000 IOPS 11am 4/3) .   I’d also make mention that ~10% of IO to this system is misaligned, which hinders us from achieving maximum performance ROI.  Lastly, this system is achieving 65-85% dedupe ratios, which is fantastic space conservation but adds to the overall workload.

  As discussed yesterday, here are our options:

·         Short term steps:

o   Stagger workloads (Symantec, et al)

o   Disable aggr snapshots (done)

o   Stagger dedupe

o   Case open on daytime snapshot correlated latency (done)

o   Update Data ONTAP

·         Long term solutions:

o   Add new disk to passive controller and balance workload or

o   Shift workload to a different or new HA pair

How to convert from 7-Mode to CDOT

Here is a publicly accessible document on converting your old 7-Mode systems to NetApp's new operating system, CDOT.

https://kb.netapp.com/support/index?page=content&id=1013517

An excerpt:

Perform the following steps to convert from Data ONTAP 7-Mode to Data ONTAP 8.0X Cluster-Mode

1. Disable 'Cluster Failover' and reboot the node to the LOADER prompt. Do not perform a takeover.
2. Boot each node to the LOADER prompt and ensure that the following variables are set:
   
   To convert from 7-Mode to Cluster-Mode:
   LOADER> set-defaults
LOADER> setenv bootarg.init.boot_clustered true
LOADER>  setenv bootarg.bsdportname
 
3. Boot the node with this command:
   boot_ontap
4. When the nodes are booting, press CTRL+C to enter the Boot menu.
5. At the Boot menu, select wipeconfig on each node.
   *******************************
* Press Ctrl-C for Boot Menu. *
*******************************
How would you like to continue booting?
(normal) Normally
(install) Install new software first
(password [user]) Change root/user password
(setup) Run setup first
(init) Initialize disks and create flexvol
(maint) Boot into maintenance mode
(syncflash) Update flash from backup config
(reboot) Reboot node
Please make a selection: wipeconfig

ONTAP 8.2 Commands

This could be useful to have handy: new/modified commands in 8.2

7-mode 8.2

https://library.netapp.com/ecmdocs/ECMP1368838/html/GUID-EE036B98-BE1B-4CFA-A3DD-85CF6C1AF0A1.html

All Commands in CDOT 8.2

https://library.netapp.com/ecm/ecm_get_file/ECMP1366832

Mapping 7-mode commands to CDOT 8.2

https://library.netapp.com/ecmdocs/ECMP1366830/html/index.html