CDOT FCP

I spent some time putting information together on SAN on CDOT.  

-          Basic Architecture

o   CDOT uses NPIV to virtualize WWPNs.  This means any node can take traffic for any vserver, and ALUA optimizes the paths.

o   When zoning, utilize these vWWPNs and not the physical WWPNs.

o   Each SVM gets its own IQN (iSCSI) or WWNN (FC).

o   8-node cluster limit today

o   LUNs can be moved to new volumes non-disruptively inside a cluster

o   Limits

§  CDOT 8.2: 8,192 LUNS per node | 49,152 LUNS per cluster | 2,048 iSCSI sessions per node

§  CDOT 8.3: 12,288 LUNS per node | 98,304 LUNS per cluster | 8,192 iSCSI sessions per node

§  Linux host (8.2): 2,048 devices (# LUNS * # paths) | 16TB LUNs

§  Windows (8.2): 255 LUNs per host, 2TB LUNs (MBR) or 16TB (GPT)

o   FAS80X0 support 4-port 8Gb cards or 2-port 16Gb cards.

-          Useful commands:

o   Network interface show (will show FCP lifs)

o   Fcp show

o   Igroup show

o   Lun show

o   system node run -node cluster1-01 fcp topology show

o   Useful SAN Setup how-to (8.1 but very applicable): https://kb.netapp.com/support/index?page=content&id=1013341

-          Foreign LUN Import: 8.3 feature, no license required.  8.3.1 enables the online import using redirect.  Cutover happens first: pause IO to foreign SAN, present LUN to NetApp.  Foreign LUN shows up as a disk in ONTAP (think V-series) and then map this disk to the original host to serve the data.  Bring everything online, data is copied to NetApp while serving data.  Once it’s all moved, shut down foreign SAN. 

You can find the 8.2 SAN config guide here:

https://library.netapp.com/ecmdocs/ECMP1196793/html/index.html

And 8.2.1 SAN Config Guide updates:

https://library.netapp.com/ecmdocs/ECMP1368692/html/accessible_toc.html

8.3 SAN config Guide:

https://library.netapp.com/ecmdocs/ECMP12458217/html/index.html

NetApp Training Brain Dump: Problem and Solution

In SAN management, you're always working to meet several requirements depending on the nature of the data.  You may need to just add more space, in which case throwing more disks at the problem might be the best solution.  Other issues can be more complex to address, such as increasing performance working under a strict budget.  Below is a detailed progression of options as we search for the best solution.

Potential Options:
1) If you add more disks, the workload might be spread over more spindles and performance may increase.
2) What speed/size disk are you using?  Do you have well tiered data management solutions?
3) It might be smart to upgrade the RAM/Cache/Controllers in your SAN storage unit.
4) Utilization management: what applications are driving this change in performance requirements?  Can your SQL queries be more efficiently written?  Or maybe backup processes are poorly designed?

5) If you have all your high-transaction data on the same disks, you may want to spread that more evenly with your low-IO data.  This may require better software to be able to manage this solution (NetApp V-Series).

6) If a small amount of your data is responsibly for a disproportionate amount of writes, you may want to implement a Storage Acceleration solution in front of slower disks, giving you the performance you need at a lower cost.
7) If high read access is required, NetApp offers cache upgrades of high performance Flash memory to reduce the impact on your disks.


NetApp V-Series: Somewhat similar to HP SVSP, you place these devices between the clients of storage (servers, etc) and providers of storage (IBM XIV, etc).  This allows you to utilize NetApp's awesome software w/ your existing SAN technology.

NetApp SA (Storage Acceleration): Again, place these between your clients and providers of storage.  The device determines what data to hold in local, very fast memory, and what to relegate back to the actual SAN disk.  Pretty much the same relationship as your laptop's memory vs hard drive.  Increases performance.

Flash Cache: Implement this in your SAN storage unit for an inexpensive solution to drastically boost read operations!  Marketing says the performance of this technology is "comparable to SSD."

NetApp Training Brain Dump: Bird's Eye View

Preparing for a deep dive into NetApp technology! In an intelligence report to King George in 1776, England's spies wrote about John Adam's strength being that he "sees large things largely."  I try to take that approach of not getting caught in minutia when approaching a new technology, to better grasp the big picture.  The next few posts will be my journey into that, and I'm sure that in trying to encapsulate complex ideas I will be slightly incorrect in some of these statements.  Nuance comes with time!  So here we go, basic terms, spelled out in English:

Product Definitions:

- FAS system (aka filer): NetApp's term for the custom machine that manages the storage. Roughly equivalent in purpose to HP EVA, IBM XIV, etc. Capable of serving storage over ethernet NAS (file based protocols like HTTP, FTP, CIFS, etc) or SAN block based protocols (FCoE, iSCSI, or FC).  FAS (Fabric Attached Storage) designates that the filer is operating on FCoE, iSCSI, or FC rather than simply as a NAS device.

- SnapVault (OSSV): NetApp's backup solution.  Allows full or incremental backups to be transfered from a server directly to a NetApp storage system.

- SnapMirror: Real time replication.  Effectively creates software layer RAID 1 by creating exact clones of volumes or qtrees (can't mirror an aggregate from what I've read).  This enables NetApp's Metrocluster.

- Metrocluster: their version of DR implementation.  Two options: stretch (both controllers in one datacenter) or fabric attached (replication across an ISL (inter-site link) with one controller in each datacenter).

- SyncMirror:

- SnapDrive:

- FlexShare: Allows you to set processing priority for volumes within an aggregate.

- iGroup: Initiator group.  All LUN's are mapped to an iGroup, which handle LUN masking based upon the client system.  The iGroups basically contain the specifications for the OS-App combo etc to communicate to the LUN.  Typically, each server (or cluster) should have its own iGroup based upon the OS, Application (SQL, VMware, etc), and SAN protocol.

Break it down: There are a few layers where the building blocks of storage are combined to form higher level concepts for easier management, each with NetApp-specific jargon.  No worries, I'm here to translate and simplify:

- Layer 1: Disk drives.  duh.

- Layer 2: RAID Group.  This is a group of up to 28 disks operating as a pool of storage, 16 best practice.  You want all the RG's in a specific aggregate to be the same size.  Two parity disks per RG.

- Layer 2.5: Plex. A plex is a physical copy of the WAFL storage within the aggregate. A mirrored aggregate consists of two plexes; unmirrored aggregates contain a single plex.  Take 11 players from the Chicago Bears and NE Patriots, and they're a football team.  Move them around a bit, and you can put them in shotgun formation.  You can say that they're a set of players (aggregate), and they're distinctly from the Bears and the Patriots (volumes in the aggregate), and that they're a formation (plex)...there are many ways to view the organization of data.

- Layer 3: Aggregate.  This is a group of RAID Groups.  A RAID group can not be assigned to more than one Aggregate.

- Layer 4: Volume. This is space carved out inside an aggregate.  Typically this is space for 1 LUN + reserve space.
- Layer 5: LUN.  This is space carved out inside a volume.  There can be multiple LUNs per volume, but that can be inadvisable.  The LUN is the actual virtual disk being presented to the server.

- Layer 6: QTree. Essentially, this is space carved out inside a LUN for a particular directory, sometimes with a hard limit.

I'll keep these definitions updated as I learn the nuances or need to make corrections.

SAN protocols for dummies

Just doing a brush up on the basic storage concepts. Noobs will appreciate the simplification in this post, experts will likely find it too oversimplified. More detail can be found in the links or more recent posts!

For reference, the OSI model:
7. Application
6. Presentation
5. Session
4. Transport
3. Network
2. Datalink
1. Physical

Quick hits*1:
SCSI, SATA, FC, and SAS are layer 1 and 2 protocols. iSCSI is a layer 5 protocol. Ethernet is a layer 2 protocol. FCP is a layer 1, 2, and above protocol. FCoE is a layer 1-6 protocol.

Photo Courtesy of FCoE.ru

Local protocols: these are how the CPU communicates to the hard drives.  They are all layer 1 and 2 protocols specifying the hardware and electronic signals needed to send data between the drive and the CPU.  This has trended toward serial protocols (away from parallel) for performance and cost reasons.

1. SCSI (Small Computer System Interface): a high performance parallel standard that specifies hardware level communication over a local BUS.

2. SATA (Serial Advanced Technology Attachment): Slow, inexpensive.  Used mainly for unimportant, low change data.

3. Fibre Channel: Fast and expensive, this serial protocol is commonly used in enterprise SANs.

4. SAS (Serial attached SCSI): The SCSI protocol was modified to take advantage of cost and speed improvements in serial technology.  This is also commonly found in enterprise SAN's.

5. FATA (Fibre Attached Technology Adapted): Slow, inexpensive.  Really is SATA wrapped in a FC interface to gain from shelf technology.   This is less used than the other types.


Network Protocols: This is how a server can communicate over your SAN to its storage, essentially virtualizes the relationship between a computer and its local drives, allowing your server to think a virtual drive in a datacenter somewhere is actually directly plugged into it.

1. iSCSI:  Protocol simulates the SCSI protocol by wrapping it in ethernet-friendly packets.
Advantage: Works over existing ethernet networks (if given enough bandwidth).
Disadvantage: Some risks involved with having all your traffic on the same cables. Theoretically high overhead since it's higher up on the stack.

2. Fibre Channel Protocol: protocol that requires special fibre cabling and an entire alternate network to support communication.
Advantage: Can be fast, separates traffic and enhances stability.
Disadvantage: Expensive, requires special cabling.

3. Fibre Channel over Ethernet: FCoE. Protocol simulates the FC protocol by wrapping it in ethernet-friendly packets.
Advantage: Can be very fast for cheaper than FC.
Disadvantage: Theoretically more overhead. Doesn't separate traffic.


Sources:
You can find a great performance discussion here:
http://jmichelmetz.wordpress.com/2010/03/24/fcoe-vs-iscsi-the-cagefight-performance

*1: FCOE Discussion:
http://www.fcoe.ru/index.php?option=com_content&task=view&id=296&Itemid=65&lang=english

FC Discussion
http://bit.ly/dQkn8h

EVAPerf Statistical Limitations

Here's a big issue I wrestled with over the last few weeks:

EVAPerf occassionally hiccups, kicking out a single data point that claims a 2Gb host port has 30Gb/s throughput, or that a single diskgroup has 200GB/s being written to it. Sometimes these are clear overflow numbers (214748.3647 showed up repeatedly) and sometimes they are just absurdly high, though unique and precise. HP recommends using 95th percentile to statistically analyze the performance of your EVA's, and these super-high numbers skew our statistics to the point of being worthless.

My solution: create duplicate, empty SQL tables and daily screen the data to move any data points over thresholds I set to those duplicate tables, where they'd be out of the scope of my automated reporting. The trouble is, where's the threshold?

After a few weeks of emailing back and forth with HP, it became clear that they weren't interested in giving any definite answer or OK'ing my calculations. Part of the reason for that is that they have no real insight into how much actual IO CAN occur in a disk group because of all the behind-the-scenes leveling, parity calculation, and striping that occurs on top of the server-generated IO.

On top of that, block size varies so widely that the IO capacity calculations that they ARE able to do give you no real concept of the throughput capacity of your hardware. For example, let's say their PerfMonkey tool said your disk configuration allowed for 6000 read and 4000 write IO/sec in your disk group. Theoretically, with block sizes of up to 64MB apiece, this means your throughput could be over 380GB/s. So we're without a solid mathematical recourse.

I settled on 50GB/s for disk groups and LUNs, and 10Gb/s for host ports (even though they're only 4Gb ports) after careful analysis of how that affected the data - ends up looking like about 40 data points per month would be moved using those as thresholds.

Working pretty nicely so far.

HP EVA fnames.conf

Our SAN environment has 120TB spread over 4 HP EVA's (3 8100's and a 4400). We've worked through numerous difficulties with these, not the least of which was the dreaded "Saturday morning slowness." Part of our efforts to combat this was to attempt to gain greater insight into where the IO was actually coming from - at the time, we simply didn't have the system in place to do this.

Our friends at HP provide EVAPerf, which kicks out CSV files with a deluge of data (200MB+ per day per array). In order to make sense of this, a good friend on the software engineering side was added as a resource: he did a great job of writing a loading program that took those CSV files and kicking them in SQL. Our company is looking into IP rights, it's that awesome.

Meanwhile, we discovered a few months ago that our CSV's were filled with WWN's, which are pretty cumbersome to work with. HP's solution is the friendly names file fnames.conf, which just allows the EVAPerf task on your server to replace the WWN's with readable, English names. So a fnames.conf file was set up to automatically recreate itself once a day.

Well, as time went on and we changed more and more, the data I was working with was increasingly filled with WWN's. Investigation Monday morning turned up this little gem:
"The fnames.conf file must reside in the directory in which HP Command View EVAPerf was installed. "

Well, whoever set this up had it creating the updated file in a c:\utilities subdirectory - we were working with a months-old fnames file, since EVAPerf was installed under c:\programs\hp... After updating this, we went from 15/150 correctly represented LUNS in one EVA's csv file to 135/150 LUNS showing up with real names, greatly simplifying our vdisk I/O statistics. Woot!

Extremely useful information on decoding EVAPerf data:
http://www.fcoe.ru/index.php?option=com_content&task=view&id=257&Itemid=46#addcomments

See here for more official info on the fnames/evaperf integration
http://h10032.www1.hp.com/ctg/Manual/c00605846.pdf