NetApp Training Brain Dump: Problem and Solution

In SAN management, you're always working to meet several requirements depending on the nature of the data.  You may need to just add more space, in which case throwing more disks at the problem might be the best solution.  Other issues can be more complex to address, such as increasing performance working under a strict budget.  Below is a detailed progression of options as we search for the best solution.

Potential Options:
1) If you add more disks, the workload might be spread over more spindles and performance may increase.
2) What speed/size disk are you using?  Do you have well tiered data management solutions?
3) It might be smart to upgrade the RAM/Cache/Controllers in your SAN storage unit.
4) Utilization management: what applications are driving this change in performance requirements?  Can your SQL queries be more efficiently written?  Or maybe backup processes are poorly designed?

5) If you have all your high-transaction data on the same disks, you may want to spread that more evenly with your low-IO data.  This may require better software to be able to manage this solution (NetApp V-Series).

6) If a small amount of your data is responsibly for a disproportionate amount of writes, you may want to implement a Storage Acceleration solution in front of slower disks, giving you the performance you need at a lower cost.
7) If high read access is required, NetApp offers cache upgrades of high performance Flash memory to reduce the impact on your disks.


NetApp V-Series: Somewhat similar to HP SVSP, you place these devices between the clients of storage (servers, etc) and providers of storage (IBM XIV, etc).  This allows you to utilize NetApp's awesome software w/ your existing SAN technology.

NetApp SA (Storage Acceleration): Again, place these between your clients and providers of storage.  The device determines what data to hold in local, very fast memory, and what to relegate back to the actual SAN disk.  Pretty much the same relationship as your laptop's memory vs hard drive.  Increases performance.

Flash Cache: Implement this in your SAN storage unit for an inexpensive solution to drastically boost read operations!  Marketing says the performance of this technology is "comparable to SSD."

NetApp Training Brain Dump: Bird's Eye View

Preparing for a deep dive into NetApp technology! In an intelligence report to King George in 1776, England's spies wrote about John Adam's strength being that he "sees large things largely."  I try to take that approach of not getting caught in minutia when approaching a new technology, to better grasp the big picture.  The next few posts will be my journey into that, and I'm sure that in trying to encapsulate complex ideas I will be slightly incorrect in some of these statements.  Nuance comes with time!  So here we go, basic terms, spelled out in English:

Product Definitions:

- FAS system (aka filer): NetApp's term for the custom machine that manages the storage. Roughly equivalent in purpose to HP EVA, IBM XIV, etc. Capable of serving storage over ethernet NAS (file based protocols like HTTP, FTP, CIFS, etc) or SAN block based protocols (FCoE, iSCSI, or FC).  FAS (Fabric Attached Storage) designates that the filer is operating on FCoE, iSCSI, or FC rather than simply as a NAS device.

- SnapVault (OSSV): NetApp's backup solution.  Allows full or incremental backups to be transfered from a server directly to a NetApp storage system.

- SnapMirror: Real time replication.  Effectively creates software layer RAID 1 by creating exact clones of volumes or qtrees (can't mirror an aggregate from what I've read).  This enables NetApp's Metrocluster.

- Metrocluster: their version of DR implementation.  Two options: stretch (both controllers in one datacenter) or fabric attached (replication across an ISL (inter-site link) with one controller in each datacenter).

- SyncMirror:

- SnapDrive:

- FlexShare: Allows you to set processing priority for volumes within an aggregate.

- iGroup: Initiator group.  All LUN's are mapped to an iGroup, which handle LUN masking based upon the client system.  The iGroups basically contain the specifications for the OS-App combo etc to communicate to the LUN.  Typically, each server (or cluster) should have its own iGroup based upon the OS, Application (SQL, VMware, etc), and SAN protocol.

Break it down: There are a few layers where the building blocks of storage are combined to form higher level concepts for easier management, each with NetApp-specific jargon.  No worries, I'm here to translate and simplify:

- Layer 1: Disk drives.  duh.

- Layer 2: RAID Group.  This is a group of up to 28 disks operating as a pool of storage, 16 best practice.  You want all the RG's in a specific aggregate to be the same size.  Two parity disks per RG.

- Layer 2.5: Plex. A plex is a physical copy of the WAFL storage within the aggregate. A mirrored aggregate consists of two plexes; unmirrored aggregates contain a single plex.  Take 11 players from the Chicago Bears and NE Patriots, and they're a football team.  Move them around a bit, and you can put them in shotgun formation.  You can say that they're a set of players (aggregate), and they're distinctly from the Bears and the Patriots (volumes in the aggregate), and that they're a formation (plex)...there are many ways to view the organization of data.

- Layer 3: Aggregate.  This is a group of RAID Groups.  A RAID group can not be assigned to more than one Aggregate.

- Layer 4: Volume. This is space carved out inside an aggregate.  Typically this is space for 1 LUN + reserve space.
- Layer 5: LUN.  This is space carved out inside a volume.  There can be multiple LUNs per volume, but that can be inadvisable.  The LUN is the actual virtual disk being presented to the server.

- Layer 6: QTree. Essentially, this is space carved out inside a LUN for a particular directory, sometimes with a hard limit.

I'll keep these definitions updated as I learn the nuances or need to make corrections.

SAN protocols for dummies

Just doing a brush up on the basic storage concepts. Noobs will appreciate the simplification in this post, experts will likely find it too oversimplified. More detail can be found in the links or more recent posts!

For reference, the OSI model:
7. Application
6. Presentation
5. Session
4. Transport
3. Network
2. Datalink
1. Physical

Quick hits*1:
SCSI, SATA, FC, and SAS are layer 1 and 2 protocols. iSCSI is a layer 5 protocol. Ethernet is a layer 2 protocol. FCP is a layer 1, 2, and above protocol. FCoE is a layer 1-6 protocol.

Photo Courtesy of FCoE.ru

Local protocols: these are how the CPU communicates to the hard drives.  They are all layer 1 and 2 protocols specifying the hardware and electronic signals needed to send data between the drive and the CPU.  This has trended toward serial protocols (away from parallel) for performance and cost reasons.

1. SCSI (Small Computer System Interface): a high performance parallel standard that specifies hardware level communication over a local BUS.

2. SATA (Serial Advanced Technology Attachment): Slow, inexpensive.  Used mainly for unimportant, low change data.

3. Fibre Channel: Fast and expensive, this serial protocol is commonly used in enterprise SANs.

4. SAS (Serial attached SCSI): The SCSI protocol was modified to take advantage of cost and speed improvements in serial technology.  This is also commonly found in enterprise SAN's.

5. FATA (Fibre Attached Technology Adapted): Slow, inexpensive.  Really is SATA wrapped in a FC interface to gain from shelf technology.   This is less used than the other types.


Network Protocols: This is how a server can communicate over your SAN to its storage, essentially virtualizes the relationship between a computer and its local drives, allowing your server to think a virtual drive in a datacenter somewhere is actually directly plugged into it.

1. iSCSI:  Protocol simulates the SCSI protocol by wrapping it in ethernet-friendly packets.
Advantage: Works over existing ethernet networks (if given enough bandwidth).
Disadvantage: Some risks involved with having all your traffic on the same cables. Theoretically high overhead since it's higher up on the stack.

2. Fibre Channel Protocol: protocol that requires special fibre cabling and an entire alternate network to support communication.
Advantage: Can be fast, separates traffic and enhances stability.
Disadvantage: Expensive, requires special cabling.

3. Fibre Channel over Ethernet: FCoE. Protocol simulates the FC protocol by wrapping it in ethernet-friendly packets.
Advantage: Can be very fast for cheaper than FC.
Disadvantage: Theoretically more overhead. Doesn't separate traffic.


Sources:
You can find a great performance discussion here:
http://jmichelmetz.wordpress.com/2010/03/24/fcoe-vs-iscsi-the-cagefight-performance

*1: FCOE Discussion:
http://www.fcoe.ru/index.php?option=com_content&task=view&id=296&Itemid=65&lang=english

FC Discussion
http://bit.ly/dQkn8h

IBM Tivoli Storage Manager

Pinged a mentor of mine on a technology I haven't had a chance to work with to get his thoughts on it. I know that the internet is full of marketing-speak and tech mumbo jumbo, and he has a singular talent for cutting through it and explaining something in terms that really matter. He gave me a pretty solid and concise summary so good I can't help but pass it on. Enjoy!

"It’s an IBM backup product. Their big efficiency is they do what’s called “incremental forever” backups. Rather than focusing backups on how many weeks you keep a tape for, they look at how many copies of a data set you want to keep. So for example, say you have 10 files you back up nightly. You write a policy that says you want to keep 3 versions of each file on tape. If one file changes every day, it’ll get backed up every day, and Tivoli will release any copy of the file that is more than 3 revisions old. But the other files will only get backed up if/when they change. The problem with Tivoli is that it runs a process of reclamation and consolidation to copy data you need to keep to new media so tapes housing data no-longer-needed can be flushed. If you don’t run this reclamation and consolidation process, you could get into a situation where you need every tape you’ve ever written to restore a single volume.

It’s a neat product, and a different way to look at protecting data. It’s a solid product, well adopted, but hell on your off-site transport costs because you’ll be shuffling tapes around daily to do the reclamation and consolidation.

One other thing you can do is leave all your tapes in the library (for primary copy) and create a synthetic full that you then take off-site."

Wiki linkage:
http://en.wikipedia.org/wiki/IBM_Tivoli_Storage_Manager

LANDesk Return Codes

Quick thing I picked up recently: Had an issue where a LANDesk task would not return successfully, even though the batch file was running no problem.  “EXIT 0” was not working, but the attached documentation suggested using “EXIT /B 0" which worked like a charm. 

http://community.landesk.com/support/docs/DOC-2320

Thoughts on IT culture

A few thoughts on the inefficiency of IT and how to solve it:

1.  Pointing out other people's mistakes is not the same thing as leadership.

2.  I have never seen someone criticized by their boss for not taking initiative: I consistently see people criticized for imperfect results when they voluntarily overburden themselves.  Do you want a silo'd, turf-war, every-man-for-himself culture where people do their best to limit their workload in order to achieve actual perfection?  Or one where people are applauded for efforts to carry a greater share of the workload in order to allow their teammates to accomplish more as well?  Leadership should consider what they are incentivizing.

3.  Don't waste talent.  There's plenty of dirty work in IT and everyone understands that, but as much as you possibly can try to not waste your engineers' time with the small stuff.  I've seen guys making six figures who change backup tapes every day (!!!) and guys making $80k who have to spend 2 hours a week creating users in AD.  If you're paying him to design bridges then he shouldn't be filling potholes, or else you're wasting capital and drastically slowing your company's technological advancement.

4.  Hire interns.  The energy, work ethic, and new perspective forces all us full timers to stay on the edge of our game.  Further: 1)  Give your interns real responsibility.  2)  Let your interns shake up the status quo, even if your full timers don't like it.

5.  You pay your talent a lot, so get them the tools to be as effective as possible.  I really can't make this point any better than Jeff Atwood and Yishan Wong, I highly recommend these two reads.

Coding Horror's Programmer's Bill of Rights:
http://www.codinghorror.com/blog/2006/08/the-programmers-bill-of-rights.html

Engineering Management by Yishan Wong:
http://www.algeri-wong.com/yishan/engineering-management.html

Server Management Software

A very underappreciated tool: server asset management.  Just knowing basic info about your servers, documented in one place, can save your people tons of time and hassle.  The ROI for keeping this type of system up to date is off the charts.  Which is why I was so surprised at my current company when I found their tool with incomplete info and missing data.  I did a complete 8 hour audit of the datacenter and true'd it up with their mgmt software.  Here were my results:

-          15 servers for which we had no record of their location or our records were wrong.

-          15 servers whose front label were incorrect.

-          5 live servers whose front labels were missing.

-          33 servers not labeled or labeled incorrectly in the back of the rack.

-          10 servers where the recorded rack info was correct, but the position in the rack was not.

-          80 servers for which we didn’t have the submodel (e.g. 7979-XXX) or the submodel that was recorded was incorrect (This hasn’t been a requirement in the past so this was mostly information gathering on my part).

-          All of the serial numbers were correct (wow).

The Rack Visual portion of of our software is now in harmony with no overlaps - before, servers were mapped as being inside other servers, or on the roof :-).  I created tickets for things I wasn't able to immediately resolve like locating hardware and working with security to get their racks properly documented.