iSCSI

Here's a few concepts I've been studying.  A TPG (Target Portal Group) is basically a method of allowing the server to communicate to your storage system via iSCSI on multiple interfaces and multiple connections on a single session.  This means you can enable MPIO by having multiple TPGs.  Here's the dummy breakdown:
- Each interface (virtual or real) can only be part of one TPG
- Each TPG can have multiple interfaces
- Each session can have multiple connections
- Each connection can only be in one session.
- Each session can only communicate through one TPG

A great use of this is when a server has multiple virtual OS's and therefore needs multiple connections to the same storage system.  If you have multiple paths, you need ALUA.

When understanding ALUA (Asymmetric Logical Unit Access), it's helpful to know it's also called Target Port Group Support.  Basically it's a protocol for determining the best path from the server to the LUN (hence the LU in ALUA).  This protocol is standardized to work with any vendor's iSCSI hardware.

ISNS is basically DNS for iSCSI, but a little smarter in that it also understands TPG's and helps systems find each other that way.

More data: http://hd.kvsconsulting.us/netappdoc/801docs/html/ontap/bsag/GUID-FF148B6E-6CCB-48CC-9547-7D063A904B40.html

NetApp SPC Results

NetApp is buzzing over the last SPC results.  The reason it's important is that we proved we're are able to do a lot with a little.  The test was a first in a few ways:

1.  It's our first performance test of the new cluster-mode software, which is going to be huge.

2.  The test was FC, where NetApp has historically been called the NAS experts by our competitors.  

3.  If you look at the hardware both in cost and amount, we did the same or better latency-wise up to 250,000 IOPS with a smaller number of controllers and disks.

I know list prices don't mean anything, but as a rough guesstimate, look at the 3PAR solution vs NetApp comparison in the recoverymonkey blog (below).  Here's the breakdown:

3PAR

List: $5,885,148

IOPS: 450,212.66

Latency: 13.67

NetApp

List: $1,672,602

IOPS: 250,039.97
Latency: 3.35ms


You could buy 2 of the NetApp solutions for half the price, 50,000 more IOPS, and 1/4 the latency of the 3PAR system.  Wow.


http://www.theregister.co.uk/2012/06/26/netapp_cluster_mode_spc_benchmark/

http://recoverymonkey.org/2012/06/20/netapp-posts-great-cluster-mode-spc-1-result


Disclaimer: I'm affiliated with NetApp and I'm appropriately biased, but not paid for any of the content on this site.

NetApp Experience: Shelf ADD => Disk Fail => Failover

During a shelf add last week, I experienced as big of a system outage as I've ever encountered on NetApp equipment.  We started seeing a few of these errors, which are normally spurious:


ses.exceptionShelfLog:info]: Retrieving Exception SES Shelf Log information on channel 0h ESH module A disk shelf ID 4.
ses.exceptionShelfLog:info]: Retrieving Exception SES Shelf Log information on channel 6b ESH module B disk shelf ID 5.


The first connection went smoothly, but when I unplugged the second connection from the existing loop, I started seeing some scary results.  Here's the order of important messages:

NOTE: Currently 14 disks are unowned. Use 'disk show -n' for additional information.

fci.link.break:error]: Link break detected on Fibre Channel adapter 0h.

disk.senseError:error]: Disk 7b.32: op 0x2a:1bc91268:0100 sector 0 SCSI:aborted command -  (b 47 1 4e)

raid.disk.maint.start:notice]: Disk /aggr3_thin/plex0/rg0/7b.32 Shelf 2 Bay 0  will be tested.

diskown.errorReadingOwnership:warning]: error 46 (disk condition triggered maintenance testing) while reading ownership on disk 7b.32

disk.failmsg:error]: Disk 7b.32 (JXWGA8UM): sense information: SCSI:aborted command(0x0b), ASC(0x47), ASCQ(0x01), FRU(0x00).

raid.rg.recons.missing:notice]: RAID group /aggr3_thin/plex0/rg0 is missing 1 disk(s).

Spare disk 0b.32 will be used to reconstruct one missing disk in RAID group /aggr3_thin/plex0/rg0.

raid.rg.recons.start:notice]: /aggr3_thin/plex0/rg0: starting reconstruction, using disk 0b.32

[disk.senseError:error]: Disk 7b.41: op 0x2a:190ca400:0100 sector 0 SCSI:aborted command - (b 47 1 4e)

diskown.errorReadingOwnership:warning]: error 46 (disk condition triggered maintenance testing) while reading ownership on disk 7b.41

[raid.disk.maint.start:notice]: Disk /aggr3_thin/plex0/rg1/7b.41 Shelf 2 Bay 9  will be tested
[disk.senseError:error]: Disk 7b.37: op 0x2a:190ca500:0100 sector 0 SCSI:aborted command - (b 47 1 4e)

raid.config.filesystem.disk.failed:error]: File system Disk /aggr3_thin/plex0/rg1/7b.37 Shelf 2 Bay 5  failed.

[disk.senseError:error]: Disk 7b.40: op 0x2a:190ca500:0100 sector 0 SCSI:aborted command - (b 47 1 4e)

raid.config.filesystem.disk.failed:error]: File system Disk /aggr3_thin/plex0/rg1/7b.40 Shelf 2 Bay 8  failed.

raid.vol.failed:CRITICAL]: Aggregate aggr3_thin: Failed due to multi-disk error

disk.failmsg:error]: Disk 7b.37 (JXWG6MLM): sense information: SCSI:aborted command(0x0b), ASC(0x47), ASCQ(0x01), FRU(0x00).

disk.failmsg:error]: Disk 7b.40 (JXWEEB3M): sense information: SCSI:aborted command(0x0b), ASC(0x47), ASCQ(0x01), FRU(0x00).

raid.disk.unload.done:info]: Unload of Disk 7b.37 Shelf 2 Bay 5 has completed successfully

raid.disk.unload.done:info]: Unload of Disk 7b.40 Shelf 2 Bay 8  has completed successfully

Waiting to be taken over.  REBOOT in 17 seconds.

cf.fsm.takeover.mdp:ALERT]: Cluster monitor: takeover attempted after multi-disk failure on partner


Long story short, this system had caused numerous issues in the past, and we replaced both a dead disk and an ESH module.  After that, the system stabilized: "Since the ESH module replacement there were no new loop or link breaks noticed in subsequent ASUPs."

BJJ: Armbar from Guard

Pretty amazing, how you learn even things you thought you knew.

My armbar from guard has always been terrible: I've spent years faking that move just to get to something else.  I could never seem to get my hips shifted quickly enough: putting my foot on their hip was too big of a giveaway.  

 But yesterday, everything changed.  Watch this video by world-class black belt of black belts Pedro Sauer: ignore the foot on the hip.  Focus on the other leg.  With no foot on the hip, you can swing your other foot across the person's back in one smooth movement, shifting your hips and catching them by surprise.  So slick.

http://www.youtube.com/watch?v=zrYiQPp0T-M

NetApp Experience: Mixed Ownership

When determining where to add a shelf to a production system, disk show -o is useful in determining which loops/stacks contain disks owned by the controller you're planning for.  When the system is properly set up, this works just fine.  When the system already has mixed ownership on a loop but single ownership on the other loops, you would obviously prefer to expand the single ownership loops.  

But disk show -o will not indicate mixed ownership, so it's a bit of a trap.  The additional step you can take would be to either a)check autosupports before for mixed ownership or b) use disk show -v to show all disks, and verify there are no disks on that loop owned by the other controller.

NetApp Experience: Amber LED

Quick hit: If you have an amber LED on your filer that won't turn off, try this:
1.  Disable CF.
2.  Halt -f  both heads.
3.  On reboot, enable CF.

Cool trick!

SOPA