NetApp Experience: MetroCluster Disk Fail

I'm working on a case for a MetroCluster right now.  The situation was started because the customer was doing power maintenance, and shut off one of the two PDU's in the rack.  In a metrocluster, between the system and the shelves there are redundant fibre channel switches, but in this case the switches had only 1 power supply per switch, but they were spread across the two PDU's.  This means that one of the two switches went offline during the work.


The system failed 9 disks in this case, and all 9 of them were being addressed over the switch that went down.  NGS has found iSCSI errors over the switch that stayed up during that time.  The only firmware that is backrev'd is ESH firmware, so we're gonna have to dig deeper for a solution.


Those 9 disks failing caused a RAID Group to fail, which caused an aggregate to fail.  We got the system back up and running by re-seating each disk slowly, with 90 seconds between each action.


Update:  Here's what NGS had to say. (Quote)

The ports on the brocade switches are not locked:

>>> [-1][01] CRITICAL [01][01][00]:switchname: Port[4] has loop port NOT LOCKED

The ports not being locked can cause a number of instability issues and is the most likely cause of the issue seen.  The information on how to lock these can be found in the following document: http://media.netapp.com/documents/tr-3548.pdf

 "Not enough power supplies are present...to satisfy disk drive and shelf power requirements." 

The logs are erroneous, there's been burts opened to correct this warning, but one PSU should not cause the disk shelf any issues other than it takes longer to get the disks spun up since it will do this in increments.

"Cluster monitor: takeover of eg-nascsi-a02 disabled (unsynchronized log)"
"Cluster Interconnect link 0 is DOWN"

It wouldn't surprise me if the syncmirror lost sync during this period based on the switch issues they experienced.  The ports not being locked can cause a large number of unusual errors.

(End Quote)


Making sense of it: Brocade's ports are categorized as E, F, L, G, etc.  An L port is a loop port, which means the switch will only initiate loop traffic.  Locking a port as an F port means that the switch won't begin treating the ports in a point-to-point relationship.  Directly from Brocade's documentation:

Credit: Brocade Fabric OS Reference 2.6

Here's a case of a NetApp customer working through this issue.

NetApp Experience: Shelf Add => Disk Fail

One of my practices when performing a shelf add is to wait in between each step, specifically between unplugging and re-connecting any cables.  My thought process on this has been that the system should be allowed time to settle to its new circumstance, specifically that the controller will need to recognize what paths it is now able to communicate to the disks on.

Digging deeper, one thing I recently learned is that the disk has two ports of communication (referred to as A and B) to the shelf modules, and they negotiate their paths from the disk to the shelf module to the fiber ports on controller.  e.g., Disk 23 port A could be connecting through shelf module B to port 2c, and disk 23 port B could be connecting through shelf module A to port 1b.

All of that is important to understanding the serious issue that failed two disks in a production cluster recently.  A single shelf (DS14mk2 750GB SATA) was connected MPHA to a clustered pair with this configuration:

disk 1d.18: disk port A to shelf module B to port 1d
disk 2b.22: disk port B to shelf module A to port 2b

After unplugging the cable from 1d to shelf module B, there was a 17 second delay and then this:

Cluster Notification mail sent: Cluster Notification from CONTROLLER (DISK CONFIGURATION ERROR) WARNING
Controller> scsi.path.excessiveErrors:error]: Excessive errors encountered by adapter 2b on disk device 2b.18.
Controller> scsi.cmd.transportError:error]: Disk device 2b.22: Transport error during execution of command: HA status 0x9: cdb 0x28:354a07b8:0048.
Controller> raid.config.filesystem.disk.not.responding:error]: File system Disk /aggr2/plex0/rg0/2b.22 Shelf 1 Bay 6 [NETAPP   X268]  is not responding.
Controller> scsi.cmd.transportError:error]: Disk device 2b.18: Transport error during execution of command: HA status 0x9: cdb 0x28:4f5e9748:0048.
Controller> disk.failmsg:error]: Disk 2b.22: command timed out.
Controller> raid.rg.recons.missing:notice]: RAID group /aggr2/plex0/rg0 is missing 1 disk(s).
Controller> raid.rg.recons.info:notice]: Spare disk 2b.27 will be used to reconstruct one missing disk in RAID group /aggr2/plex0/rg0.

Controller> diskown.errorReadingOwnership:warning]: error 23 (adapter error prevents command from being sent to device) while reading ownership on disk 2b.18

Analysis:
These two disks failed as a result of an HBA issue last night.  When a path is disconnected, any disks that are owned over that path are engineered to use the redundant path.  When we disconnected port 1d, the HBA in slot 2 produced errors that halted this normal renegotiation for two of the disks.  Because the disks were not able to operate on the redundant path, the system considered the disks to be in a failed state and began reconstructing that data to spare disks.  When this happened, we halted work to investigate and remediate.  We'll probably just RMA the disks to reduce variables.

NetApp Support recommendation: Re-seat this HBA, which would require a failover/giveback to perform.  Another option would be to replace the HBA (which is what we'll probably do).

Update: NGS (NetApp Support) has changed their minds and now think this is a disk firmware issue.  This disk firmware was backrev'd a couple years still, and their explanation is that iSCSI errors caused by the firmware pile up over time and eventually cross a threshold and cause an HBA to consider that disk incommunicable.  There's no warning on the system that this HBA can't talk to that disk, and all the traffic is routed through the redundant path.

In this case, we had two disks that were in this situation and when I unplugged the return path (the path they were active on) they tried to fail over to the other path and could not.  NGS believes this was just a pure chance, struck by lightning situation.

I'll post the bug report on this soon: the gist of it is that between 40C and 50C, a latching mechanism can get "stuck" and error out, but will quickly recover.  I'm skeptical of this because the highest temperature observed in this shelf was 36C.

2nd Update: As best as I can tell, the disk firmware update did the trick.  We went through with shelf adds last night without seeing the same behavior.  We did, however, see what we believe to be a separate issue.

NetApp Training Brain Dump: Experimenting

Quick notes on things I tested today:

• If you run a disk fail command, you will have to wait a few hours for the data on that disk to be copied to a spare.
• There is a -i trigger for the disk fail command that will immediately fail the disk, without copying the data.
• If you have no spares and you have a disk that is not assigned and is not in use, you have to assign that disk to the controller before it will be used as a spare.  If you have options disk.auto_assign on, it will have already been assigned to a controller.  In either case, you won't need to add the disk to an aggregate: the system detects it as a spare and grabs it in the place of the failed disk.
• To see how many failed disks you have, use vol status -f
• To see how many spares you have, use vol status -s
• If you want to see the status of your disks, disk show won't do it.  You'll need to use disk show -v to see failed disks, and neither will show spare disks as being spare.  
• You can't resize an aggregate's RAID roups.  You can however use aggr options raidsize to set the size for new RAID Groups that are created for this aggregate.