NetApp Experience: MetroCluster Disk Fail

I'm working on a case for a MetroCluster right now.  The situation was started because the customer was doing power maintenance, and shut off one of the two PDU's in the rack.  In a metrocluster, between the system and the shelves there are redundant fibre channel switches, but in this case the switches had only 1 power supply per switch, but they were spread across the two PDU's.  This means that one of the two switches went offline during the work.


The system failed 9 disks in this case, and all 9 of them were being addressed over the switch that went down.  NGS has found iSCSI errors over the switch that stayed up during that time.  The only firmware that is backrev'd is ESH firmware, so we're gonna have to dig deeper for a solution.


Those 9 disks failing caused a RAID Group to fail, which caused an aggregate to fail.  We got the system back up and running by re-seating each disk slowly, with 90 seconds between each action.


Update:  Here's what NGS had to say. (Quote)

The ports on the brocade switches are not locked:

>>> [-1][01] CRITICAL [01][01][00]:switchname: Port[4] has loop port NOT LOCKED

The ports not being locked can cause a number of instability issues and is the most likely cause of the issue seen.  The information on how to lock these can be found in the following document: http://media.netapp.com/documents/tr-3548.pdf

 "Not enough power supplies are present...to satisfy disk drive and shelf power requirements." 

The logs are erroneous, there's been burts opened to correct this warning, but one PSU should not cause the disk shelf any issues other than it takes longer to get the disks spun up since it will do this in increments.

"Cluster monitor: takeover of eg-nascsi-a02 disabled (unsynchronized log)"
"Cluster Interconnect link 0 is DOWN"

It wouldn't surprise me if the syncmirror lost sync during this period based on the switch issues they experienced.  The ports not being locked can cause a large number of unusual errors.

(End Quote)


Making sense of it: Brocade's ports are categorized as E, F, L, G, etc.  An L port is a loop port, which means the switch will only initiate loop traffic.  Locking a port as an F port means that the switch won't begin treating the ports in a point-to-point relationship.  Directly from Brocade's documentation:

Credit: Brocade Fabric OS Reference 2.6

Here's a case of a NetApp customer working through this issue.

NetApp Insights: MetroCluster

My main criticism of NetApp's MetroCluster implementation is the same as this guy's; it has single points of failure.

Let's rewind a bit.  NetApp has a product called a Fabric MetroCluster, in which you pretty much pick up one controller out of your HA pair and move it to another datacenter (I'm simplifying things).  It's a good implementation in that it spreads the reliability of a single system out across two datacenters and replicates in real time.  It's a bad implementation in that it's still a single system.

Everything can fail, so in SAN, the name of the game is redundancy.  This is why customers buy TWO controllers when they purchase a HA system, even though both controllers are going into the same datacenter: each controller has ~6 single points of failure, and if it goes down, you still need your data to be served.  By providing a redundant controller, you can lose a controller and your customers won't even notice.  That's why we refer to a HA clustered pair as a single system: the cluster is a unit, a team.

You don't have the same luxury (without massive expense) when you spread your cluster across two datacenters.  The reason you geographically locate your SAN system in the same datacenter as your servers is that there's a large amount of traffic going back and forth from the SAN system to the servers.  Trying to pump all that traffic through an inter-site link (ISL aka inter-switch link) requires a serious pipe, which is very expensive.

If your SAN system goes down, the DR plan is typically to failover the clustered servers at the same time as the SAN system, a complex and often risky procedure.  By failing both over, you ensure the traffic does not need to travel over the ISL, which would likely create latencies beyond the tolerances of your applications.  But a better solution is to make sure your SAN system is redundant in the first place so you don't need to fail over.

This is why NetApp's current MetroCluster implementation falls short: it has 6 single points of failure that would require you to either push all traffic through your ISL or fail EVERYTHING over.  That's not, in my opinion, enterprise-class.

Good news though - looks like NetApp might be planning on fixing this to allow a clustered pair at each datacenter.