NetApp Experience: Bad Slot

Really very interesting things have happened lately. I had a shelf add that kicked out a ridiculous amount of errors for one disk on the new shelf:

disk.senseError:error]: Disk 2d.53: op 0x28:0000a3e8:0018 sector 0 SCSI:hardware error - (4 44 0 3)


diskown.RescanMessageFailed:warning]: Could not send rescan message to eg-naslowpc-h01. Please type disk show on the console for it to scan the newly inserted disks.


diskown.errorReadingOwnership:warning]: error 46 (disk condition triggered maintenance testing) while reading ownership on disk 2d.53


Disk 2d.53: op 0x28:0000a3f0:0008 sector 0 SCSI:hardware error - (4 44 0 3)
diskown.AutoAssignProblem:warning]: Auto-assign failed for disk 2d.53

The weird thing was that the messages just continued to loop rather than just fail the disk.  We swapped a new disk into that slot, and the old disk into a different slot to see if the disk was bad: turns out, the slot is bad.

We also tried reseating shelf Module B on that shelf.  NetApp Support informed me that "Module A handles communication to the even numbered disks by default, and Module B the odd disks."  I don't think this is true.

We're working with the customer to find a good resolution for this.  Since downtime is difficult to accomplish, we may try to swap out the shelf chassis while the system is running.  We'll see :-)

NetApp Experience: Hardware

Got a few more hardware knowledge hits for you.  For one, there is a best practice around connecting shelves on a SAS PCI card, and it has to do with the internal architecture of the card.  The general idea is that there's two single points of failure inside each card called ASIC's.  Basically, A/B (or 1/2) are paired to one ASIC, and C/D (or 3/4) to the other ASIC.

For this reason, when you are connecting a stack to a single SAS PCI card (which you should try to avoid in the first place, but is occasionally unavoidable) you should use A/C as the start of the paths, and B/D should be the return paths.


Onboard ASIC's are paired between 0a-0b, 0c-0d, 0e-0f, etc.

Second, I ran into a customer that had a TON of problems with a system.  It showed up in all sorts of weird ways, leading the admins to update ONTAP and all the firmware.  They finally traced the issue to a single disk, which they replaced.  But the replacement disk failed, and so did the next replacement disk, which they pulled and left the problem slot empty.  Long story short, we swapped out the entire shelf chassis, pulling out disks, ESH modules, and power supplies and placing them in the new chassis.  We made the call to put an entirely new disk into the new chassis after all this.

Although this was a success in resolving the customer's issue, one interesting note was that the ESH modules did not retain the shelf ID.  It turns out that while the shelf ID is retained in the ESH module's volatile memory,  it is actually stored permanently in the internal circuitry of the shelf chassis, and read by the ESH module upon boot up.  Whoa!