NetApp Experience: Shelf ID

Encountered something cool recently that totally stumped NetApp experts: a DS4243 shelf whose shelf ID had gone crazy.  The ID was set to 19 when it should have been set to 11, and the 1 was blinking.  The system recognized the ID as 19 and functioned normally, but the shelf would not respond to the shelf-ID selector button that should have allowed me to change it.  There was a disk drive missing in slot 4: this turned out to be unrelated as far as I can tell.  At the software level, ACP and everything else just saw the ID as 19! Steps I tried:

- Power cycle the shelf (no effect).
- Change shelf ID (Wouldn't respond).
- Reseat the IOM modules (no effect).
- Update firmware (no effect).
- Replacing missing drive (no effect).

Got on the phone with NGS, and at the end of the day there was nothing else we could try.  They shipped out a new chassis and we swapped it out, placing the old disks, power supplies, and IOM modules into the new chassis.  Set the new chassis's shelf ID and everything worked great!

Details for future reference:
1TB DS4243 with IOM3's hooked up to a 6080 cluster, MPHA.  2 stacks of 2 shelves.

NetApp Experience: Hardware

Got a few more hardware knowledge hits for you.  For one, there is a best practice around connecting shelves on a SAS PCI card, and it has to do with the internal architecture of the card.  The general idea is that there's two single points of failure inside each card called ASIC's.  Basically, A/B (or 1/2) are paired to one ASIC, and C/D (or 3/4) to the other ASIC.

For this reason, when you are connecting a stack to a single SAS PCI card (which you should try to avoid in the first place, but is occasionally unavoidable) you should use A/C as the start of the paths, and B/D should be the return paths.


Onboard ASIC's are paired between 0a-0b, 0c-0d, 0e-0f, etc.

Second, I ran into a customer that had a TON of problems with a system.  It showed up in all sorts of weird ways, leading the admins to update ONTAP and all the firmware.  They finally traced the issue to a single disk, which they replaced.  But the replacement disk failed, and so did the next replacement disk, which they pulled and left the problem slot empty.  Long story short, we swapped out the entire shelf chassis, pulling out disks, ESH modules, and power supplies and placing them in the new chassis.  We made the call to put an entirely new disk into the new chassis after all this.

Although this was a success in resolving the customer's issue, one interesting note was that the ESH modules did not retain the shelf ID.  It turns out that while the shelf ID is retained in the ESH module's volatile memory,  it is actually stored permanently in the internal circuitry of the shelf chassis, and read by the ESH module upon boot up.  Whoa!

NetApp Experience: Controller Panic

Was shadowing a shelf add recently and got to observe a pretty hairy situation.  Here's the rundown:

11:00pm

1. A DS14mk4 shelf was added to a production HA FAS6080 running ONTAP 7.3.3.  The shelf was intended to be shelf 2 in the loop, but the shelf ID was still set to 1 when it was added.
2. Panic and Failover occurred from the controller who owned all the disks on that shelf. 
3. New shelf ID is set to 2, the correct ID.
4. The partner node assigned soft ID's to the new disks.
5. The partner did not recognize all of the real shelf 1's disks, and began rebuilding.
6. As many as 8 disks began rebuilding in bay 27, 28, or 29 of several loops.  Seems like this client keeps their spares in the last couple bays of the second disk shelf per loop, or the first bay in the third shelf.

Errors generated by adding the shelf with a wrong shelf ID, in chronological order:

fci.device.invalidate.soft.address   adapterName="0a" deviceName="0a.0 (0x04000000)" hardLoopId="17" 
scsi.cmd.selectionTimeout              deviceType="Disk" deviceName="0a.17" 
disk.ioFailed                                     deviceName="0a.17" 
scsi.cmd.noMorePaths                    deviceType="Disk" deviceName="0a.22" 
scsi.cmd.noMorePaths                    deviceType="Disk" deviceName="0a.23" 

03:00am:  Disks completed rebuilding.  8 disks on real shelf 1 still not being recognized.

04:00am:  FSE arrives onsite.

04:30am:  20 minute outage action plan developed:

• Shut down all systems accessing the data.
• Disable protocols.
• Halt both controllers (take them offline).
• Reboot disk shelf 1 and 2.
• Boot up controllers.

06:00am:  No action taken.  Customer and NetApp decided to let the system stay stable into production hours and address it the next night.

10:00pm:  Action plan started (shut off systems accessing the data, etc).
10:19pm:  Both controllers shut down.
10:28pm:  Both controllers up and functioning normally.

Notes:

1. No outage occurred until the controlled failback.
2. No data loss occurred.
3. We have no insight into the effect on performance.

Take aways:

1. Having lots of spares can pay off.
2. Make sure there are no more than two disks per RG on any one shelf.
3. Human error is much more likely than mechanical failure or software bug to cause a major disruption.