IntroductionThis document shows various UCS failure scenarios and method to test them. These tests are good to check for correct UCS system behavior in case of a failure. A UCS system consists of one or two UCS 6100 series switches or fabric interconnects. A UCS system with two fabric interconnects is typically deployed in active-active pair that supports failover. UCS 2100 fabric extender enables the UCS 6100 fabric interconnect to provide all access-layer switching needs for the connected servers. This traffic is then switched to its required destination by the fabric interconnect and no switching, whatsoever, is done by the fabric extender. This document shows how to test the following failure scenarios using the UCS Manager CLI: Fabric Interconnect failure UCS Blade failure Server Port failure Uplink Port failureTesting Fabric Interconnect failureThe method here is to reboot one FI switch, preferably the subordinate fabric interconnect switch. The expected result is to check the "show cluster state" command will show the primary fabric interconnect "UP" and the subordinate fabric interconnect "DOWN". The fabric interconnect cluster will show "NOT READY".Sample TestOn Fabric Interconnect A (Primary):UCS1-FA-A# show cluster stateCluster Id: 0x603bca7e0b7311e1-0x9987547fee1fd575A: UP, PRIMARYB: UP, SUBORDINATEHA READYOn Fabric Interconnect B (Subordinate):UCS1-FA-B(local-mgmt)# rebootThe switch will be rebooted. Are you sure? (yes/no):yesRead from remote host 10.134.166.89: Connection reset by peerConnection to 10.134.166.89 closed.Verify on the Primary Fabric Interconnect:UCS1-FA-A# show cluster stateCluster Id: 0x603bca7e0b7311e1-0x9987547fee1fd575A: UP, PRIMARYB: DOWN, INAPPLICABLEHA NOT READYPeer Fabric Interconnect is downTesting UCS Blade failureThis method requires reboot of a UCS blade server. The expected result is that the Overall Status coloumn in the "show server status" command will show "Power Off" while the blade is rebooting.Sample Test[UCS1-80 ~]$ rebootVerify on the Primary Fabric Interconnect:UCS1-FA-A# show server statusServer Slot Status Availability Overall Status Discovery------- --------------------------------- ------------ --------------------- ---------

10/1 Equipped Unavailable Ok Complete10/2 Equipped Unavailable Ok Complete10/3 Equipped Unavailable Ok Complete10/4 Equipped Unavailable Ok Complete10/5 Equipped Unavailable Ok Complete10/6 Equipped Unavailable Ok Complete10/7 Equipped Unavailable Ok Complete10/8 Equipped Unavailable Power Off CompleteTesting Server Port failureThis method requires to disable a server port (interface); e.g. server port 1/9 in fabric interconnect B. The expected result is that the server's "show interface" command in the fabric will show the port's Admin State "Disabled" and Oper State "Failed".Sample TestUCS1-FA-A# scope eth-serverUCS1-FA-A /eth-server # scope fabric bUCS1-FA-A /eth-server/fabric # show interfaceInterface:Slot Id Port Id Admin State Oper State Lic State Grace Prd State Reason---------- ---------- ----------- ---------------- -------------------- --------------- ------------ 1 1 Enabled Up License Ok 0 1 10 Enabled Up License Ok 0 1 11 Enabled Up License Ok 0

1 6 Enabled Up License Ok 0 1 7 Enabled Up License Ok 0 1 8 Enabled Up License Ok 0 1 9 Enabled Up License Ok 0UCS1-FA-A /eth-server/fabric # enter interface 1 9UCS1-FA-A /eth-server/fabric/interface # disableUCS1-FA-A /eth-server/fabric/interface # commit-bufferUCS1-FA-A /eth-server/fabric/interface # exitTo Verify:UCS1-FA-A /eth-server/fabric # show interfaceInterface:Slot Id Port Id Admin State Oper State Lic State Grace Prd State Reason---------- ---------- ----------- ---------------- -------------------- --------------- ------------ 1 1 Enabled Up License Ok 0 1 10 Enabled Up License Ok 0 1 11 Enabled Up License Ok 0

1 6 Enabled Up License Ok 0 1 7 Enabled Up License Ok 0 1 8 Enabled Up License Ok 01 9 Disabled Failed Unknown 0 Admin config changeUCS1-FA-A / eth-server # exitTesting Uplink Port failureThis method requires to disable services on an uplink port, e.g. Uplink Port 1/39 on Fabric B. The expected result is that the fabric's uplink "show interface" command will show the port's Admin State "Disabled" and Oper State "Admin Down".Sample TestUCS1-FA-A# scope eth-uplinkUCS1-FA-A /eth-uplink # scope fabric bUCS1-FA-A /eth-uplink/fabric # show interfaceInterface:Slot Id Port Id Admin State Oper State Lic State Grace Period State Reason---------- ---------- ----------- ---------------- -------------------- --------------- ------------1 36 Disabled Admin Down Unknown 0 Administratively down1 40 Disabled Admin Down Unknown 0 Administratively downMember Port:Port-channel Slot Port Oper State State Reason Lic State Grace Period------------ ----- ----- --------------- ----------------------------------- -------------------- ------------1 1 35 Up License Ok 01 1 39 Up License Ok 02 1 33 Sfp Not Present Unknown Unknown 02 1 37 Sfp Not Present Unknown Unknown 0UCS1-FA-A /eth-uplink/fabric # enter interface 1 39UCS1-FA-A /eth-uplink/fabric/interface* # disableUCS1-FA-A /eth-uplink/fabric/interface* # commit-bufferUCS1-FA-A /eth-uplink/fabric/interface # exitTo verify:UCS1-FA-A /eth-uplink/fabric # show interfaceInterface:Slot Id Port Id Admin State Oper State Lic State Grace Period State Reason---------- ---------- ----------- ---------------- -------------------- --------------- ------------1 36 Disabled Admin Down Unknown 0 Administratively down1 39 Disabled Admin Down Unknown 0 Administratively down1 40 Disabled Admin Down Unknown 0 Administratively downMember Port:Port-channel Slot Port Oper State State Reason Lic State Grace Period------------ ----- ----- --------------- ----------------------------------- -------------------- ------------1 1 35 Up License Ok 02 1 33 Sfp Not Present Unknown Unknown 02 1 37 Sfp Not Present Unknown Unknown 0UCS1-FA-A /eth-uplink/fabric # exit