Support Central: KB #80 - One of the Nodes in an HA pair goes into a Continuous restart loop

[#1]

Have you experienced an HA pair going into a continuous restart loop?  We've got the answer!

 

PROBLEM SUMMARY

One of the nodes in an HA pair has suddenly gone into a continuous HA restart loop. This node was working perfectly for a long time and there were no changes in network or switch port configurations in the recent past.

 

CAUSE

Console or LCD output:

[2011/03/15 10:01:31.615] Infoblox system initializing...
[2011/03/15 10:01:32.615] LAN port IP X.X.X.X, netmask 255.255.- - - . -, gateway X.X.X.X
[2011/03/15 10:01:56.966] Starting services...
login: [ 10:02:12.435] System restart: HA restart...

Incorrect switch port settings or external network issues are the most likely reasons for this problem.  Typically HA restart loop occurs when both nodes in the HA pair try to become active at the same time because they do not see each other's VRRP multicasts for a brief period and one node goes into HA restart continuously. Even if there was an existing network configuration issue, sometimes both nodes will work in active/passive states without any problem as long as the passive continuously receives VRRP multicasts from the active. However, in such scenarios, HA issues suddenly surface after an event.  For example, an HA failover on the Infoblox HA pair due to temporary network disruption, an administrator initiated HA failover, a hardware replacement, a configuration change, NIOS upgrades or a similar event can trigger HA issues if there was an existing underlying cause.  These events may cause switch ports to renegotiate and not having the recommended switch port settings can introduce delay during the renegotiation affecting the timely transmission/reception of VRRP messages.

 

RESOLUTION

What are the switch settings to be checked?

• Portfast should be enabled on all switch ports connected to an Infoblox device.
• Turn off trunking on all switch ports connected to an Infoblox device.
• Disable "port channel" on all switch ports connected to an Infoblox device.
• Match speed/duplex settings of the switch ports with the interface settings of  the Infoblox device.
• Verify that all four interfaces of the HA pair are connected to the same VLAN.

Command to use for Cisco switches:

• Command for Cisco IOS to do all of the above is: switchport host
• Command for Cisco CatOS to do all of the above is: set port host <interface/port>

If you are not sure whether your Cisco switches are configured per recommendation above, please run this command on all four switch ports (LAN/HA of both nodes) and check whether this stabilizes the HA. For other vendors please use equivalent configuration commands.  

Please refer to the following KBs for detailed information on HA configuration:

KB # 112
KB # 131
KB # 79
KB # 132
KB # 119

[#2]

I had this exact same problem a few days ago, the IOS command "switchport host" immediately fixed the problem. The network engineer I was working with hadn't seen that command before, they were used to setting the port parameters manually, so they learnt something too! :-)

Paul

[#3]

Glad that it worked out for you guys!