Support Central: KB #2896: NIOS hardware replacement/swaps for HA and grid configurations

[#1]

Each week, a "hot topic" from support will be posted.  This week - KB Article 2896.

Cause

There may be a time when you have to replace an existing NIOS appliance. This includes replacement of existing hardware with the same model, or even an upgrade from one model to another. 

Resolution

Following, you will find instructions for both preparation before the physical swap, as well as instructions on how to do the actual swap. Before replacing/installing any hardware, make sure you have read the "Read Me First" sheet of paper (for important information on minimum NIOS releases per hardware type) that comes with each Infoblox appliance (assuming a real/non-virtual appliance is being installed).

Please note that power cords and serial cables are NOT supplied with RMA replacements. If you need them, please contact Infoblox Support.

SPECIAL NOTE for all hardware swaps:

When receiving RMA hardware (Return Merchandise Request - a hardware replacement for an existing appliance under maintenance), the replacement will not contain any licenses. It is VERY important that you install licenses on your hardware BEFORE attempting the hardware swap.  You may retrieve you licenses online by logging in to the Infoblox Support web portal and clicking on the "License Key" link (on the left side of the page, towards the bottom, then selecting the type of appliance you need a license key for.  You will also need to know the Hardware ID of the appliance. Simply log in to the appliance, via the CLI, and run the command "show hwid".) To install the license key, you may either use the GUI, or the CLI (via the "set license"command).

If you are unable to retrieve your license keys, send an email to support@infoblox.com, letting them know the hardware ID and/or serial number of the appliance.

SPECIAL NOTE for all vNIOS appliances (VMWare, Cisco, and Riverbed):

License keys will also be needed for vNIOS (VMWare, Riverbed, and Cisco) and can be retrieved via the same method as documented above under the "SPECIAL NOTE FOR ALL HARDWARE SWAPS" section.

Preparations

Standalone, non-Grid device:

Since there is no hardware redundancy for a standalone, non-grid appliance, it is important that a NIOS backup is done before the hardware swap.  Save this backup file (which will have a default file extension of .bak) somewhere you won’t lose it.

High Availability (HA) pair, non-Grid:

Since HA consists of a pair of appliances, there is built in redundancy (as far as the database goes) so you can take advantage of the HA pair in limiting the amount of downtime you and your client machines will experience.  It is still a good idea, however, to take a backup (just in case) of the HA pair and store it in case of issue.  It probably will not be needed, however.  If you have a preference as to which node in the HA pair you want to swap first, or you are only swapping one of the nodes, make sure that it is the PASSIVE node. You may need to do a forced failover of the HA pair to accomplish this.

Grid Master, HA pair:

Since the Grid master is an HA pair, the same procedure should be followed as any other HA pair.  Make sure you take a backup of the grid, just in case, but other than that, no other preparation should be needed.

Grid Member, single box:

This scenario is the best case scenario as there is almost nothing that needs to be done in regards to preparation.  Make sure you take a look at the special notes (below) that apply to all model/configuration types.

Grid Master, single box:

This scenario can be a bit tricky as you do not have HA for your Grid Master, so it is VERY important that you have a Grid Master Candidate configure before continuing.  Failure to verify this important step can make the hardware swap very ugly!  In preparation for the swap of the single box Grid Master, a Grid Master promotion needs to be done.  This will temporarily make the Grid Master Candidate the new Grid Master, and drop the current Grid Master to a member of the grid.  In order to do this, you will want to login via the console (serial console, VMWare console, or SSH) and run the command "set promote_master" on the Grid Master Candidate (if the Grid Master Candidate is an HA pair, this should be done from the ACTIVE node.)  Please review the NIOS Administrators Guide for more information on the use of a Grid Master Candidate for more information.

SPECIAL NOTE for Grid Masters/Members:

You will not be able to join a node to a Grid (master or member) if the appliance does not contain a Grid license before attempting the join.  Use the instructions found above under the "SPECIAL NOTE FOR ALL HARDWARE SWAPS" to obtain/install the Grid license.

Performing the Hardware Swap

When performing the hardware swap, the procedure will depend on the configuration.  Here are the different procedures you should follow, listed by configuration type.

Standalone, non-Grid device:

Again, the challenge here is the lack of hardware redundancy.  There will be downtime during this procedure.

1. Power off the existing appliance (assuming it is still running)
2. Plug in all of the Ethernet cables, and power cable(s) in to the new appliance. Make sure you plug the cables in to the same ports they were plugged in to on the original appliance as there may be multiple subnets involved (ex: MGMT, LAN1, LAN2, and HA.)
3. Power on the new appliance.
4. Via the serial console, VMWare console, CLI, or Web GUI, confirm/reprogram the IP information of the appliance to match the IP address of the old unit. You only need basic IP information (LAN1, and possibly the MGMT port if you are using the MGMT port for grid communications).
5. Do one last validation that all licenses have been installed. You can do this via the CLI command "show licenses", or via the GUI.
6. Restore the database from the backup of the original unit, making sure you select the "Force Restore" check box, and select the bullet that overwrite the IP information from the Backup. We want to make sure that all original IP information is reinstalled on the new appliance.

The system will restart and will come up with the database of the original appliance.

High Availability (HA) pair, non-Grid:

In the preparation, you have made sure that the first (or possibly only) node you will be swapping is now the PASSIVE node, so follow these procedures to swap the PASSIVE node:

1. Record the VIP address of your HA pair.
2. Power off the PASSIVE node (assuming it is running).
3. Install the new PASSIVE node hardware, installing all cable in the proper ports.
4. Power up the new PASSIVE node.
5. Via the serial console, VMWare console, CLI, or Web GUI, confirm/reprogram the IP information of the appliance to match the IP address of the old unit.  You only need basic IP information (LAN1, and possibly the MGMT port if you are using the MGMT port for grid communications).
   a.) Do not join the HA pair at this time.  Simply change the IP address.
   b.) After changing the IP address, test network connectivity from the CLI via the PING command.  PING the router IP address, as well as the VIP of the HA pair.  Both should respond before you continue.
6. Using either the GUI, or the CLI (via the command "set membership"), join the PASSIVE to the ACTIVE.
7. If you are only swapping one node of your HA pair, you have completed the task so please stop here!
8. Assuming you are upgrading the other node of the HA pair, perform a "force failover" of the HA pair.  The new hardware will now be the ACTIVE, the old hardware will be PASSIVE.  Repeat steps 1 thru 6 to swap the second node.

Grid Member, single box:

1. Record the VIP address of your Grid Master (assuming it’s an HA pair), or the LAN IP address.
2. Power off the single member you are replacing (assuming it is running).
3. Install the new hardware, installing all cables in the proper ports.
4. Power up the new node.
5. Via the serial console, VMWare console, CLI, or Web GUI, confirm/reprogram the IP information of the appliance to match the IP address of the old unit. You only need basic IP information (LAN1, and possibly the MGMT port if you are using the MGMT port for grid communications).
   a.) Do not join the node to the grid at this time.  Simply change the IP address.
   b.) After changing the IP address, test network connectivity from the CLI via the PING command.  PING the router IP address, as well as the          VIP/LAN of the Grid Master.  Both should respond before you continue.
6. Using either the GUI, or the CLI (via the command "set membership"), join the node to the grid.

Grid Member, HA pair:

Joining an HA pair node to a grid is no different than joining a single member to a grid.  The same procedure should be used, with one minor addition. To limit downtime, always swap the PASSIVE unit first.  After it is up and running, do an HA Failover, then do the second node.

Grid Master, single box:

Since the original Grid Master is now a Grid Member (a Grid Master promotion was done during the preparation), simply follow the procedure for swapping a single Grid Member. When the swap is complete, and the node is fully synchronized, you can return the Grid Master to its original state by logging in to it’s CLI and running the command "set promote_master".  This will force it to become the Grid Master again.

Grid Master, HA pair:

Again, the same rules apply for the Grid Master HA pair that apply to a Grid Member HA pair, with a couple of noteworthy exceptions:

1. Always replace the PASSIVE unit first.
2. When doing the HA failover (after completing the swap of the PASSIVE) you will lose your GUI connections to the grid.  Also, all members will temporarily become "offline members", then they will rejoin the grid, connecting to the new Grid Master ACTIVE node.