Troubleshooting Chassis Cluster ISSU Failures
This topic includes the following sections:
Viewing the ISSU Progress
Problem
Description: Rather than wait for an ISSU failure, you can display the progress of the ISSU as it occurs, noting any message where the ISSU was unsuccessful. Providing that message to TAC can help resolve the issue.
Solution
After starting an ISSU, issue the show chassis cluster information issu command. Output similar to the following is sent to the console to indicate the progress of the ISSU for all Services Processing Units (SPUs).
Note: Any management session to secondary node will be disconnected. Shutdown NOW! [pid 2480] ISSU: Backup RE Prepare Done Waiting for node1 to reboot. Current time: Tue Apr 22 14:37:32 2014 Max. time to complete: 15min 0sec. Note: For real time ISSU status, open a new management session and run <show chassis cluster information issu> for detail information node1 booted up. Waiting for node1 to become secondary Current time: Tue Apr 22 14:40:32 2014 Max. time to complete: 60min 0sec. Note: For real time ISSU status, open a new management session and run <show chassis cluster information issu> for detail information node1 became secondary. Waiting for node1 to be ready for failover ISSU: Preparing Daemons Current time: Tue Apr 22 14:41:27 2014 Max. time to complete: 60min 0sec. Note: For real time ISSU status, open a new management session and run <show chassis cluster information issu> for detail information Secondary node1 ready for failover. Installing package '/var/tmp/junos-srx5000-12.1I20140421_srx_12q1_x47.0-643920-domestic.tgz' ... Verified SHA1 checksum of issu-indb.tgz Verified junos-boot-srx5000-12.1I20140421_srx_12q1_x47.0-643920.tgz signed by PackageDevelopment_12_1_0 Verified junos-srx5000-12.1I20140421_srx_12q1_x47.0-643920-domestic signed by PackageDevelopment_12_1_0
Stopping ISSU Process When it Halts During an Upgrade
Problem
Description: The ISSU process halts in the middle of an upgrade.
Solution
If the ISSU fails to complete and only one device in the cluster has been upgraded, you can roll back to the previous OS on that device alone by using the following commands on the upgraded device:
- Abort ISSU on both nodes using the request chassis cluster in-service-upgrade abort command.
- Rollback the image using the request system software rollback command with node option.
- Reboot the rolled back node using the request system reboot command.
Recovering the Node in Case of a Failed ISSU
Problem
Description: The ISSU procedure stops progressing.
Solution
Open a new session on the primary device and issue the request chassis cluster in-service-upgrade abort command.
This step aborts an in-progress ISSU . This command must be issued from a session other than the one on which you issued the request system in-service-upgrade command that launched the ISSU. If the node is being upgraded, this command cancels the upgrade. The command is also helpful in recovering the node in case of a failed ISSU.
When an ISSU encounters an unexpected situation that necessitates an abort, the system message provides you with detailed information about when and why the upgrade stopped and recommendations for the next steps to take.
For example, the following message is issued when a node fails to become RG-0 secondary when it boots up:
Rebooting Secondary Node
Shutdown NOW!
[pid 2120]
ISSU: Backup RE Prepare Done
Waiting for node1 to reboot.
node1 booted up.
Waiting for node1 to become secondary
error: wait for node1 to become secondary failed (error-code: 5.1)
ISSU aborted. But, both nodes are in ISSU window.
Please do the following:
1. Log on to the upgraded node.
2. Rollback the image using rollback command with node option
Note: Not using the 'node' option might cause
the images on both nodes to be rolled back
3. Make sure that both nodes (will) have the same image
4. Ensure the node with older image is primary for all RGs
5. Abort ISSU on both nodes
6. Reboot the rolled back node
{primary:node0} | Note: If you attempt to upgrade a device pair running a Junos OS image earlier than Release 9.6, the ISSU will fail without changing anything about either device in the cluster. Devices running Junos OS Releases earlier than 9.6 must be upgraded separately using individual device upgrade procedures. If the secondary device experiences a power-off condition before it boots up using the new image specified when the ISSU is initiated, when power is restored the newly upgraded device will still be waiting to end the ISSU. To end the ISSU, issue the request chassis cluster in-service-upgrade abort command. |
