Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

 
ON THIS PAGE
 

Upgrading Contrail Networking with Red Hat Openstack 13 using ISSU

This document provides steps to upgrade Contrail Networking with an in-service software upgrade (ISSU) in an environment using Red Hat Openstack Platform 13 (RHOSP13).

When to Use This Procedure

Use this procedure to upgrade Contrail Networking when it is running in environments using RHOSP13.

This procedure has been validated for the following Contrail Networking upgrades:

Table 1: Contrail Networking with RHOSP13 Validated Upgrade Scenarios

Starting Contrail Networking Release

Target Contrail Networking Upgrade Release

5.0 or 5.1

1907

1907

1908

1908

1909

1909

1910

1910

1911

1911

1912

Starting in Contrail Networking Releases 1912.L0 and 2003, use the Zero Impact Upgrade (ZIU) procedure to upgrade Contrail Networking in environments using Red Hat Openstack orchestration. See Updating Contrail Networking using the Zero Impact Upgrade Process in an Environment using Red Hat Openstack.

If you want to use this procedure to upgrade your Contrail Networking release to other releases, you must engage Juniper Networks professional services. Contact your Juniper representative for information on working with professional services.

Before you begin

  • Obtain the ContrailImageTag value for your Contrail Networking release. You can obtain this value from the readme files at the following locations:

  • Enable RHEL subscription for the overcloud nodes.

  • Enable SSH migration for the Compute nodes if you do not have CEPH or alike storage.

    Upgrading the compute nodes requires workload migrations and CEPH or alike storage allows VM migration.

    • Modify MigrationSshKey value at ~/tripleo-heat-templates/environments/contrail/contrail-services.yaml file.

      The MigrationSshKey parameter with SSH keys for migration is typically provided during the overcloud deployment. The parameter is used to pass SSH keys between computes nodes to allow a VM to migrate from one compute node to another. The MigrationSshKey parameter is an optional parameter that can be added to the contrail-services.yaml file. The parameter is not included in the contrail-services.yaml file by default.

      Run the following commands to find out the SSH keys:

      (undercloud) [stack@queensa ~]$ cat .ssh/id_rsa

      (undercloud) [stack@queensa ~]$ cat .ssh/id_rsa.pub

  • Backup the Contrail configuration database.

    See How to Backup and Restore Contrail Databases in JSON Format.

Procedure

  1. Get Contrail TripleO Heat Templates (Stable/Queens branch) from https://github.com/Juniper/contrail-tripleo-heat-templates.

    Take a back up of the existing directory if you are copying the latest directory, contrail-tripleo-heat-templates. You need to restore the configuration in contrail-net.yaml, contrail-services.yaml, compute-nic-config.yaml (for compute node running kernel mode), and contrail-dpdk-nic-config.yaml (for compute node running dpdk mode) files.

  2. Update Contrail TripleO Puppet module to the latest version and prepare Swift Artifacts, as applicable.
  3. Prepare docker registry with Contrail Networking images. It can be undercloud or a separate node.
  4. Update the version of Red Hat running in the undercloud.
    Note:

    This procedure updates the version of Red Hat running in the undercloud before deploying the Contrail Controller In-Service Software Upgrade (ISSU) node in 5.

    You can deploy the Contrail Controller In-Service Software Upgrade (ISSU) node before performing this step if there is a reason to change the sequence in your environment.

    Before you begin the upgrade process:

    • Identify the Red Hat software to run with your version of Contrail Networking. See Contrail Networking Supported Platforms.

    • If you have updated the undercloud using a copy of the heat templates, copy the heat templates from /usr/share/openstack-tripleo-heat-templates to /home/stack/tripleo-heat-templates.

    • Add the new server nodes as bare metal nodes, and run introspection on the nodes to make them ready for deployment. These steps are summarized in Setting Up the Overcloud.

    For details about performing this upgrade process, refer to RedHat Chapter 3. Upgrading the undercloud.

    If you come across an issue during the update, see Failed upgrade run command for any overcloud node.

  5. Deploy the Contrail Controller In-Service Software Upgrade (ISSU) node.
    1. Prepare new server node and create flavor contrail-controller-issu for the ISSU node.The hardware requirements for ISSU node is the same as for the Contrail Controller Node.
    2. Prepare the parameters in the yaml file, ~/tripleo-heat-templates/environments/contrail/contrail-issu.yaml:

      • ContrailIssuSshKey—Generate and set the ssh keys. You require SSH access between ISSU and Contrail Controller nodes.

        ContrailIssuSshKey is same as MigrationSshKey .

      • ContrailIssuImageTag—Set the new docker images tag for the upgrade procedure.

      • ContrailControllerIssuCount—Set the required number of ISSU nodes. The value can be 1 or 3 and is dependent on various cluster requirements including cluster size, expected upgrade duration, etc.

    3. Update ServiceNetMap parameter in the ~/tripleo-heat-templates/environments/contrail/contrail-services.yaml file.

      ContrailIssuControlNetwork—Set the same value as ContrailControlNetwork. The default value is tenant.

    4. Run deploy command with all the parameters used for deployment and the new environment file.
    5. Check the status of Contrail service on the ISSU node.

      All services must be active .

      sudo contrail-status

  6. Prepare for the upgrade procedure.
    1. Update the parameter ContrailImageTag to the new version.
    2. Download the new OpenStack container and use the new overcloud_images.yaml environment file which has the new containers.

      Upload the OpenStack containers.

    3. Run the openstack overcloud upgrade prepare --stack overcloud --templates ~/tripleo-heat-templates command with all the options from deploy and the ISSU node to update the heat templates.

      The files that are updated in this step vary by deployment. In the following example, the overcloud_images.yaml, network-isolation.yaml, contrail-plugins.yaml, contrail-services.yaml, contrail-net.yaml, contrail-issu.yaml, and roles_data.yam are prepared for the overcloud update.

  7. Run In-Service Software Upgrade (ISSU) sync.
    1. Make SSH connection to the ISSU node.
      Note:

      If you have 3 ISSU nodes deployed, you must perform SSH operations and run scripts on the same node for the entire procedure.

    2. Locate ISSU directory.

      cd /etc/contrail/issu

    3. Pair ISSU node with the old cluster.

      ./issu_node_pair.sh

    4. Check the status of Contrail service on the ISSU node.

      The config_devicemgr, config_schema, and config_svcmonitor containers should all be in the inactive state. All other containers should be in the active state.

    5. Run the ISSU sync container.

      ./issu_node_sync.sh

    6. Check ISSU container logs.
    7. Restart contrail_control_control container on all the ISSU nodes.
      Note:

      The issu_node_sync script is run in step 7.e.

      ISSU nodes are not rebooted during this upgrade procedure when these instructions can be precisely followed. ISSU node reboots, however, are sometimes required in specialized circumstances.

      If an ISSU node is rebooted after step 7.e, rerun the issu_node_sync script:

      This script starts the issu_node_sync container and stops the config_devicemgr, config_schema, and config_svcmonitor containers.

      After running the issu_node_sync script, you can verify that the issu-run-sync container is active and running:

      docker ps -a | grep issu-run-sync

      You must also restart the contrail_control_control container on all the ISSU nodes after the issu_node_sync script is run:

  8. Upgrade the Compute nodes.

    Perform these steps on all the Compute Nodes.

    1. Select the Compute node for upgrade and migrate workload from it.
    2. Verify the migrated instance has active state.
    3. Upgrade the selected Compute Nodes.

      You can use comma-separated list for the various Compute nodes.

      Run the following commands on the undercloud node:

      Run the following commands on the undercloud node:

    4. If the compute nodes use a new kernel or new system-level components after step 8.c, perform the following steps:

      1. Reboot the selected nodes.

      2. For kernel-mode compute nodes:

        Make SSH connection to the upgrades nodes.

    5. If reboot is not required after step 8.c, re-initialize vhost0 interfaces on all the DPDK mode compute nodes.

      Make SSH connection to the upgraded Compute nodes and run the following commands:

    6. Check the status of Contrail service on the upgraded Compute nodes.

      sudo contrail-status

      The status must be active.

  9. Upgrade Contrail Plugins including Neutron, Heat, etc. on OpenStack controllers and connect them to the ISSU node.

    Example for environment with a single OpenStack controller:

    Example for environment with multiple Openstack controllers (3 controllers shown):

  10. Disconnect the ISSU node from the Contrail control plane.
    1. Make SSH connection to ISSU node.
    2. Run the following commands:
    3. Check the status of Contrail service on the ISSU node.

      sudo contrail-status

      The status must be active or backup.

  11. Upgrade the Contrail control plane node.
    1. Run the following commands:
    2. Check the status of Contrail service on the Contrail control plane node.

      sudo contrail-status

      The status must be active or backup.

  12. Upgrade Contrail Analytics and Contrail AnalyticsDB nodes:

    Example for an environment with three Contrail Analytics and three Contrail AnalyticsDB nodes:

  13. Connect the ISSU node to the upgraded Contrail control plane node.
    1. Make SSH connection to the ISSU node.
    2. Pair the ISSU node with upgraded Contrail control plane.

      cd /etc/contrail/issu ./issu_node_pair.sh add pair_with_new

    3. Sync data with new Contrail control plane.

      issu_config=issu_revert.conf ./issu_node_sync.sh

    4. Restart control container on the upgraded nodes.

      Run the following command from the Director.

      openstack server list --name "overcloud-contrailcontroller-" -c Networks -f value | cut -d'=' -f2 | xargs -i ssh heat-admin@{} sudo docker restart contrail_control_control

  14. Run the post upgrade task on the compute nodes and the Openstack controllers.
  15. Disconnect ISSU and upgraded Contrail control plane.
    1. Make SSH connection to ISSU node.
    2. Un-pair ISSU node with the old Contrail cluster.
  16. Reconnect the OpenStack nodes and Compute nodes to the upgraded control plane.

    Run the command with all the parameters from deploy.

  17. If the nodes use new kernel or new system level components, reboot the OpenStack controller and Contrail controller nodes.

    • Reboot OpenStack controllers as mentioned in section 5.1 of RedHat Rebooting the Overcloud chapter.

    • Reboot Contrail controllers one by one.

      Make SSH connection to each controller and perform sudo reboot.You must wait till the node is rebooted and Contrail services are up.

      sudo contrail-status

  18. Check the status of Contrail service on all the upgrades nodes.

    sudo contrail-status

    The status must be active.

  19. Remove the ISSU node from the cluster.

    set ContrailControllerIssuCount: 0

    Run stack deploy command with all the parameters.

    openstack overcloud deploy \ … -e ~/tripleo-heat-templates/environments/contrail/contrail-issu.yaml

Troubleshoot

Following are the known issues:

Failed upgrade run command for OpenStack controller

Problem

Description

You see the following error:

For details, refer to https://access.redhat.com/solutions/4122571.

Solution

  • Make SSH connection to the OpenStack controller node.

  • Run the following command:sudo docker rm cinder_volume_init_bundle

  • Check if the cinder volume is in failed resources list.

    sudo pcs status

  • Check if the cinder volume is not in failed resource list.

    sudo pcs resource cleanup

  • Re-run the upgrade run command.

Failed upgrade run command for any overcloud node

Problem

Description

You see the following error:

Solution

This is a broken default behavior if a variable is missing.

Edit the tripleo-heat-templates/common/deploy-steps.j2 to apply the following change:

After editing the deploy-steps.j2, run the prepare command as given in step 5.6.c.Once it is completed, continue the upgrade procedure where you left off.

Related Documentation