Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

 
ON THIS PAGE
 

How to Upgrade From Contrail Networking Release 3.x or 4.x with RHOSP10 to Contrail Networking Release 1907 with RHOSP13

The goal of this topic is to provide a combined procedure to upgrade Red Hat OpenStack Platform (RHOSP) from RHOSP 10 to RHOSP 13 by leveraging Red Hat Fast Forward Upgrade (FFU) procedure while simultaneously upgrading Contrail Networking from Release 4.1.x to Release 1907. The procedure leverages the In Service Software Upgrade (ISSU) procedure from Contrail to minimize the downtime.

The downtime will be reduced by not requiring extra server reboots in addition to the ones that the RHOSP FFU procedure already requires for Kernel/RHEL upgrades.

Refer to Red Hat OpenStack Platform 13 Fast Forward Upgrades for details on RHOSP 10 to RHOSP 13 Fast Forward Upgrade (FFU) procedure of OpenStack Platform environment from one long life version to the next long life version.

Note:

This procedure is also applicable for upgrading Contrail Networking Release 3.x or 4.x with RHOSP10 to Contrail Networking Release 5.1.x with RHOSP13.

When to Use This Procedure

We recommend engaging Juniper Networking professional services for assistance with performing this procedure.

This procedure is used when you are running Contrail Networking Release 3 or Contrail Networking Release 4 in an environment using Red Hat Openstack Platform 10 (RHOSP 10) for orchestration and want to upgrade your environment to a Contrail Networking 19 release using RHOSP13. The procedure leverages the Red Hat Fast Forward Upgrade (FFU) procedure for the RHOSP upgrade and the In Service Software Upgrade (ISSU) procedure for the Contrail Networking upgrade.

The procedure in this document has been validated for the following Contrail Networking upgrade scenarios:

Table 1: Validated Upgrade Scenarios

Starting Contrail Networking Release

Target Upgraded Contrail Networking Release

X

X

X

X

X

X

If you want to use this procedure to upgrade your Contrail Networking release to other releases, you must engage Juniper Networks professional services. Contact your Juniper representative for additional information.

Before You Begin

Access ContrailImageTag located at README Access to Contrail Registry 19XX.

Note:

You must enable FFU RedHat subscription for overcloud nodes as the upgrade involves yum update which needs RPM repositories enabled. The subscription must include access to both OSP10 and OSP13 repositories (rhel-7-server-openstack-13-rpms). Additionally, the subscription must have access to the repo rhel-server-rhscl-7-rpms repository. ISSU node uses python27-python-pip from this repository.

ISSU nodes must have internet access during installation to install docker-compose via pip.

If you do not have internet access, after ISSU node installation, you must download required packages manually an a node with internet access, transfer packages on all the ISSU nodes and install them.

Upgrading Contrail Networking Release 3.x or 4.x with RHOSP10 to Contrail Networking Release 1907 with RHOSP13

  1. Follow chapter 2 from step 2.1 through step 2.8 of Red Hat OpenStack Platform 13 Fast Forward Upgrades to prepare for an Openstack Platform Upgrade.
    Note:

    Do not reboot the compute nodes as mentioned in step 2.9.

  2. Prepare to deploy Contrail In-Service Software Upgrade (ISSU) node.
    1. Update Contrail TripleO Heat Templates (Stable/Newton branch) from https://github.com/Juniper/contrail-tripleo-heat-templates.
    2. Update Contrail TripleO Puppet module to the latest version and prepare Swift Artifacts.
    3. Prepare docker registry with Contrail Networking R1907 images. It can be undercloud or a separate node.

      In addition to Contrail Networking images, the required ansible deployer image is contrail-kolla-ansible-deployer which is available at https://hub.docker.com/r/opencontrailnightly/contrail-kolla-ansible-deployer/tags.

      You must use the same tag as of other Contrail Networking containers.

      To prepare images:

      contrail-tripleo-heat-templates/tools/contrail/import_contrail_container_docker.sh

      For details on building container, refer to https://github.com/Juniper/contrail-deployers-containers.

  3. Deploy Contrail In-Service Software Upgrade (ISSU) node.
    1. Prepare new bare-metal server node and create flavor contrail-controller-issu for ISSU node.The hardware requirements for ISSU node is the same as for the Contrail Controller Node.
    2. Prepare parameters in the yaml file.
    3. Run deploy command with all parameters used for deployment and new environment file.
    4. If you do not have internet access, after ISSU node installation, you must download required packages manually an a node with internet access, transfer packages on all the ISSU nodes and install them.

      To do so:

      1. Prepare required modules for docker-compose.

        Run the following commands on the a RHEL7 node which has internet access:

      2. Upload the archive to all the ISSU nodes.

      3. Install packages on all the ISSU nodes.

  4. Run In-Service Software Upgrade (ISSU).
    1. Make SSH connection to ISSU node.
      Note:

      If you have 3 ISSU nodes deployed, you must perform SSH operations and run scripts on the same node for the entire procedure.

    2. Deploy ISSU node.

      cd /etc/contrail/issu ./issu_node_deploy.sh

    3. Check status of Contrail Networking service.

      sudo contrail-status

      The status must be active.

    4. Pair ISSU node with the old cluster.

      ./issu_node_pair.sh

    5. Check status of Contrail service on ISSU node.

      sudo contrail-status

      Note:

      All services must be active except config_devicemgr, config_schema and config_svcmonitor.

    6. Run ISSU sync container.

      ./issu_node_sync.sh

    7. Check ISSU container logs.
  5. Upgrade vRouter on Compute nodes.

    Perform these steps on all the Compute Nodes.

    1. Select the Compute node for upgrade and migrate workload from it.
    2. Make SSH connection to Compute node.
    3. Deploy new vRouter

      cd /etc/contrail/issu/ ./issu_compute_deploy.sh

    4. Verify the network-functions-vrouter-ffu-env-pre file.

      You required this file because os-net-config is outdated in Newton and doesn’t provide the required variables for Contrail Networking Release 5.x or Release 1907 and later.

      For kernel mode and DPDK, the file is created automatically by the script mentioned in the step 5.d . Verify the file is created as per your deployment.

    5. Reboot the node.

      sudo reboot

    6. Make SSH connection to the compute node and verify node status.

      modinfo vrouter (for kernel mode) sudo contrail-status

  6. Upgrade undercloud to OpenStack Plaform (OSP) 13.

    Follow chapter 3 till chapter 5 of Red Hat OpenStack Platform 13 Fast Forward Upgrades to configure a container image source and prepare for the overcloud upgrade.

  7. Copy the TripleO (OOO, OpenStack on OpenStack) Heat Templates and prepare united Contrail parameters for OSP 10 with new parameters for OSP 13.
    1. Backup old TripleO Heat Templates.
    2. Set united for OSP 10 and OSP 13 parameters.
    3. Define NIC templates.
    4. Define role parameters for Computer Node, DPDK, TSN, etc.
    5. Remove contrail-artifacts, if any to avoid rewriting contrail Puppets.
    6. Make SSH connection to ISSU node.
    7. Stop the ISSU service and un-pair ISSU node with the old Contrail Control plane.

      cd /etc/contrail/issu/ sudo docker rm --force issu-run-sync ./issu_node_pair.sh del

  8. Upgrade overcloud to OpenStack Plaform (OSP) 13.

    Follow chapter 6 of Red Hat OpenStack Platform 13 Fast Forward Upgrades.

    1. You can use this role file—tripleo-heat-templates/roles_data_contrail_ffu.yaml or update the role file with ISSU role and ISSU services.
    2. Add the following command environment files:

      • tripleo-heat-templates/environments/contrail/contrail-plugins.yaml

      • tripleo-heat-templates/environments/contrail/contrail-services.yaml

      For example:

    3. Perform the following steps after upgrading all the controller nodes as stated in step 6.2 of Red Hat OpenStack Platform 13 Fast Forward Upgrades:

      1. Upgrade Contrail control plane nodes.

        openstack overcloud upgrade run --nodes ContrailController,ContrailAnalytics,ContrailAnalyticsDatabase,ContrailControllerIssu --skip-tags validation

      2. Check status of Contrail Networking service.

        sudo contrail-status

        The status must be active.

      3. Make SSH connection to ISSU node.

      4. Pair ISSU node with upgraded Contrail control plane.

        cd /etc/contrail/issu ./issu_node_pair.sh add pair_with_new

      5. Check status of Contrail Networking service.

        sudo contrail-status

        On ISSU node, the status for all the services must be active.

        On Contrail control nodes, the status for all the services except config_device_manager, config_schema and config_svc_monitor must be active. The status for config_device_manager, config_schema and config_svc_monitor must be inactive.

      6. Sync data with new Contrail control plane.

        issu_config=issu_revert.conf ./issu_node_sync.sh

      7. Check ISSU Logs.

        sudo docker logs issu-run-sync sudo docker exec issu-run-sync cat /var/log/contrail/issu_contrail_run_sync.log

    4. Follow Step 6.3 of Red Hat OpenStack Platform 13 Fast Forward Upgrades to upgrade compute nodes.
    5. Perform the following steps after upgrading all the compute nodes as stated in step 6.4 of Red Hat OpenStack Platform 13 Fast Forward Upgrades:

      1. Check status of Contrail Networking service on compute nodes.

        sudo contrail-status

        The status must be active.

      2. Make SSH connection to ISSU node.

      3. Disable ISSU node.

        sudo docker rm --force issu-run-sync

      4. Un-pair ISSU node with the old Contrail cluster.

        ./issu_node_pair.sh del pair_with_new

      5. Check status of Contrail Networking service on control nodes.

        sudo contrail-status

        The status must be active.

  9. Follow steps 6.5 till 6.10 of Red Hat OpenStack Platform 13 Fast Forward Upgrades to upgrade CEPH storage node, converged nodes, etc.
  10. Finalize ISSU upgrade.

    Remove ISSU node from the cluster.

    set ContrailControllerIssuCount: 0

    Run stack deploy command.

  11. Follow chapter 7 of Red Hat OpenStack Platform 13 Fast Forward Upgrades to execute post upgrade steps.

See Also