Upgrade Procedure for RHOSP-based Contrail 4.1.3 or 4.1.4 to Contrail 4.1.4.1
This section presents the steps to upgrade a RHOSP-based Contrail deployment from Contrail version 4.1.3 or 4.1.4 to Contrail version 4.1.4.1.
Prerequisites
Ensure you have a cloud up and running with RHOSP10 and Contrail 4.1.3 or Contrail 4.1.4 before you proceed with the upgrade procedure.
This procedure has been validated with the following Contrail, Red Hat, and OpenStack versions.
Contrail Version |
Red Hat Version |
OpenStack Version |
---|---|---|
4.1.4 |
RHEL 7.5 (3.10.0-862.14.4.e17.x86_64) |
RHOSP10 (packages dated October 29, 2018) |
4.1.3 |
RHEL 7.5 (3.10.0-862.11.6.el7.x86_64) |
RHOSP10 (packages dated October 29, 2018) |
Post-Installation
After the installation, you’ll have a cloud networking running RHOSP10 and Contrail 4.1.4.1. The Red Hat Enterprise Linux (RHEL) kernel version remains at 7.5 during this procedure.
Table 2 summarizes the post-installation software versions.
Contrail Version |
Red Hat Version |
OpenStack Version |
---|---|---|
4.1.4.1 |
RHEL 7.5 (3.10.0-862.14.4.e17.x86_64) |
RHOSP10 (packages dated October 29, 2018) |
Net-SNMP Package Requirements
Contrail version R4.1.4.1 supports net-snmp package version 5.7.2-43 to support SNMP in most environments.
The net-snmp packages come from Red Hat, with the exception of the net-snmp-python-5.7.2-43.el7.x86_64.rpm package which is provided in the Contrail repository.
Table 3 summarizes the net-snmp depend packages and their associated repository locations.
Net-SNMP Depend Packages |
Initial Repository |
---|---|
net-snmp-5.7.2-43.el7.x86_64.rpm |
Red Hat Satellite |
net-snmp-agent-libs-5.7.2-43.el7.x86_64.rpm |
Red Hat Satellite |
net-snmp-libs-5.7.2-43.el7.x86_64.rpm |
Red Hat Satellite |
net-snmp-python-5.7.2-43.el7.x86_64.rpm |
Contrail |
net-snmp-utils-5.7.2-43.el7.x86_64.rpm |
Red Hat Satellite |
If you are working in an environment with devices that have dependencies on older net SNMP packages, replace the net SNMP packages with the following version 37-based SNMP packages.
Net-SNMP Depend Packages |
Initial Repository |
---|---|
net-snmp-5.7.2-37.el7.x86_64.rpm |
Red Hat Satellite |
net-snmp-agent-libs-5.7.2-37.el7.x86_64.rpm |
Red Hat Satellite |
net-snmp-libs-5.7.2-37.el7.x86_64.rpm |
Red Hat Satellite |
net-snmp-python-5.7.2-37.el7.x86_64.rpm |
Contrail |
net-snmp-utils-5.7.2-37.el7.x86_64.rpm |
Red Hat Satellite |
If you need to replace the SNMP packages, run the createrepo
command to create the repository and complete the update.
Acquire the Software
To download the software images for this procedure:
Go to the Juniper Networks Support site for Contrail.
Select OS as Contrail and Version as 4.1.4. Download the images that apply to your environment.
Preparing the Environment
Log in to the undercloud as the stack user.
$ su – stack
Update the Contrail repositories.
Backup the Contrail 4.1.3 or 4.1.4 packages to a repository with a different name.
In this example, the Contrail 4.1.3 packages are moved to a repository named contrail-R4-1-3.
[stack@undercloud ~]$ cd /var/www/html/ [stack@undercloud html]$ sudo mv contrail/ contrail-R4-1-3
Create a new repository directory to store the Contrail 4.1.4.1 packages:
[stack@undercloud html]$ sudo mkdir contrail
Copy the downloaded file—in the provided sample, the file is contrail-install-packages_4.1.4.1-10-newton.tgz—to the Contrail repository created in Step 2.
Note:This step assumes that you've already downloaded the Contrail software. See Acquire the Software.
[stack@undercloud contrail]$ ls -lrt total 377104 -rw-r--r--. 1 root root 386151602 Mar 14 06:58 contrail-install-packages_4.1.4.1-10-newton.tgz
Untar the downloaded tgz file.
[stack@undercloud contrail]$ sudo tar -xvf contrail-install-packages_4.1.4.1-10-newton.tgz
Ensure your satellite has a supported net-snmp package setup. The net-snmp packages in the Contrail repository and the satellite repository must match. See Net-SNMP Package Requirements.
Create a repository in the new directory:
[stack@undercloud contrail]$ pwd /var/www/html/contrail [stack@undercloud contrail]$ sudo createrepo .
If the createrepo command is not available, download the createrepo package from Red Hat (Red Hat subscription required).
(Clusters deployed using Swift Puppet files only) If your Contrail 4.1 cluster was deployed using Swift Puppet, perform these steps:
Remove overcloud artifacts from the undercloud:
[stack@undercloud ~]$ swift delete overcloud-artifacts puppet-modules.tgz overcloud-artifacts
Delete the deployments-artifacts.yaml file if the file is present.
[stack@undercloud ~]$ ls /home/stack/.tripleo/environments/deployment-artifacts.yaml [stack@undercloud ~]$ rm -rf /home/stack/.tripleo/environments/deployment-artifacts.yaml
Clean the repositories and confirm that all repositories are available.
[stack@undercloud ~]$ sudo yum clean all [stack@undercloud ~]$ sudo yum repolist
Note:The undercloud is often updated at this point of the Contrail Networking upgrade procedure. The undercloud does not need to be updated for this procedure since the RHEL and RHOSP versions are unchanged.
Ensure the undercloud has the latest Contrail R4.1.4.1 contrail packages:
[stack@undercloud ~]$ rpm -qa | grep contrail puppet-contrail-4.1.4.1-10.el7.noarch contrail-tripleo-heat-templates-4.1.4.1-10.el7.noarch contrail-tripleo-puppet-4.1.4.1-10.el7.noarch python-gevent-1.1rc5-1contrail1.el7.x86_64
Review the ironic node-list output to confirm the following statuses for each package::
Power state is power on.
Provision State is active.
Maintenance is False.
[stack@undercloud ~]$ ironic node-list +-------------------------------+----------+--------------+-------------+ | Name | Power | Provisioning | Maintenance | | | State | State | | +-------------------------------+----------+--------------+-------------+ | controller-3 | power on | active | False | | compute-5c5s35 | power on | active | False | | contrail-controller1 | power on | active | False | | contrail-analytics1 | power on | active | False | | contrail-controller-3 | power on | active | False | | contrail-controller-2 | power on | active | False | | contrail-analytics-database1 | power on | active | False | | controller-2 | power on | active | False | | controller1 | power on | active | False | | compute-5c5s37 | power on | active | False | | compute-5c5s36 | power on | active | False | | contrail-analytics-2 | power on | active | False | | contrail-analytics-3 | power on | active | False | | compute-5c5s38 | power on | active | False | | contrail-analytics-database-3 | power on | active | False | | contrail-analytics-database-2 | power on | active | False | +-------------------------------+----------+--------------+-------------+
Note:This output presentation has been modified for readability. The UUID and Instance UUID fields were removed as part of this modification.
Verify that all OpenStack servers are in the Active state.
[stack@undercloud ~]$ openstack server list +-------------------------------------------------+--------+ | Name | Status | +-------------------------------------------------+--------+ | overcloud-contrailanalytics-2-4-1-4-7-7 | ACTIVE | | overcloud-controller-0-4-1-4-7-7 | ACTIVE | | overcloud-contrailanalytics-0-4-1-4-7-7 | ACTIVE | | overcloud-contrailanalyticsdatabase-2-4-1-4-7-7 | ACTIVE | | overcloud-contrailanalytics-1-4-1-4-7-7 | ACTIVE | | overcloud-contrailanalyticsdatabase-0-4-1-4-7-7 | ACTIVE | | overcloud-contrailcontroller-1-4-1-4-7-7 | ACTIVE | | overcloud-contrailanalyticsdatabase-1-4-1-4-7-7 | ACTIVE | | overcloud-contrailcontroller-2-4-1-4-7-7 | ACTIVE | | overcloud-contrailcontroller-0-4-1-4-7-7 | ACTIVE | | compute-0-4-1-4-rhel-7-7 | ACTIVE | | overcloud-contraildpdk-0-4-1-4-7-7 | ACTIVE | | overcloud-contraildpdk-1-4-1-4-7-7 | ACTIVE | | compute-1-4-1-4-rhel-7-7 | ACTIVE | +-------------------------------------------------+--------+
Note:This output presentation has been modified for readability. The ID, Image Name, and Networks fields were removed as part of this modification.
Note:Overcloud packages are often downloaded from the Contrail satellite at this point of other Contrail upgrade procedures. The overcloud is not upgraded in this procedure so these steps do not need to be performed.
Observe the contrail-status on all Contrail nodes. All services in the Contrail nodes, except the controller (OpenStack), should be up and running before proceeding with the upgrade.
[stack@undercloud ~]$ source stackrc [stack@undercloud ~]$ for i in $(nova list | grep contrail | awk '{print $12}' | cut -d '=' -f2); do ssh heat-admin@$i sudo contrail-status; done
The Contrail repository must be enabled on all overcloud nodes.
Ensure that all overcloud node contrail repository pointers are properly pointing to the contrail repository.
Contrail Analytics Example:
[root@overcloud-contrailanalytics-0 heat-admin]# cat /etc/yum.repos.d/contrail.repo [Contrail] name=Contrail Repo baseurl=http://192.168.24.1/contrail enabled=1 gpgcheck=0 protect=1 metadata_expire=30
Before proceeding with the upgrade, consider performing the following tasks to optimize your results:
Clean the yum cache on all overcloud virtual machines.
Disable the subscriptions from the satellite to ensure new packages are not accidentally loaded during the procedure.
Synch the satellites to the Oct 29, 2018 satellite repo end date.
Upgrade the Contrail Heat Templates
You must copy the new Contrail Heat templates package to the undercloud node, while retaining a copy of the Heat templates that were used for the existing deployment.
Make a copy of all of the Heat templates that were used for deployment and save the copies, because the existing files will be overwritten by the new versions. The templates to copy are of the form contrail-services.yaml, contrail-net.yaml, and so on.
Note:Red Hat does not support changing IP address of the existing cluster as a part of upgrade. Do not change IP address of the cluster while creating new tripleo-heat-templates
Copy the new contrail-tripleo-heat templates to the undercloud node.
sudo cp -r /usr/share/contrail-tripleo-heat-templates/environments/contrail /home/stack/tripleo-heat-templates/environments/ sudo cp -r /usr/share/contrail-tripleo-heat-templates/puppet/services/network/* /home/stack/tripleo-heat-templates/puppet/services/network
Note:The directory /home/stack/tripleo-heat-templates is user defined, it can be User Defined-directory>/openstack-tripleo-heat-templates
Modify the Yum Update Script for TripleO Puppet
The following Puppet commands must be added to the yum_update
script before starting the upgrade. The
script is located at:
/home/stack/tripleo-heat-templates/extraconfig/tasks/yum_update.sh
Update the following Puppet commands in the
yum_update.sh
after the line“echo -n "false" > $heat_outputs_path.update_managed_packages”
.Refer to the following patch for details regarding the exact placement of the commands patch: https://github.com/Juniper/contrail-tripleo-heat-templates/blob/stable/newton/environments/contrail/yum_updates.patch
yum install -y contrail-tripleo-puppet puppet-contrail
rsync -a /usr/share/contrail-tripleo-puppet/ /usr/share/openstack-puppet/modules/tripleo/
Update the fields
*contrail version
and*contrail repo
incontrail-services.yaml
.Default parameter for
contrailVersion
is 4.Filename:/home/stack/tripleo-heat-templates/environments/contrail/contrail-services.yaml.
Add the following parameters:
ContrailVersion: 4
ContrailRepo : <location of the contrail repo>
Note:/home/stack/tripleo-heat-templates directory is user defined and it can be the directory name under stack user.
Update the Overcloud Deployment Plan
Update the current plan by re-running the command used for cloud deployment and adding the suffix
- -update-plan-only
.openstack overcloud deploy –update-plan-only
Example:
openstack overcloud deploy --update-plan-only --templates /home/stack/tripleo-heat-templates/ \ --roles-file /home/stack/tripleo-heat-templates/environments/contrail/roles_data.yaml \ -e /home/stack/tripleo-heat-templates/environments/puppet-pacemaker.yaml \ -e /home/stack/tripleo-heat-templates/environments/contrail/contrail-services.yaml \ -e /home/stack/tripleo-heat-templates/environments/contrail/network-isolation.yaml \ -e /home/stack/tripleo-heat-templates/environments/contrail/contrail-net.yaml \ -e /home/stack/tripleo-heat-templates/environments/ips-from-pool-all.yaml \ -e /home/stack/tripleo-heat-templates/environments/network-management.yaml \ -e /home/stack/tripleo-heat-templates/extraconfig/pre_deploy/rhel-registration/environment-rhel-registration.yaml \ -e /home/stack/tripleo-heat-templates/extraconfig/pre_deploy/rhel-registration/rhel-registration-resource-registry.yaml \ --libvirt-type qemu
Make a copy of the existing deploy script to the
update-stack.sh
. Theupdate-stack.sh
is the script used to update the overcloud plan, and it references the same templates that were used to deploy the stack. All files used for the overcloud update should be identical to the files used for deployment, exceptcontrail-services
file that was updated with the latestcontrail-version
andcontrail-repo
.cp deploy.sh update-stack.sh
Update the deployment plan.
./update-stack.sh
Example:
[stack@undercloud ~]$ ./update-stack.sh Removing the current plan files Uploading new plan files Started Mistral Workflow. Execution ID: 998a1b40--a034-8cff453acfb1 Plan updated Deploying templates in the directory /tmp/tripleoclient-JulIDe/tripleo- heat-templates Overcloud Endpoint: http://10.0.0.35:5000/v2.0 Overcloud Deployed
Upgrade the Overcloud
The steps in this section are service disrupting, and should only be performed within a maintenance window.
Update the overcloud stack.
$ openstack overcloud update stack -i overcloud on_breakpoint: [u'overcloud-contrailanalyticsdatabase-0'] Breakpoint reached, continue? Regexp or Enter=proceed (will clear 4386bdc7-5087-4a4d-865c-0b0181ce9345), no=cancel update, C-c=quit interactive mode: IN_PROGRESS IN_PROGRESS IN_PROGRESS IN_PROGRESS IN_PROGRESS COMPLETE update finished with status COMPLETE
Verify the overcloud
stack status
, thecontrail-status
, and thecontrail-version
after the upgrade.Overcloud Stack Status
[stack@undercloud ~]$ openstack stack list +------------+-----------------+----------------------+----------------------+ | Stack Name | Stack Status | Creation Time | Updated Time | +------------+-----------------+----------------------+----------------------+ | overcloud | UPDATE_COMPLETE | 202--03-17T23:30:26Z | 2020-03-17T22:40:01Z | +------------+-----------------+----------------------+----------------------+
Note:The openstack stack list output presentation has been modified for readability. The ID field was removed as part of this modification.
Contrail Stack Status
sudo for i in $(nova list | grep contrail | awk '{print $12}' | cut -d '=' -f2); do ssh heat-admin@$i sudo contrail-status; done
Contrail Version Check
for i in $(nova list | grep contrail | awk '{print $12}' | cut -d '=' -f2); do ssh heat-admin@$i sudo contrail-version; ssh heat-admin@$i sudo contrail-status -d ; done
Upgrade Cautions
The steps to perform the overcloud upgrade are service disrupting, and should only be performed within a maintenance window.
Contrail-Named Services Remains Down After Upgrade
In rare cases, the contrail-named service stays down after an upgrade.
Workaround: Restart the service once using the service contrail-named restart command.
To identify and respond to this issue:
overcloud-contrailcontroller-1 contrail]# service contrail-named status -d contrail-named FATAL Exited too quickly (process log may have details) [root@overcloud-contrailcontroller-1 contrail]# [root@overcloud-contrailcontroller-1 contrail]# service contrail-named restart