Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

TechLibrary
Navigation
Table of Contents
Guide That Contains This Content
[+] Expand All
[-] Collapse All

Download This Guide

 

Related Documentation

 

Contrail In-Service Software Upgrade

Contrail In-Service Software Upgrade (ISSU) Overview

Starting with Contrail Release 3.2, it is possible to upgrade an existing Contrail installation with a newer version of Contrail, without interrupting system operation, by performing an in-service software upgrade (ISSU).

For ISSU, the Contrail controller cluster is upgraded side-by-side with a parallel setup, and the compute nodes are upgraded in place.

It is possible to perform ISSU with Fabric commands to consolidate the manual commands necessary for ISSU.

This section describes the steps to perform Contrail ISSU, including detailed manual command-line (CLI) procedures, and the corresponding Fabric commands created to streamline the ISSU process.

The candidate cluster for an ISSU upgrade should have the OpenStack and Contrail processes installed on separate nodes.

There will be a maintenance window during which the upgrade process is ongoing. However, existing workloads will not be or will be very minimally disturbed.

The essential ISSU steps are:

  1. Prepare the system.
  2. Configure and run time synchronization.
  3. Upgrade compute nodes.
  4. Rollback compute nodes if needed.
  5. Finalize the upgrade.

Fabric Commands for Upgrade Procedure

Contrail ISSU can be performed manually or you can use Fabric commands that consolidate some of the manual steps.

Fabric commands for ISSU should be initiated from the newer Contrail controller.

Table 1 compares the fab commands with the corresponding CLI commands. For more details about the CLI commands, see Details of the CLI Manual ISSU Procedure.

Table 1: ISSU Fabric Commands and CLI Commands

Fabric Commands

Manual Commands and Summary Description

fab issu_contrail_migrate_config

New roles are defined in the testbed.py to provide information about the existing controller.

  1. Start a parallel setup—install the same number of Contrail controller instances that are currently running, and that need an upgrade with newer software.
    • The newer Contrail instances can point to the existing database, or install a newer database as needed for migration.
  2. Prepare the system for upgrade—BGP peer the Contrail controllers of the existing version and the newer version.
  3. Freeze the Contrail north bound communication.
  4. Perform initial configuration synchronization:

    issu_contrail_pre_sync.py

  5. Perform a runtime configuration sync between the existing Contrail controller and the new controller:

    issu_contrail_run_sync.py

    This sync is used until the Contrail north bound communication is migrated to the new controller.

  6. Unfreeze the north bound communication, and the system is ready for compute node migration.

Up to this point in ISSU, there is no traffic impact on the compute nodes.

fab issu_contrail_migrate_compute

  1. Initiate compute nodes migration:

    Note: During this stage there will be traffic impact on the compute nodes in migration.

    • The compute nodes can be upgraded one at a time or in groups.
    • VMs can be spawned on the upgraded compute nodes.
    • The north bound communication is open, and the Contrail controller is in service.
  2. When all compute nodes are migrated, freeze the north bound communication.

fab issu_contrail_rollback_compute

  1. Rollback the compute nodes if needed. Up to this stage, if there is an issue during the upgrade process, the system administrator can use the rollback command to rollback the compute nodes.

Once a newer state is created through the newer controller, rollback is not advised.

fab issu_contrail_finalize

  1. Finalize the upgrade. Migrate the rest of the state and perform post-sync.
  2. Open the north bound communication, and point it to the newer controller.
  3. Clean up the previous nodes for api, control, analytics, and database.
  4. Finalize the new nodes of api, analytics, and database (new control nodes were already added.)

The system is now upgraded.

Details of the CLI Manual ISSU Procedure

This section provides details for each step of the CLI manual procedure for Contrail ISSU.

  1. Start a parallel setup—install the same number of Contrail controller nodes as are currently running, and that need an upgrade with newer software. Use the following guidelines:
    • The newer Contrail nodes can point to the existing database, or you can install a newer database as needed for migration.

      If the nodes point to the same database, ensure that the databases are logically separated for both the versions by configuring the following:

      • cluster_id
      • contrail-analytics-api.conf
      • contrail-api.conf
      • contrail-device-manager.conf
      • contrail-discovery.conf
      • contrail-schema.conf
      • contrail-snmp-collector.conf
      • contrail-svc-monitor.conf
      • kafka_prefix in contrail-alarm-gen.conf and contrail-collector.conf
    • OpenStack RabbitMQ should not be on the Contrail controller nodes. RabbitMQ can be external to either version of the Contrail controller nodes, and it can be shared.

      If RabbitMQ is shared, ensure the controllers are logically separated, by configuring rabbit_vhost in the following:

      • contrail-api.conf
      • contrail-device-manager.conf
      • contrail-schema.conf
      • contrail-svc-monitor.conf
    • To avoid changing OpenStack configuration files during the migration procedure, the Contrail controller should be front-ended with a load balancer, typically HAProxy.

      Additionally, if the database is shared, it should also be front-ended with a load balancer, typically HAProxy.

  2. Prepare the system for upgrade.
    • BGP peer the Contrail controllers of the existing version and the newer version:

      python /opt/contrail/utils/provision_control.py --host_name <host-name> --host_ip <host-ip> --api_server_ip <api-server-ip> --api_server_port <api-server-port> --oper add --admin_user admin --admin_password secret123 --admin_tenant_name admin --router_asn 64512 --ibgp_auto_mesh

    • On each of the control nodes, ensure that the only services running under supervisor_config are discovery, api, and ifmap services:

      openstack-config --set /etc/contrail/supervisord_config.conf include files \"/etc/contrail/supervisord_config_files/contrail-api.ini /etc/contrail/supervisord_config_files/contrail-discovery.ini /etc/contrail/supervisord_config_files/ifmap.ini\"

  3. Freeze the north bound communication of Contrail.
  4. Initiate the pre-sync.

    issu_contrail_pre_sync.py

  5. Initiate the runtime sync between old the Contrail controller and new controller.

    issu_contrail_run_sync.py

    The runtime sync operates until the ISSU procedure completes. The runtime sync is daemonized and runs under supervisord, using the following commands:

    • touch /etc/supervisor/conf.d/contrail-issu.conf
    • openstack-config --set /etc/supervisor/conf.d/contrail-issu.conf program:contrail-issu command 'contrail-issu-run-sync'
    • openstack-config --set /etc/supervisor/conf.d/contrail-issu.conf program:contrail-issu numprocs 1
    • openstack-config --set /etc/supervisor/conf.d/contrail-issu.conf program:contrail-issu process_name '%(process_num)s'
    • openstack-config --set /etc/supervisor/conf.d/contrail-issu.conf program:contrail-issu redirect_stderr true
    • openstack-config --set /etc/supervisor/conf.d/contrail-issu.conf program:contrail-issu stdout_logfile '/var/log/issu-contrail-run-sync-%(process_num)s-stdout.log'
    • openstack-config --set /etc/supervisor/conf.d/contrail-issu.conf program:contrail-issu stderr_logfile '/dev/null'
    • openstack-config --set /etc/supervisor/conf.d/contrail-issu.conf program:contrail-issu priority 440
    • openstack-config --set /etc/supervisor/conf.d/contrail-issu.conf program:contrail-issu autostart true
    • openstack-config --set /etc/supervisor/conf.d/contrail-issu.conf program:contrail-issu killasgroup false
    • openstack-config --set /etc/supervisor/conf.d/contrail-issu.conf program:contrail-issu stopsignal KILL
    • openstack-config --set /etc/supervisor/conf.d/contrail-issu.conf program:contrail-issu exitcodes 0
    • service supervisor restart
  6. Unfreeze the north bound communication.

    Up to this point in the ISSU procedure, there is no traffic impact on compute nodes.

  7. Initiate compute node migration. There will be traffic impact during this process, for the compute nodes that are in migration. Compute nodes can be upgraded one at a time or in groups.
    1. Prepare the compute nodes for the upgrade by running the following commands on each compute node:
      • openstack-config --del /etc/contrail/supervisord_vrouter_files/contrail-vrouter-agent.ini program:contrail-vrouter-agent autostart
      • openstack-config --del /etc/contrail/supervisord_vrouter_files/contrail-vrouter-agent.ini program:contrail-vrouter-agent killasgroup
    2. Upgrade the Python package on each of the compute nodes.
    3. Change the settings in the compute nodes to point to the new controller. Repeat the following commands for each compute node.
      • openstack-config --set /etc/contrail/contrail-vrouter-agent.conf DISCOVERY server <new-discovery-ip>
      • openstack-config --set /etc/contrail/supervisord_vrouter_files/contrail-vrouter-agent.ini program:contrail-vrouter-agent autostart true
      • openstack-config --set /etc/contrail/supervisord_vrouter_files/contrail-vrouter-agent.ini program:contrail-vrouter-agent killasgroup true
      • openstack-config --set /etc/contrail/contrail-vrouter-nodemgr.conf DISCOVERY server %s <new-discovery-ip>
      • service supervisor-vrouter restart
    4. VMs can be spawned on the upgraded compute nodes.
  8. When all compute nodes are migrated, freeze the north bound communication.

    Note: If the north bound communication is still open, Contrail control is in service. Once all compute nodes are migrated, freeze the north bound.

  9. Rollback the compute nodes if needed. Up to this point, if there is an issue during the upgrade process, the administrator can use the rollback process to return the compute nodes to their original state. For more information, see Rollback of Compute Nodes .
  10. Finalize the upgrade. Migrate the rest of the state by using the following commands.
    1. Stop the Contrail services on the older version.
      Service supervisor-config stop

Service supervisor-analytics stop

Service supervisor-control stop

Service supervisor-webui stop
    2. Perform post-sync.
      rm -f /etc/supervisor/conf.d/contrail-issu.conf

service supervisor restart

contrail-issu-post-sync

contrail-issu-zk-sync
    3. Update the supervisor-config files.
      • openstack-config --set /etc/contrail/supervisord_config.conf include files \"/etc/contrail/supervisord_config_files/*.ini\"
      • service supervisor-config restart
    4. Run contrail-status to verify all of the services are up and running.
  11. Open the north bound communication and change it to point to the new controller. If you are using HAProxy, change the backend to point to the new controller instead of the old controller.
  12. Clean up the previous configuration of the api, control, analytics, and database nodes by running the following commands on each configured node.
    • python provision_config_node.py --api_server_ip <new-api-server-ip> --host_name <old-api-server-name> --host_ip <old-api-server-ip> --oper del --admin_user admin --admin_password secret123 --admin_tenant_name admin
    • python provision_analytics_node.py --api_server_ip <new-api-server-ip> --host_name <old-analytics-server-name> --host_ip <old-analytics-ip> --oper del --admin_user admin --admin_password secret123 --admin_tenant_name admin
    • python provision_control.py --api_server_ip <new-api-server-ip> --api_server_port 8082 --router_asn 64512 --admin_user admin --admin_password secret123 --admin_tenant_name admin --host_name <old-control-node-name> --host_ip <old-control-ip> --oper del
    • python provision_database_node.py --api_server_ip 10.87.64.53 --host_name <old-database-host-name> --host_ip <old-database-ip> --oper del --admin_user admin --admin_password secret123 --admin_tenant_name admin
  13. Finalize the new installation. Add new api, analytics, and database configurations by running the following commands on each configured node.

    Note: New control nodes were already added at the beginning of the ISSU procedure.

    • python provision_config_node.py --api_server_ip <new-api-server-ip> --host_name <new-api-server-host-name> --host_ip <new-api-server-ip> --oper add --admin_user admin --admin_password secret123 --admin_tenant_name admin
    • python provision_analytics_node.py --api_server_ip <new-api-server-ip> --host_name <new-analytics-server-host-name> --host_ip <new-analytics-server-ip> --oper add --admin_user admin --admin_password secret123 --admin_tenant_name admin
    • python provision_database_node.py --api_server_ip <new-api-server-ip> --host_name <new-database-host-name> --host_ip <new-database-ip> --oper add --admin_user admin --admin_password secret123 --admin_tenant_name admin

Rollback of Compute Nodes

If a failure is encountered upon upgrading the compute nodes, the system administrator can issue the rollback command, and all of the upgraded compute nodes can be rolled back to their original configuration.

Note: Rollback of compute nodes is not recommended during or after the step “Finalize the upgrade” (migrate the rest of the state). Doing so could cause loss of data.

  1. Prepare the compute nodes for the rollback by issuing the following commands on each of the compute nodes.
    • openstack-config --del /etc/contrail/supervisord_vrouter_files/contrail-vrouter-agent.ini program:contrail-vrouter-agent autostart
    • openstack-config --del /etc/contrail/supervisord_vrouter_files/contrail-vrouter-agent.ini program:contrail-vrouter-agent killasgroup
  2. Downgrade the Python package on each of the compute nodes.
  3. Once the compute node is downgraded, change the settings in the compute node to point to the old controller.
    • openstack-config --set /etc/contrail/contrail-vrouter-agent.conf DISCOVERY server <old-discovery-ip>
    • openstack-config --set /etc/contrail/supervisord_vrouter_files/contrail-vrouter-agent.ini program:contrail-vrouter-agent autostart true
    • openstack-config --set /etc/contrail/supervisord_vrouter_files/contrail-vrouter-agent.ini program:contrail-vrouter-agent killasgroup true
    • openstack-config --set /etc/contrail/contrail-vrouter-nodemgr.conf DISCOVERY server %s <old-discovery-ip>
    • service supervisor-vrouter restart
  4. Repeat this procedure for each of the upgraded compute nodes.
 

Related Documentation

 

Modified: 2016-12-09

Download This Guide

 

Related Documentation

 

Modified: 2016-12-09

Previous Page Next Page