Contrail Cloud Configuration

Hardware Profiles

A hardware profile allows administrators to apply the same configuration to a group of servers acting as compute or storage nodes.

As new servers are added to a deployment, each server might have different disk and network hardware. The method that networks and storage are configured may differ between servers.

Servers are associated to hardware profiles in the compute-nodes.yml file. The leaf number is also set in this file. This sample configuration snippet from the compute-nodes.yml file shows a hardware profile configuration:

The hardware profile configurations are applied to servers using the site.yml and overcloud-nics.yml files, using the sections of the files that use the following semantic:

where:

• The role is one of the following values: ComputeKernel, ComputeDPDK, ComputeSriov, CephStorage

• The leaf number is the number of the leaf. For instance, 0.

• The Hardware profile tag is any alphanumeric string starting with a capital letter. We strongly recommend using the Hw[number] convention.

For example, the following sample names could be used for compute nodes of different types in leaf 0:

• ComputeKernel0Hw1 for kernel-mode

• ComputeDpdk0Hw1 for DPDK

• ComputeSriov0Hw1 for SR-IOV

Two sample hardware profile configurations within the site.yml file are provided below. The example allocates different SCSI disks for local VM ephemeral storage using variable HCTL.

Example of hardware profiles in the overcloud-nics.yml file are provided later in this reference architecture.

IPMI Network

The IPMI network is generally a set of network addresses reserved for hardware management within a data center. The management switches must be configured with access to the default gateway for the IPMI network. Servers in the environment must have IP addresses statically allocated for IPMI or sent via DHCP using the MAC address as the key for address allocation.

Provision Network

The properties of the provision network are configured in the undercloud: section of the site.yml file.

A sample configuration snippet from the site.yml file:

A server PXE boots from the provisioning network and receives an IP address via DHCP when it is provisioned. The boot preference is then changed to disk, the IP address is configured by the Ironic service into the operating system, and the server boots from the disk with the same IP address.

External Network

The External network is used for cloud users to access the public API addresses of the control hosts. A VIP IP address as well as a pool of IP addresses that can be used by DHCP is specified.

The External network parameters are set in the site.yml file.

A sample External network configuration snippet from the site.yml file:

There are only a limited number of control hosts. The external network, therefore, can be a small subnet of IP addresses.

The External network is associated with an interface on control hosts in the control-host-nodes.yml files.

Management

The properties of the management network are configured in the network: section of the site.yml file.

A sample Management network configuration snippet from the site.yml file:

Internal API, Tenant, Storage, and Storage Management Networks

Red Hat Openstack 13 (RHOSP 13) supports the capability to use separate subnets per rack. This feature is used for the networks that are connected to the IP Fabric in this reference architecture. These are the internal API, tenant, storage and storage management networks. Each of these networks is assigned a supernet IP address range (/16), which includes all the rack subnets (/24) for that network.

The concept of spine-leaf networking in TripleO is described in

Red Hat OpenStack Platform 13: Spine Leaf Networking

TripleO uses the term leaf to group servers that have shared connectivity. In this Contrail Cloud reference architecture, leaves in the TripleO context refer to grouped servers in the same rack. In this reference architecture, a Red Hat leaf is implemented by a management switch and a pair of top of rack switches (ToR switches).

The names of networks in configuration files follow the convention established in the compute and storage nodes for a rack, or a leaf. The names are a concatenation of base network name and leaf number. Leaf numbers are assigned to nodes in the compute_nodes.yml file. Base network names include InternalApi, Storage, StorageMgmt, and Tenant networks.

Compute nodes in Leaf 0 should use networks:

• InternalApi0

• Storage0

• Tenant0

Compute nodes in Leaf 1 should have defined networks:

• InternalApi1

• Storage1

• Tenant1

Compute nodes in Leaf N should have defined networks:

• InternalApiN

• StorageN

• TenantN

For an example of splitting a supernet, see the example below for the Internal API and Tenant networks. The same procedure must be performed for the remaining networks used for compute and storage nodes (Storage, Storage Mgmt).

In the following configuration snippet, these parameters are set for all networks:

• supernet - supernet definition.

• cidr - subnet configuration for a leaf with the first subnet used for the controllers.

• default_route - static route definition pointing to a “supernet” via a given operating system interface, such as bond1, vhost0, or other interfaces.

• vrouter_gateway - default route definition for vrouter encapsulated traffic in the overlay network. This variable is defined as a gateway parameter in the /etc/contrail/contrail-vrouter-agent.conf file. This gateway IP address is used to reach DC gateways (MX routers and other vRouters to setup MPLSoUDP or other overlays).

• role - the role or hardware profile that assigns the subnet. The first subnet is always for controllers and the remaining subnets are assigned to compute and storage nodes based on the leaf identifier.

When a network is specified to share an interface in the role_network_config.yml file, the network is assigned a VLAN. The VLAN can be identical in all racks when VXLAN is used, which is how the VLANs are labelled in this reference architecture.

Jumphost Overview

The jumphost:

• hosts the undercloud. The undercloud is a VM responsible for provisioning and managing all control hosts, storage nodes, and compute nodes in a Contrail Cloud. All Contrail-related setup and configuration is performed through the undercloud in a Contrail Cloud.

• stores Contrail Cloud configuration-related files. The YAML files that configure Contrail Cloud are stored on the jumphost. The Ansible scripts that apply the configurations made in the YAML files to the Contrail Cloud nodes are also stored on the jumphost.

  The Contrail Cloud scripts are stored in the /var/lib/contrail_cloud directory.

• hosts the Contrail Command web user interface virtual machine.

• runs Red Hat Enterprise Linux with only base packages installed.

• provides SNAT for traffic from the Provisioning network to intranet and external networks.

• provides access to the servers of a Contrail Cloud environment if a management network is not present

A jumphost must be operational as a prerequisite for a Contrail Cloud installation. The jumphost should not run any virtual machines besides the undercloud and Contrail Command VMs.

Figure 1 illustrates the jumphost network connections.

Figure 1: Jumphost Network Connections

The Intranet network is the network configured manually before the Contrail Cloud 13 packages are downloaded. This network is used by Contrail Cloud to download packages and to provide outside connectivity for the nodes in a Contrail Cloud environment to reach outside networks. The Intranet network IP address can be the IP address from the External network.

The jumphost is configured during the installation process to use an IP masquerade for SNAT of outbound traffic from hosts connected to the provisioning network. The “public” IP address on the jumphost—the IP address of the “Intranet” port—is used as a source address for traffic exiting the cloud during provisioning. This IP address should be permitted in firewalls to allow access to public repositories, Juniper Satellite servers, and Red Hat subscription managers. If external access is not permitted from the jumphost for security purposes, a Capsule proxy server can be used as described in Capsule Configuration. See Miscellaneous.

You can also configure a proxy, in which case the “Intranet” port should have access to the proxy. Note that the externally accessible VIP addresses for Openstack, Appformix, and Contrail APIs should be excluded from proxying (to avoid issues during provisioning).

A sample proxy configuration snippet from the site.yml file:

Where 10.10.100.50, 10.10.10.51, 10.10.10.52 are VIPs in the External network.

Adding an IP Address for the Jumphost to the Provision Network

The jumphost should be allocated an IP address in the Provision network to enable SSH to the other hosts in the Contrail Cloud environment. This IP address allocation is a convenience to enable troubleshooting from the jumphost.

This IP address is added to the jumphost in the site.yml file as shown in this configuration snippet:

Contrail Command Configuration

Contrail Command is a standalone solution based on two containers: contrail_command and ccontrail_psql. Contrail Command has no HA capabilities and only new UI services are provided. The Contrail Command VM is created on the jumphost.

The Contrail Command web UI can be reached in a Contrail Cloud environment by entering this URL in a web browser: https://[jumphost-IP-address]:9091

Contrail Command can be accessed after a Contrail Cloud deployment without user configuration. Contrail Cloud configuration parameters are updated in the site.yml file.

A sample site.yml file configuration snippet where the Contrail Command parameters are updated:

Authentication details for Contrail Command are provided in the vault-data.yml file.

A sample vault-data.yml file configuration snippet with modified Contrail Command attributes:

Control Node VM Resources

This reference architecture is designed to support a large-scale Contrail Cloud environment. The following memory resources should be allocated to controller VMs to support this architecture.

* The Openstack Controller VM size is significantly smaller than Red Hat’s recommended Openstack Controller VM size. The VM uses less resources in Contrail Cloud because several network functions are handled by Contrail Networking and telemetry functions are performed by Appformix. See Red Hat OpenStack Platform 13: Recommendations for Large Deployments to see the recommended VM sizes.

The following configuration snippet from the control_hosts:vm: section of the site.yml files configures control host options:

Controller Host Network Configuration

Corresponding interfaces of controller hosts are in the same L2 network, and are preferably deployed in different racks and connected to leaf devices in the EVPN-VXLAN IP Fabric. Figure 2 illustrates these connections.

Figure 2: Control Host Network Connections

The following VMs are running on the control hosts.

• OS - Openstack Controller

• CC - Contrail Controller

• CA - Contrail Analytics

• CADB - Contrail Analytics DB

• AFX - Appformix

• TSN - ToR Service Node (optional)

  A TSN is enabled in the control_hosts: vm: contrail-tsn: hierarchy within the site.yml file. The function has been renamed CSN (Contrail Service Node) in Contrail Networking but tripleO manifests continue to use the TSN term.

Figure 3 illustrates how the physical and logical interfaces on a control host connect to it’s management switch and leaf switches.

Figure 3: Control Host—Physical and Logical Interfaces

Figure 4 illustrates how networks and ports are assigned to bridges on a controller node.

Figure 4: Control Host—Physical Interface to Bridge Allocation

Physical NICs typically contain two ports which are named nic3 and nic4 in the first physical NIC.

Table 4 shows which files are used for configuring each network.

The configuration for bond interfaces is performed in the control_host_nodes_network_config: hierarchy of the control-hosts-nodes.yml file. Bond interfaces should be configured with the following parameters:

• Linux bond - mode 4/LACP (802.3ad)

• Hash policy - layer3+4

• Use ovs-bridge and linux_bond. An OVS-bridge Linux bridge is configurable, but is not recommended by Red Hat

This configuration snippet from the control-hosts.nodes.yml file shows various control host configurations, including bond interface configurations.

Mapping Controller VM Interfaces to Host Bridges

The control host VM interface connections to the bridges configured in Controller Host Network Configuration is done in the control_hosts: hierarchy of the control-host-nodes.yml file.

A sample configuration snippet:

The first interface—eth0—must connect to the bridge for the provision network to allow the VM to PXE boot. The other interface names must be sequential, which matches the sample configuration snippet. You should configure one interface for each bridge.

Appformix VM Configuration

AppFormix is provided with the Contrail Cloud bundle. AppFormix provides monitoring and troubleshooting for the networking and server infrastructure of Contrail Cloud. Appformix provides the same services for the workloads running in Contrail Cloud. For additional information on Appformix, see the Appformix TechLibrary page.

Appformix provides a WebUI as well as a REST API. The WebUI and the REST API are exposed to the Internal API and External networks.

The recommended Appformix deployment—which is also the default deployment—is deployed in a 3-node configuration for high availability.

Appformix node configuration for Contrail Cloud is defined in the site.yml file. A sample configuration snippet:

The network configuration of Appformix is defined in the overcloud-nics.yml file.

A sample configuration snippet:

The following resources are automatically monitored when AppFormix is installed in Contrail Cloud.

• Openstack API endpoints and processes

• Contrail API endpoint and processes

• Openstack MySQL

• Rabbit cluster status

• Compute nodes including vRouter, Nova compute, operating system health and metrics

Appformix is not automatically configured to monitor the physical networking infrastructure, but adapters (without any configuration) for monitoring network devices can be installed during a Contrail Cloud deployment. AppFormix must be configured manually to monitor network devices after deployment. See the Network Devices section of the Appformix User Guide.

Network-related adapters can be installed for Appformix during a Contrail Cloud deployment in the site.yml file.

A sample configuration snippet:

Custom Appformix plugins can also be installed during a Contrail Cloud deploying in the site.yml file.

A sample configuration snippet:

For more information, see Extensibility Using Plug-Ins in the Appformix User Guide.

Contrail Service Node (CSN) - Optional

The Contrail Service Node (CSN) provides DHCP, ARP, and multicast services when Contrail is managing the full lifecycle of bare-metal servers, including provisioning the OS. CSNs are not needed in Contrail Cloud deployments that aren’t using bare-metal servers.

The triple-O templates for Contrail Cloud 13 release continue to use the Red Hat term ToR Service Node(TSN) to refer to the CSN function in Contrail Cloud. This reference architecture, therefore, uses both terms.

To enable CSN support in Contrail Cloud, edit the compute_hosts: hierarchy in the site.yml file.

A sample configuration snippet:

TSN VMs are created on all controller hosts by default. We recommend running TSNs on two control hosts in Contrail Cloud environments that include at least one bare-metal server (BMS). TSN VMs should not be run in Contrail Cloud environments that are not using a BMS. You can change the number of TSN instances in the control_hosts: hierarchy in the site.yml file.

Compute Node Configuration

Compute nodes in Contrail Cloud are installed in racks and connected to a pair of top-of-rack (ToR) switches and a management switch. The ToR switches are the leaf nodes in the EVPN-VXLAN IP Fabric.

The networking for compute nodes is configured to place compute nodes in racks that compose separate Layer 3 subnets. Each rack is it’s own separate layer 3 for the networks that are used by tenant workloads. Figure 5 illustrates this compute node networking structure.

Figure 5: Compute Node Network Connections

Figure 6 illustrates how the physical and logical interfaces of a compute node connect it to a management switch and the IP Fabric leaf switches.

Figure 6: Compute Node Network Interfaces

Figure 7 illustrates networks and port assignments to bridges on a compute node.

Figure 7: Compute Node to Bridge Connections

Table 5 shows which files are used for configuring each network connection on a compute node.

Provisioning and optional Management networks are connected via out-of-band management switches and must be configured as a Layer 2 stretch across the management switches.

Compute Node Networking

The data plane interfaces of compute nodes can be configured to support the following forwarding methods.

Your traffic forwarding method choice depends on the traffic profile expectations for each individual compute node. A Contrail Cloud environment can have different compute nodes configured with different interface types, and workloads can be placed on the most appropriate compute node using various technologies, such as OpenStack availability zones.

The IP address of the management interface and the addressing and configuration of bond interfaces are configured in the overcloud-nics.yml file.

Compute Resources

The following guidelines should be following to optimize vRouter performance.

• Guarantee minimum 4 cores for host operating system and minimum 2 of them can be used by vRouter kernel module. There is no mechanism to allocate cores but we assuming that Host OS processes will consume no more than 2 cores and remaining 2 a kernel scheduler will allocate for vRouter.

• Guarantee minimum 8GB RAM for host operating system where 4GB will be used for vRouter.

vRouters running in kernel mode should achieve up to 500kpps per vRouter (<1000 flows in a table) when these guidelines are followed.

Bond Interface Configuration

The following options should be set for bond interfaces when kernel mode is used:

• bond_mode: 4 (IEEE 802.3ad)

• bond_policy: layer3+4

The bond configuration is defined in profile definitions in the overcloud-nics.yml file:

Optimizations

We recommend increasing the maximum number of flows per vRouter from the default value of 500,000 to 2 million flows to increase performance and scaling. We also recommend allocating 4 threads for flow processing.

These configuration parameters are specified in the site.yml file. A configuration snippet:

Adding Kernel-Mode Compute Nodes

A compute node operates in kernel mode when it is added into the compute_nodes_kernel: hierarchy in the compute-nodes.yml file.

A sample configuration snippet:

DPDK Bond Interface Configuration

LACP bonding is under DPDK control when a vRouter is in DPDK mode. There is no Linux bond.

LACP bonding is defined in the overcloud-nics.yml file and configured using these options:

• bond_mode: 4 (IEEE 802.3ad)

• bond_policy: layer3+4

A sample overcloud-nics.yml file for the bond configuration:

The LACP rate with DPDK is taken during the LACP negotiation process between switches acting as LACP partners and applied to what is configured on the switches. Most Juniper switches are configured in fast LACP mode by default, and the vRouter applies that setting.

If you want to force LACP into fast LACP mode, set the LACP_RATE: field to 1 in the site.yml file:

Performance Tuning for DPDK

To maximize throughput for a vRouter in DPDK mode, set the following parameters:

• Allocate 4 threads for flow processing in the vRouter agent.

• Allocate huge pages.

• Allocate CPUs for DPDK, Host OS, and Nova.

• Increase the flow table size to 2 million from the 500,000 default setting.

• Increase the buffer sizes to 2048 to reduce packet drops from microbursts.

• Double the vrouter memory pool size to 131072.

• Set the CPU scaling governor into performance mode.

Enabling CPU performance mode

CPU frequency scaling enables the operating system to scale the CPU frequency to save power. CPU frequencies can be scaled automatically depending on the system load, in response to ACPI events, or manually by user space programs. To run the CPU at the maximum frequency, set the scaling_governor parameter to performance.

In the BIOS all power saving options should be disabled, including power performance tuning, CPU P-State, CPU C3 Report and CPU C6 Report. Select Performance as the CPU Power and Performance policy.

The configuration can be defined in the site.yml file as an extra_action (post deployment action) parameter.

A sample configuration snippet from the site.yml file:

Configuring Flow Threads and Huge Pages

Memory that does not need to be used by the operating system should be segmented into huge pages to maximize efficiency. For instance, suppose a server with 256GB RAM needs 8GB for the OS (including the vRouter) and therefore has 248GB remaining for other functions. This server should allocate the remaining memory into a 1GB huge page to maximize memory usage.

Additionally, some 2MB huge pages should be configured to maximize memory usage for the vRouter.

The number of threads used for flow processing and huge page allocations are configured in the site.yml file.

A sample configuration snippet from the site.yml file:

Ceph Configuration

Ceph storage configuration is defined in the site.yml file.

Disk mapping for Ceph Storage nodes need to be defined on overcloud: disk_mapping: hierarchy for each storage node. The recommended disk settings are 4 OSD disks + 1 SSD journal disk. This configuration can be scaled appropriately; for instance, 8 OSD disks + 2 SSD journal disks for a larger environment. CPU, RAM, and storage traffic needs to be considered when using multiple 4 OSDs+1 journal bundles.

The mapping section of the site.yml file allows disks to be referenced by name rather than by device id.

A sample site.yml configuration file snippet to configure disk mappings:

This sample site.yml configuration file snippet configures Ceph storage for a 4 OSD disks + 1 SSD journal bundle:

The disk layouts, notably, are per hardware profile.

Ceph service configuration hides the complexity of computing placement groups, by default. This configuration can be manually provided in the site.yml file if required, such as in cases where the environment has a small number of OSDs.

Storage Node Network Configuration

In this reference architecture, separate bonds are used on storage nodes for the Storage Mgmt (replication) and Storage (user access) networks.

Figure 12 illustrates these connections.

Figure 12: Storage Node Networks

Figure 13 illustrates how the physical and logical interfaces of a storage node connect to a management switch and leaf switches.

Figure 13: Storage Node—Physical and Logical Interfaces

Figure 14 illustrates how the physical interfaces of a storage host connect to the bridges configured on it.

Figure 14: Storage Node—Interfaces

Table 7 shows which files are used for configuring each network connection on a storage node.

The IP address of the management interface and addressing and configuration of bond interfaces on a storage node are configured according to the configuration files that are used by the Contrail Cloud provisioning system. The ports on each storage host are configured as follows.

Linux bond configuration is defined per storage host in the overcloud-nics.yml file,

A sample configuration snippet from the overcloud-nics.yml file:

Where:

• NIC-0 - onboard copper NIC 1/10G

• NIC-1 and NIC-2 - 2 ports 10G/25G/40G Intel Fortville family NICs in PCI slot connected to NUMA0