Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

external-header-nav
keyboard_arrow_up
list Table of Contents
file_download PDF
keyboard_arrow_right

Contrail Insights Installation for OpenStack Helm

date_range 30-May-22

OpenStack Helm is a project that provides Helm charts for deploying OpenStack services on a Kubernetes cluster. Contrail Insights can be deployed to monitor both the Kubernetes resources as well as the OpenStack resources from a single Dashboard.

Architecture and Terminology

Kubernetes cluster nodes

Primary and worker nodes of the Kubernetes cluster being monitored by Contrail Insights. These nodes will run the Contrail Insights Agent.

OpenStack cluster nodes

Kubernetes nodes from the cluster above that are running OpenStack services (Keystone, Nova, Neutron, and so on) in containers.

Contrail Insights Platform node

Node on which Contrail Insights Platform components will be installed. Should be able to reach the Kubernetes cluster nodes.

Figure 1: Contrail Insights and OpenStack Helm WorkflowContrail Insights and OpenStack Helm Workflow

Requirements

The following are the requirements for installing Contrail Insights for OpenStack Helm.

  • Supported Kubernetes versions: 1.8.x and later

  • See Contrail Insights General Requirements for hardware and software requirements.

  • API access to Kubernetes API server. Contrail Insights reads information about the Kubernetes cluster from the API server. The token provided during configuration must provide sufficient permission for read-only API calls. In addition, Contrail Insights Platform must be able to open a connection to the host and port on which the API server runs.

  • API access to OpenStack services. Contrail Insights reads information about the OpenStack cluster through the APIs exposed by the various OpenStack services. The user credentials provided during configuration must provide sufficient permission for read-only API calls. In addition, Contrail Insights Platform must be able to open a connection to the host and port on which the API server runs.

  • Note:

    Upgrade notice: Starting with Contrail Insights 3.2.6, the requirement for a license file is removed. If you are installing a version earlier than 3.2.6, a license is required prior to installation.

    You can obtain a license key from mailto:APPFORMIX-KEY-REQUEST@juniper.net. Provide the following information in your request:

    content_copy zoom_out_map
    Group name:
    Target customers or use:
    Cluster type: OpenStack on Kubernetes
    Number of hosts:
    Number of instances:
    

Workflow

The installation consists of the following steps:

  1. Initial setup.

  2. Prerequisites for Agent installation.

  3. Configuring Kubernetes.

  4. Configuring OpenStack.

  5. Installing Contrail Insights.

  6. Optional configuration.

Initial Setup

Run the following commands for initial setup:

  1. Install these required packages on the Contrail Insights Platform node.

    content_copy zoom_out_map
    #Ubuntu
    apt-get update
    apt-get install python-pip python-dev build-essential libssl-dev libffi-dev
    pip install ansible==2.3.0.0 markupsafe httplib2
    
    content_copy zoom_out_map
    #RHEL/CentOS
    yum install epel-release                                #Enable EPEL repository
    In case the above command does not work, manually download and install the epel-release 
    package with one of the below commands, depending on your system’s version.
    yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
    yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-6.noarch.rpm
    
    yum groupinstall 'Development Tools'                    #Install development tools
    yum install openssl-devel libffi libffi-devel           #Dependencies
    yum install python-pip python-devel                     #Install Pip
    pip install ansible==2.3.0                              #Install Ansible 2.3
    pip install markupsafe httplib2                         #Dependencies
    
    Note:

    For RHEL, the following iptables rule is needed to access port 9000.

    content_copy zoom_out_map
    iptables -t filter -A IN_public_allow -p tcp --dport 9000 -j ACCEPT
  2. Edit the /etc/hosts file on the Contrail Insights Platform node and enter the IP addresses of the OpenStack on Kubernetes cluster nodes.

    content_copy zoom_out_map
    vi /etc/hosts
    <kube-master-ip> k8s-master 
    <kube-worker1-ip> k8s-node1 
    <kube-worker2-ip> k8s-node2 
  3. Set up passwordless SSH between the Contrail Insights Platform node and the OpenStack on Kubernetes cluster nodes. Run the following commands to generate and copy the SSH public keys to all the nodes.

    content_copy zoom_out_map
    ssh-keygen -t rsa
    ssh-copy-id -i ~/.ssh/id_rsa.pub root@k8s-master
    ssh-copy-id -i ~/.ssh/id_rsa.pub root@k8s-node1
    ssh-copy-id -i ~/.ssh/id_rsa.pub root@k8s-node2
    ssh-copy-id -i ~/.ssh/id_rsa.pub root@<IP of Contrail Insights Platform node>

Prerequisites for Agent Installation

Contrail Insights uses the libvirt library to collect data from virtual machines running on an OpenStack compute node. Run the following command on all OpenStack computes to install the library:

content_copy zoom_out_map
On Ubuntu hosts:
sudo apt-get install libvirt0:amd64
On Centos hosts:
sudo yum install libvirt-client

Configure Kubernetes

Contrail Insights reads information about resources in your Kubernetes clusters. The software requires the cluster-admin role or another role that gives it read-only access to all objects in the cluster. We recommend that you create a new Service Account for Contrail Insights and assign it the cluster-admin role. If you do not create a new Service Account, then you must provide the token from an existing Service Account that has the required access during the configuration of Contrail Insights.

To create a new Service Account with the required access for Contrail Insights, perform the following steps in the OpenStack on Kubernetes cluster primary node:

  1. Create a YAML file with the following:

    content_copy zoom_out_map
    apiVersion: v1
    kind: ServiceAccount
    metadata:
    name: appformix
    
  2. Create the appformix Service Account using the file created in Step 1.

    content_copy zoom_out_map
    $ kubectl create -f <file>.yaml
  3. Confirm that the Service Account has been created. Make a note of its namespace.

    content_copy zoom_out_map
    $ kubectl describe serviceaccount appformix
    Name:         appformix
    Namespace:    default
    Labels:       <none>
    Annotations:  <none>
    Image pull secrets:  <none>
    Mountable secrets:   appformix-token-pkljk
    Tokens:              appformix-token-pkljk
    Events:       <none>
    
  4. Add the cluster-admin role to the appformix Service Account as follows, substituting namespace for the namespace noted in Step 3.

    content_copy zoom_out_map
    $ kubectl create clusterrolebinding appformix-binding --clusterrole=cluster-admin --serviceaccount=<namespace>:appformix
  5. Run the following command to confirm that the appformix Service Account has the required access:

    content_copy zoom_out_map
    $ kubectl auth can-i get nodes --as=system:serviceaccount:<namespace>:appformix --all-namespaces

    The output of the command should be yes.

  6. Contrail Insights must be configured to communicate with the Kubernetes cluster. Get the following details from the Kubernetes cluster to use during the Contrail Insights installation.

    kubernetes_cluster_url

    This is the URL of the Kubernetes API Server. To get this value, run the following command on the OpenStack on Kubernetes cluster:

    content_copy zoom_out_map
    $ kubectl cluster-info | grep 'Kubernetes master'
    Kubernetes master is running at https://172.24.1.173:6443
    
    kubernetes_auth_token

    This is the authentication token of the appformix Service Account. To get this value, run the following commands on the OpenStack on Kubernetes cluster:

    content_copy zoom_out_map
    $ kubectl describe serviceaccount appformix
    Name:         appformix
    Namespace:    default
    Labels:       <none>
    Annotations:  <none>
    Tokens:       appformix-token-pkljk
    [...]
    $ kubectl describe secret appformix-token-pkljk
    Name:         appformix-token-pkljk
    Namespace:    default
    [...]
    token:      eyJwb[...]
    

Configure OpenStack Administrator Account for Contrail Insights

Contrail Insights reads information about all projects configured in your OpenStack cluster. The software requires administrator privilege. We recommend you create a new administrator account for Contrail Insights. If you do not create a new administrator account, then you must provide the username and password of an existing administrator account during the configuration of Contrail Insights.

To create an administrator account for Contrail Insights, perform the following steps in the OpenStack Horizon dashboard:

  1. Create a user account and name it appformix.

  2. Select a new project for the user account.

  3. Select role as admin.

Install Contrail Insights

To install Contrail Insights:

  1. Download the Contrail Insights installation packages from software downloads to the Contrail Insights Platform node. Get the following files:

    content_copy zoom_out_map
    contrail-insights-<version>.tar.gz
    contrail-insights-dependencies-images-<version>.tar.gz
    contrail-insights-kubernetes-images-<version>.tar.gz
    contrail-insights-openstack-images-<version>.tar.gz
    contrail-insights-platform-images-<version>.tar.gz
    

    If you are installing a version earlier than 3.2.6, copy the Contrail Insights license file to the Contrail Insights Platform node.

  2. Unzip contrail-insights-<version>.tar.gz. This package contains all the Ansible playbooks required to install Contrail Insights.

    content_copy zoom_out_map
    tar -xvzf contrail-insights-<version>.tar.gz
    cd contrail-insights-<version>/
    
    Note:

    The remaining steps should be executed from within the contrail-insights-<version>/ directory. Although the product name changed from "AppFormix" to "Contrail Insights," the UI and internal command paths continue to show AppFormix and will reflect the new name at a later date.

  3. Using sample_inventory as a template, create an inventory file for the installation. List the Kubernetes cluster nodes in the compute section and the Contrail Insights Platform node in the appformix_controller section. List the node running the OpenStack controller services in the openstack_controller section.

    content_copy zoom_out_map
    cp sample_inventory inventory
    vi inventory
    [compute]
    k8s-master
    k8s-node1
    k8s-node2
    [appformix_controller]
    <IP of the Contrail Insights Platform node>
    [openstack_controller]
    <IP of the Kubernetes node(s) running OpenStack controller services>
    
  4. Create a directory called group_vars. Create a file named all inside this directory with configuration variables required by Contrail Insights.

    content_copy zoom_out_map
    mkdir group_vars
    vi group_vars/all
    appformix_docker_images: 
    - /path/to/contrail-insights-platform-images-<version>.tar.gz
    - /path/to/contrail-insights-dependencies-images-<version>.tar.gz
    - /path/to/contrail-insights-kubernetes-images-<version>.tar.gz 
    - /path/to/contrail-insights-openstack-images-<version>.tar.gz
    appformix_dns_version: 2
    kubernetes_platform_enabled: True 
    openstack_platform_enabled: True 
    kubernetes_cluster_url: <URL from Configuring Kubernetes step 4 above>
    kubernetes_auth_token: <token from Configuring Kubernetes step 4 above>
    

    If you are installing a version earlier than 3.2.6, include the path to the Contrail Insights license file in group_vars/all:

    content_copy zoom_out_map
    appformix_license: path/to/<contrail-insights-license-file>.sig
  5. Source the openrc file from the OpenStack Controller node in the environment in which the Ansible playbooks are going to be executed. The file should look like the following:

    content_copy zoom_out_map
    $ cat openrc
    export OS_CLOUD=openstack_helm
    export OS_USERNAME='admin'
    export OS_PASSWORD='password'
    export OS_PROJECT_NAME='admin'
    export OS_PROJECT_DOMAIN_NAME='default'
    export OS_USER_DOMAIN_NAME='default'
    export OS_AUTH_URL='http://keystone.openstack.svc.cluster.local/v3'
    $ source openrc
    
  6. Run the Ansible playbook.

    content_copy zoom_out_map
    ansible-playbook -i inventory appformix_openstack_on_kubernetes.yml

    Playbook should run to completion without any errors.

  7. Log into the Contrail Insights Dashboard at:

    content_copy zoom_out_map
    http://<contrail-insights-platform-node-ip>:9000

    In the Auth Service dropdown list, three options are provided:

    OpenStack

    Log in with OpenStack credentials and view only OpenStack resources.

    Kubernetes

    Log in with Contrail Insights credentials and view only Kubernetes resources.

    AppFormix

    Log in with Contrail Insights credentials and view both OpenStack and Kubernetes resources.

    For the last two options, the token to use for login is available at /opt/appformix/etc/appformix_token.rst on the Contrail Insights Platform node.

Optional Configuration

To install Contrail Insights certified plug-ins on the cluster, include the following variables in the group_vars/all file:

content_copy zoom_out_map
appformix_plugins: <list of certified plugins to be installed>
appformix_openstack_log_plugins: <list of OpenStack log plugins to be installed>

For example:

content_copy zoom_out_map
appformix_plugins:
  - { plugin_info: 'certified_plugins/cassandra_node_usage.json' }
  - { plugin_info: 'certified_plugins/contrail_vrouter.json' }
  - { plugin_info: 'certified_plugins/zookeeper_usage.json' }
  - { plugin_info: 'certified_plugins/heavy_hitters.json' }
appformix_openstack_log_plugins:
  - { plugin_info: 'certified_plugins/cinder_api_logparser.json',
      log_file_path: '/var/log/cinder/cinder-api.log' }
  - { plugin_info: 'certified_plugins/glance_logparser.json',
      log_file_path: '/var/log/glance/glance-api.log' }
  - { plugin_info: 'certified_plugins/keystone_logparser.json',
      log_file_path: '/var/log/apache2/keystone_access.log,/var/log/httpd/keystone_wsgi_admin_access.log,/var/log/keystone/keystone.log' }

For a list of all Contrail Insights certified plug-ins that can be installed, look for the entries starting with plugin_info in the file roles/appformix_defaults/defaults/main.yml.

The OpenStack log parser plug-ins parse the API log files of each OpenStack service to collect metrics about API calls and response status codes. To install these plug-ins, add them to the variable appformix_openstack_log_plugins in group_vars/all, as shown above. Each plug-in entry in this list requires a parameter called log_file_path to be specified. This parameter should be set to the complete path to the service's API log file on the OpenStack Controller node(s). Multiple comma-separated paths can be specified.

To identify the correct log file to be specified in log_file_path, look for entries like the following, containing a client IP address, REST call type, and response status code:

content_copy zoom_out_map
2019-04-02 06:50:13.103 3465 INFO nova.osapi_compute.wsgi.server [req-d07e953a-6921-4224-a056-afb6ff69adde 953ea56a96b944b3b170a299af9e87bd 10c9e8809feb4bd1b55955d9c2ed5aba - - -] 172.18.0.6 "GET /v2/10c9e8809feb4bd1b55955d9c2ed5aba/os-hypervisors/detail HTTP/1.1" status: 200 len: 1427 time: 0.0208740
2019-04-02 06:50:13.183 3465 INFO nova.osapi_compute.wsgi.server [req-34b2f686-9eb5-4112-b3fc-e0b37798a302 953ea56a96b944b3b170a299af9e87bd 10c9e8809feb4bd1b55955d9c2ed5aba - - -] 172.18.0.6 "GET /v2/10c9e8809feb4bd1b55955d9c2ed5aba/servers/detail?all_tenants=1&status=SHELVED_OFFLOADED HTTP/1.1" status: 200 len: 211 time: 0.0754580

Default locations for these files are listed in the variable appformix_openstack_log_factory_plugins in roles/appformix_defaults/defaults/main.yml.

On containerized OpenStack environments, log files are generated inside the containers running the OpenStack services. However, they have to be available on the OpenStack controller host for the Contrail Insights plug-ins to be able to read them. The path specified in log_file_path should be the location of the file on the OpenStack Controller host.

Note:

In Contrail Insights 3.0, all OpenStack log parser plug-ins have to be specified in the variable appformix_openstack_log_plugins. When upgrading from an earlier version to 3.0, make sure to move all OpenStack log parser plug-ins defined in appformix_plugins to appformix_openstack_log_plugins. Also, in Contrail Insights 3.0, all entries in this list have to be specified with a log_file_path value, as described in example above.

Remove a Node from Contrail Insights

Edit the inventory file and add appformix_state=absent to each node that you want to remove from Contrail Insights.

content_copy zoom_out_map
 # Example naming schemes are as below:
 #   hostname ansible_ssh_user='username' ansible_sudo_pass='password'

 # List all Compute Nodes
 [compute]
 172.16.70.5 appformix_state=absent
 172.16.70.17

Run Ansible with the edited inventory file.

content_copy zoom_out_map
ansible-playbook -i inventory appformix_openstack_on_kubernetes.yml

This removes the node and all its resources from Contrail Insights.

external-footer-nav