Contrail Insights Installation for OpenStack Helm
OpenStack Helm is a project that provides Helm charts for deploying OpenStack services on a Kubernetes cluster. Contrail Insights can be deployed to monitor both the Kubernetes resources as well as the OpenStack resources from a single Dashboard.
Architecture and Terminology
Kubernetes cluster nodes | Primary and worker nodes of the Kubernetes cluster being monitored by Contrail Insights. These nodes will run the Contrail Insights Agent. |
OpenStack cluster nodes | Kubernetes nodes from the cluster above that are running OpenStack services (Keystone, Nova, Neutron, and so on) in containers. |
Contrail Insights Platform node | Node on which Contrail Insights Platform components will be installed. Should be able to reach the Kubernetes cluster nodes. Figure 1: Contrail Insights and OpenStack Helm Workflow
|
Requirements
The following are the requirements for installing Contrail Insights for OpenStack Helm.
Supported Kubernetes versions: 1.8.x and later
See Contrail Insights General Requirements for hardware and software requirements.
API access to Kubernetes API server. Contrail Insights reads information about the Kubernetes cluster from the API server. The token provided during configuration must provide sufficient permission for read-only API calls. In addition, Contrail Insights Platform must be able to open a connection to the host and port on which the API server runs.
API access to OpenStack services. Contrail Insights reads information about the OpenStack cluster through the APIs exposed by the various OpenStack services. The user credentials provided during configuration must provide sufficient permission for read-only API calls. In addition, Contrail Insights Platform must be able to open a connection to the host and port on which the API server runs.
- Note:
Upgrade notice: Starting with Contrail Insights 3.2.6, the requirement for a license file is removed. If you are installing a version earlier than 3.2.6, a license is required prior to installation.
You can obtain a license key from mailto:APPFORMIX-KEY-REQUEST@juniper.net. Provide the following information in your request:
Group name: Target customers or use: Cluster type: OpenStack on Kubernetes Number of hosts: Number of instances:
Workflow
The installation consists of the following steps:
Initial setup.
Prerequisites for Agent installation.
Configuring Kubernetes.
Configuring OpenStack.
Installing Contrail Insights.
Optional configuration.
Initial Setup
Run the following commands for initial setup:
Install these required packages on the Contrail Insights Platform node.
#Ubuntu apt-get update apt-get install python-pip python-dev build-essential libssl-dev libffi-dev pip install ansible==2.3.0.0 markupsafe httplib2
#RHEL/CentOS yum install epel-release #Enable EPEL repository In case the above command does not work, manually download and install the epel-release package with one of the below commands, depending on your system’s version. yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-6.noarch.rpm yum groupinstall 'Development Tools' #Install development tools yum install openssl-devel libffi libffi-devel #Dependencies yum install python-pip python-devel #Install Pip pip install ansible==2.3.0 #Install Ansible 2.3 pip install markupsafe httplib2 #Dependencies
Note:For RHEL, the following iptables rule is needed to access port 9000.
iptables -t filter -A IN_public_allow -p tcp --dport 9000 -j ACCEPT
Edit the /etc/hosts file on the Contrail Insights Platform node and enter the IP addresses of the OpenStack on Kubernetes cluster nodes.
vi /etc/hosts <kube-master-ip> k8s-master <kube-worker1-ip> k8s-node1 <kube-worker2-ip> k8s-node2
Set up passwordless SSH between the Contrail Insights Platform node and the OpenStack on Kubernetes cluster nodes. Run the following commands to generate and copy the SSH public keys to all the nodes.
ssh-keygen -t rsa ssh-copy-id -i ~/.ssh/id_rsa.pub root@k8s-master ssh-copy-id -i ~/.ssh/id_rsa.pub root@k8s-node1 ssh-copy-id -i ~/.ssh/id_rsa.pub root@k8s-node2 ssh-copy-id -i ~/.ssh/id_rsa.pub root@<IP of Contrail Insights Platform node>
Prerequisites for Agent Installation
Contrail Insights uses the libvirt
library to collect data from virtual machines running on an OpenStack
compute node. Run the following command on all OpenStack computes
to install the library:
On Ubuntu hosts: sudo apt-get install libvirt0:amd64 On Centos hosts: sudo yum install libvirt-client
Configure Kubernetes
Contrail Insights reads information about resources in your
Kubernetes clusters. The software requires the cluster-admin
role or another role that gives it read-only access to all objects
in the cluster. We recommend that you create a new Service Account
for Contrail Insights and assign it the cluster-admin
role. If you do not create a new Service Account, then you must
provide the token from an existing Service Account that has the required
access during the configuration of Contrail Insights.
To create a new Service Account with the required access for Contrail Insights, perform the following steps in the OpenStack on Kubernetes cluster primary node:
Create a YAML file with the following:
apiVersion: v1 kind: ServiceAccount metadata: name: appformix
-
Create the
appformix
Service Account using the file created in Step 1.$ kubectl create -f <file>.yaml
Confirm that the Service Account has been created. Make a note of its namespace.
$ kubectl describe serviceaccount appformix Name: appformix Namespace: default Labels: <none> Annotations: <none> Image pull secrets: <none> Mountable secrets: appformix-token-pkljk Tokens: appformix-token-pkljk Events: <none>
-
Add the
cluster-admin
role to theappformix
Service Account as follows, substituting namespace for the namespace noted in Step 3.$ kubectl create clusterrolebinding appformix-binding --clusterrole=cluster-admin --serviceaccount=<namespace>:appformix
Run the following command to confirm that the
appformix
Service Account has the required access:$ kubectl auth can-i get nodes --as=system:serviceaccount:<namespace>:appformix --all-namespaces
The output of the command should be
yes
.Contrail Insights must be configured to communicate with the Kubernetes cluster. Get the following details from the Kubernetes cluster to use during the Contrail Insights installation.
kubernetes_cluster_url This is the URL of the Kubernetes API Server. To get this value, run the following command on the OpenStack on Kubernetes cluster:
$ kubectl cluster-info | grep 'Kubernetes master' Kubernetes master is running at https://172.24.1.173:6443
kubernetes_auth_token This is the authentication token of the
appformix
Service Account. To get this value, run the following commands on the OpenStack on Kubernetes cluster:$ kubectl describe serviceaccount appformix Name: appformix Namespace: default Labels: <none> Annotations: <none> Tokens: appformix-token-pkljk [...] $ kubectl describe secret appformix-token-pkljk Name: appformix-token-pkljk Namespace: default [...] token: eyJwb[...]
Configure OpenStack Administrator Account for Contrail Insights
Contrail Insights reads information about all projects configured in your OpenStack cluster. The software requires administrator privilege. We recommend you create a new administrator account for Contrail Insights. If you do not create a new administrator account, then you must provide the username and password of an existing administrator account during the configuration of Contrail Insights.
To create an administrator account for Contrail Insights, perform the following steps in the OpenStack Horizon dashboard:
Create a user account and name it appformix.
Select a new project for the user account.
Select role as admin.
Install Contrail Insights
To install Contrail Insights:
Download the Contrail Insights installation packages from software downloads to the Contrail Insights Platform node. Get the following files:
contrail-insights-<version>.tar.gz contrail-insights-dependencies-images-<version>.tar.gz contrail-insights-kubernetes-images-<version>.tar.gz contrail-insights-openstack-images-<version>.tar.gz contrail-insights-platform-images-<version>.tar.gz
If you are installing a version earlier than 3.2.6, copy the Contrail Insights license file to the Contrail Insights Platform node.
Unzip
contrail-insights-<version>.tar.gz
. This package contains all the Ansible playbooks required to install Contrail Insights.tar -xvzf contrail-insights-<version>.tar.gz cd contrail-insights-<version>/
Note:The remaining steps should be executed from within the
contrail-insights-<version>/
directory. Although the product name changed from "AppFormix" to "Contrail Insights," the UI and internal command paths continue to show AppFormix and will reflect the new name at a later date.Using
sample_inventory
as a template, create an inventory file for the installation. List the Kubernetes cluster nodes in thecompute
section and the Contrail Insights Platform node in theappformix_controller
section. List the node running the OpenStack controller services in theopenstack_controller
section.cp sample_inventory inventory vi inventory [compute] k8s-master k8s-node1 k8s-node2 [appformix_controller] <IP of the Contrail Insights Platform node> [openstack_controller] <IP of the Kubernetes node(s) running OpenStack controller services>
Create a directory called
group_vars
. Create a file namedall
inside this directory with configuration variables required by Contrail Insights.mkdir group_vars vi group_vars/all appformix_docker_images: - /path/to/contrail-insights-platform-images-<version>.tar.gz - /path/to/contrail-insights-dependencies-images-<version>.tar.gz - /path/to/contrail-insights-kubernetes-images-<version>.tar.gz - /path/to/contrail-insights-openstack-images-<version>.tar.gz appformix_dns_version: 2 kubernetes_platform_enabled: True openstack_platform_enabled: True kubernetes_cluster_url: <URL from Configuring Kubernetes step 4 above> kubernetes_auth_token: <token from Configuring Kubernetes step 4 above>
If you are installing a version earlier than 3.2.6, include the path to the Contrail Insights license file in
group_vars/all
:appformix_license: path/to/<contrail-insights-license-file>.sig
Source the
openrc
file from the OpenStack Controller node in the environment in which the Ansible playbooks are going to be executed. The file should look like the following:$ cat openrc export OS_CLOUD=openstack_helm export OS_USERNAME='admin' export OS_PASSWORD='password' export OS_PROJECT_NAME='admin' export OS_PROJECT_DOMAIN_NAME='default' export OS_USER_DOMAIN_NAME='default' export OS_AUTH_URL='http://keystone.openstack.svc.cluster.local/v3' $ source openrc
Run the Ansible playbook.
ansible-playbook -i inventory appformix_openstack_on_kubernetes.yml
Playbook should run to completion without any errors.
Log into the Contrail Insights Dashboard at:
http://<contrail-insights-platform-node-ip>:9000
In the Auth Service dropdown list, three options are provided:
OpenStack Log in with OpenStack credentials and view only OpenStack resources.
Kubernetes Log in with Contrail Insights credentials and view only Kubernetes resources.
AppFormix Log in with Contrail Insights credentials and view both OpenStack and Kubernetes resources.
For the last two options, the token to use for login is available at /opt/appformix/etc/appformix_token.rst on the Contrail Insights Platform node.
Optional Configuration
To install Contrail Insights certified plug-ins on the cluster,
include the following variables in the group_vars/all
file:
appformix_plugins: <list of certified plugins to be installed> appformix_openstack_log_plugins: <list of OpenStack log plugins to be installed>
For example:
appformix_plugins: - { plugin_info: 'certified_plugins/cassandra_node_usage.json' } - { plugin_info: 'certified_plugins/contrail_vrouter.json' } - { plugin_info: 'certified_plugins/zookeeper_usage.json' } - { plugin_info: 'certified_plugins/heavy_hitters.json' } appformix_openstack_log_plugins: - { plugin_info: 'certified_plugins/cinder_api_logparser.json', log_file_path: '/var/log/cinder/cinder-api.log' } - { plugin_info: 'certified_plugins/glance_logparser.json', log_file_path: '/var/log/glance/glance-api.log' } - { plugin_info: 'certified_plugins/keystone_logparser.json', log_file_path: '/var/log/apache2/keystone_access.log,/var/log/httpd/keystone_wsgi_admin_access.log,/var/log/keystone/keystone.log' }
For a list of all Contrail Insights certified plug-ins that
can be installed, look for the entries starting with plugin_info
in the file roles/appformix_defaults/defaults/main.yml
.
The OpenStack log parser plug-ins parse the API log files of
each OpenStack service to collect metrics about API calls and response
status codes. To install these plug-ins, add them to the variable appformix_openstack_log_plugins
in group_vars/all
, as shown above. Each plug-in entry in this list requires a parameter
called log_file_path
to be specified. This
parameter should be set to the complete path to the service's API
log file on the OpenStack Controller node(s). Multiple comma-separated
paths can be specified.
To identify the correct log file to be specified in log_file_path
, look for entries like the following,
containing a client IP address, REST call type, and response status
code:
2019-04-02 06:50:13.103 3465 INFO nova.osapi_compute.wsgi.server [req-d07e953a-6921-4224-a056-afb6ff69adde 953ea56a96b944b3b170a299af9e87bd 10c9e8809feb4bd1b55955d9c2ed5aba - - -] 172.18.0.6 "GET /v2/10c9e8809feb4bd1b55955d9c2ed5aba/os-hypervisors/detail HTTP/1.1" status: 200 len: 1427 time: 0.0208740 2019-04-02 06:50:13.183 3465 INFO nova.osapi_compute.wsgi.server [req-34b2f686-9eb5-4112-b3fc-e0b37798a302 953ea56a96b944b3b170a299af9e87bd 10c9e8809feb4bd1b55955d9c2ed5aba - - -] 172.18.0.6 "GET /v2/10c9e8809feb4bd1b55955d9c2ed5aba/servers/detail?all_tenants=1&status=SHELVED_OFFLOADED HTTP/1.1" status: 200 len: 211 time: 0.0754580
Default locations for these files are listed in the variable appformix_openstack_log_factory_plugins
in roles/appformix_defaults/defaults/main.yml
.
On containerized OpenStack environments, log files are generated
inside the containers running the OpenStack services. However, they
have to be available on the OpenStack controller host for the Contrail
Insights plug-ins to be able to read them. The path specified in log_file_path
should be the location of the file on
the OpenStack Controller host.
In Contrail Insights 3.0, all OpenStack log parser plug-ins
have to be specified in the variable appformix_openstack_log_plugins
. When upgrading from an earlier version to 3.0, make sure to move
all OpenStack log parser plug-ins defined in appformix_plugins
to appformix_openstack_log_plugins
. Also,
in Contrail Insights 3.0, all entries in this list have to be specified
with a log_file_path
value, as described
in example above.
Remove a Node from Contrail Insights
Edit the inventory file and add appformix_state=absent
to each node that you want to remove from Contrail Insights.
# Example naming schemes are as below: # hostname ansible_ssh_user='username' ansible_sudo_pass='password' # List all Compute Nodes [compute] 172.16.70.5 appformix_state=absent 172.16.70.17
Run Ansible with the edited inventory file.
ansible-playbook -i inventory appformix_openstack_on_kubernetes.yml
This removes the node and all its resources from Contrail Insights.