Performing a Health Check of Infrastructure Components

date_range 30-Sep-21

arrow_backward

arrow_forward

After you install or upgrade CSO, you can run the components_health.sh script to perform a health check of all infrastructure components. This script detects whether any infrastructure component has failed and displays the health status of the following infrastructure components:

SaltStack
Cassandra
MariaDB
Swift
Redis
ArangoDb
Keystone
Elasticsearch
Elk Elasticsearch
Icinga
RabbitMQ
Etcd
Rsyslog
Kubernetes
ELK Logstash
ELK Kibana
ZooKeeper
Contrail Analytics
VRR
Microservices

To check the status of infrastructure components:

Navigate to the CSO directory in the startupserver_1 VM.

For example:

content_copy zoom_out_map
root@host:~/# cd Contrail_Service_Orchestration_6.1.0
root@host:~/Contrail_Service_Orchestration_6.1.0#

Run the components_health.sh script.

To check the status of one of infrastructure components, run the following command:

content_copy zoom_out_map
root@startupserver_1:/opt/Contrail_Service_Orchestration_6.1.0# ./components_health.sh --component=<component_name>
For Example:
root@startupserver_1:/opt/Contrail_Service_Orchestration_6.1.0# ./components_health.sh --component=elasticsearch

Run the following command to check the health of all the infrastructure components.

content_copy zoom_out_map
root@startupserver_1:/opt/Contrail_Service_Orchestration_6.1.0# ./components_health.sh 

After a couple of minutes, the status of each infrastructure component is displayed.

For example:

content_copy zoom_out_map
INFO     Updating the mine and syncing the grains
INFO     ************************************************************************
INFO     HEALTH CHECK FOR INFRASTRUCTURE COMPONENTS STARTED IN CENTRAL ENVIRONMENT
INFO     ************************************************************************

INFO     Health Check for Infrastructure Component Saltstack Started
INFO     The Infrastructure Component Saltstack is Healthy

INFO     Health Check for Infrastructure Component Etcd Started
INFO     The Infrastructure Component Etcd is Healthy

INFO     Health Check for Infrastructure Component Mariadb Started
INFO     The Infrastructure Component Mariadb is Healthy

INFO     Health Check for Infrastructure Component Keystone Started
INFO     The Infrastructure Component Keystone is Healthy

INFO     Health Check for Infrastructure Component Swift Started
INFO     The Infrastructure Component Swift is Healthy

INFO     Health Check for Infrastructure Component Elk_Kibana Started
INFO     The Infrastructure Component Elk_Kibana is Healthy

INFO     Health Check for Infrastructure Component Rsyslog Started
INFO     The Infrastructure Component Rsyslog is Healthy

INFO     Health Check for Infrastructure Component Kubernetes Started
INFO     The Infrastructure Component Kubernetes is Healthy

INFO     Health Check for Infrastructure Component Vrr Started
INFO     The Infrastructure Component Vrr is Healthy

INFO     Health Check for Infrastructure Component Zookeeper Started
INFO     The Infrastructure Component Zookeeper is Healthy

INFO     Health Check for Infrastructure Component Arangodb Started
INFO     The Infrastructure Component Arangodb is Healthy

INFO     Health Check for Infrastructure Component Icinga Started
INFO     The Infrastructure Component Icinga is Healthy

INFO     Health Check for Infrastructure Component Redis Started
INFO     The Infrastructure Component Redis is Healthy

INFO     Health Check for Infrastructure Component Rabbitmq Started
INFO     The Infrastructure Component Rabbitmq is Healthy


INFO     Health Check for Infrastructure Component Elk_Kibana Started
INFO     The Infrastructure Component Elk_Kibana is Healthy

INFO     Health Check for Infrastructure Component Rsyslog Started
INFO     The Infrastructure Component Rsyslog is Healthy

INFO     Health Check for Infrastructure Component Kubernetes Started
INFO     The Infrastructure Component Kubernetes is Healthy

INFO     Health Check for Infrastructure Component Vrr Started
INFO     The Infrastructure Component Vrr is Healthy

INFO     Health Check for Infrastructure Component Zookeeper Started
INFO     The Infrastructure Component Zookeeper is Healthy

INFO     Health Check for Infrastructure Component Arangodb Started
INFO     The Infrastructure Component Arangodb is Healthy

INFO     Health Check for Infrastructure Component Icinga Started
WARNING  Icinga is not running in 1 node
INFO     The Infrastructure Component Icinga is Healthy

INFO     Health Check for Infrastructure Component Redis Started
INFO     The Infrastructure Component Redis is Healthy

INFO     Health Check for Infrastructure Component Rabbitmq Started
INFO     The Infrastructure Component Rabbitmq is Healthy

INFO     Health Check for Infrastructure Component Elk_Logstash Started
INFO     The Infrastructure Component Elk_Logstash is Healthy

INFO     Health Check for Infrastructure Component Elasticsearch Started
INFO     The Infrastructure Component Elasticsearch is Healthy

INFO     Health Check for Infrastructure Component Elk_Elasticsearch Started
INFO     The Infrastructure Component Elk_Elasticsearch is Healthy

INFO     Health Check for Infrastructure Component Cassandra Started
INFO     The Infrastructure Component Cassandra is Healthy

INFO     Health Check for Infrastructure Component Contrail_Analytics Started
INFO     The Infrastructure Component Contrail_Analytics is Healthy

INFO     Health Check for Infrastructure Component Microservices Started
INFO     The Infrastructure Component Microservices is Healthy

INFO     Overall result:
INFO     The following Infrastructure Components are Healthy:
INFO     ['Saltstack', 'Etcd', 'Mariadb', 'Keystone', 'Swift', 'Elk_Kibana', 'Rsyslog', 'Kubernetes', 'Vrr', 'Zookeeper', 'Arangodb', 'Icinga', 'Redis', 'Rabbitmq', 'Elk_Logstash', 'Elasticsearch', 'Elk_Ela
sticsearch', 'Cassandra', 'Contrail_Analytics', 'Microservices']

If the components_health.sh script detects any service as unhealthy, then it displays an error message as shown in the following sample output.

content_copy zoom_out_map
root@startupserver1:~/opt/Contrail_Service_Orchestration_6.1.0# ./components_health.sh
INFO     Updating the mine and syncing the grains
INFO     ************************************************************************
INFO     HEALTH CHECK FOR INFRASTRUCTURE COMPONENTS STARTED IN CENTRAL ENVIRONMENT
INFO     ************************************************************************

 

INFO     Health Check for Infrastructure Component Saltstack Started
INFO     Attempt: 1 - Retrying Health Check for Component Saltstack
INFO     Attempt: 2 - Retrying Health Check for Component Saltstack
ERROR    The Infra Component : Saltstack is Unhealthy


ERROR    ERROR: SaltStack is Unhealthy. Please run CSO recovery to recover SaltStack

To recover the service, you must run the recovery.sh script. See Recovering CSO Services.

arrow_backward PREVIOUS Viewing Information About Microservices

NEXT arrow_forward Recovering CSO Services

Contrail Service Orchestration Monitoring and Troubleshooting Guide

Performing a Health Check of Infrastructure Components

Stay in touch

Helped me install or use the product
Accelerated my deployment

Discuss the product with a co-worker
Make a purchase inquiry