Contrail Insights with Kafka

Set Up Kafka

Note:

Contrail Insights does not explicitly create Kafka topics. The Kafka broker cluster should be configured to auto-create topics. Alternatively, you can manually manage the topic creation. If you already have Kafka running, you can skip Set Up Kafka and go directly to Set Up Contrail Insights with Kafka.

Setting up Kafka as a Docker container:

1. Create a Docker network for this Kafka container and its dependencies to be connected to by running the following command:

2. Next, bring up Zookeeper for Kafka to work:

3. Bring up the Kafka container by running the following. The variable ip_address must be specified appropriately.

4. If Kafka with SSL is required, then additional parameters are required:

   The secret_files_path should be replaced where all of the keystore and truststore files are present. For an example of how to create the above keystores and truststores, reference: https://github.com/confluentinc/cp-docker-images/blob/5.2.1-post/examples/kafka-cluster-ssl/secrets/create-certs.sh.

   Note:

   If Kafka with SSL is required, then all of the hosts monitored by Contrail Insights must have at least Python version of 2.7.9. It is also required that the Certificate Authority (CA) used for the certificates for the Kafka broker(s) be a trusted CA on all of the hosts monitored by Contrail Insights. In order for Contrail Insights containers to communicate with the Kafka broker(s), the CA file must be set as a group_vars/all variable appformix_kafka_ssl_ca at installation time.

   Now that Kafka is set up, next you can configure Contrail Insights with Kafka.

Set Up Contrail Insights with Kafka

To configure Contrail Insights with Kafka, a POST request must be sent to an Contrail Insights Platform API:

The following fields must be sent in this request:

Name	The name of the Kafka cluster, which can be anything.
BootstrapServers	A list of host/port pairs to use for establishing the initial connection to the Kafka cluster. Each item in the list is a string in the format host:port.

To send a POST body request using Ansible:

Run the POST body request, which is similar to the following:

If Kafka has been set up with SSL, then an additional field is needed:

To send a POST body request from the Contrail Insights Dashboard:

1. Select Settings in the upper right corner, then select AppFormix Settings > Kafka. Next, click + Add Config.

   Figure 1: AppFormix Settings for Kafka Page

2. Enter a name for the Kafka configuration and list the BootstrapServers as a comma separated list of strings with each string in the host:port format.

3. Click Setup after the fields have been populated.

   Note:

   The following steps are for streaming network telemetry data to Kafka. All Contrail Insights alarms are automatically sent to Kafka once Kafka has been configured as stated in the earlier procedures above. There are no additional steps needed for alarms. See Contrail Insights Alarms With Kafka.

4. Click + Add Subscription to create a subscription.

5. Next create a Topic, select devices, and then select which Sensors/MIBs you want sent to Kafka. The specified data will then be sent to Kafka under the specified topic. Click Create Subscription after the fields are populated.

Messages from Contrail Insights to Kafka

After configuration, messages from Contrail Insights are received by an appropriate Kafka consumer.

In the following command, bootstrap_server is one of the bootstrap servers specified in the BootstrapServers variable above and topic is the topic that was specified in the subscription created. :

This command outputs messages to standard output. Output for topic grpc-components with sensor /components/ selected will look something like:

Contrail Insights Alarms With Kafka

Contrail Insights Alarms are configured to automatically send alerts to Kafka, if Kafka has been configured in Contrail Insights. See Set Up Contrail Insights with Kafka. Contrail Insights sends alarms with the topic as the alarm's name. For example, Alarm name host_cpu is sent to Kafka with topic host_cpu.