Aggregate Discovery and Alarms with OpenStack Heat Services
Contrail Insights integration with Heat has two independent aspects: discovery and alarms.
Heat Stack Discovery
The Contrail Insights OpenStack Adapter uses Heat APIs to discover Heat stacks in an OpenStack cluster. Each stack is represented as an aggregate in Contrail Insights with the label Heat. When OpenStack Adapter discovers a Heat stack, OpenStack Adapter configures an aggregate in the Contrail Insights Platform, and adds any virtual machines and virtual networks defined by the Heat stack as members of the aggregate. See Figure 1.
Discovery functionality is provided by Contrail Insights OpenStack Adapter as part of the standard OpenStack integration. Discovery does not require any modifications to the OpenStack controller, OpenStack configuration, or Heat service, and does not require installation of the Contrail Insights Heat plug-in.
Figure 2 shows two Heat stacks that were discovered by Contrail Insights: stack1 and ubuntu_stack. Each stack is represented in Contrail Insights as a Mixed aggregate. A Mixed aggregate may contain entities of different types, such as virtual machine and virtual network. To view aggregate Heat stacks, select Dashboard, then from the Infrastructure tab, select Aggregates.
Figure 3 shows the entities in the Heat stack ubuntu_stack. There are two virtual machines, ubuntu_vm1 and ubuntu_vm2, and one virtual network, ubuntu_network1. To view this page, select Dashboard, then from the Infrastructure tab, select Aggregates. From the Aggregate Select One tab, select the entity to view.
Contrail Insights Alarm Resource Types
In addition to discovery of Heat stacks, Contrail Insights provides a Heat plug-in that defines two new resource types for Heat templates.
| OS::AppFormix::Alarm | This resource type is used to define single alarms for monitoring resources in a Heat stack. |
| OS::AppFormix::CompositeAlarm | This resource type is used to define composite alarms for monitoring resources in a Heat stack. The alarms and composite alarms are configured in Contrail Insights when a Heat stack is created from the template, and are evaluated by the Contrail Insights stream-based, distributed analysis engine. |
You benefit by maintaining monitoring configuration in the same template that defines the resources (for example, virtual machines) to be monitored. Contrail Insights alarms and composite alarms are configured at the time that resources in the Heat stack are instantiated. Further, these alarms can be used to trigger scale-up and scale-down of Heat AutoScaling policies, which enables Heat to react more quickly and accurately due to the responsive and fine-grained Contrail Insights alarms.
When a Heat stack is created using a Heat template that has
an OS::AppFormix::Alarm or OS::AppFormix::CompositeAlarm resource, Heat will pass
the resource properties to Contrail Insights Heat plug-in to configure
the alarm in Contrail Insights. Figure 4 illustrates the interaction between Heat, Contrail Insights Heat
plug-in, and Contrail Insights Platform in the following sequence
of events.
User instantiates a Heat stack from a template with an
OS::AppFormix::AlarmorOS::AppFormix::CompositeAlarmresource.Heat passes the alarm properties to Contrail Insights Heat plug-in.
When the resource
OS::AppFormix::Alarmis used, Contrail Insights Heat plug-in configures an alarm in Contrail Insights Platform using the Contrail Insights REST API. The URL for Contrail Insights Platform is a configuration parameter (see “Install Contrail Insights Heat Plug-in”). The alarm is configured with mode set toEvent. The alarm will generate a notification for each interval in which the condition for the alarm is satisfied.When the resource
OS::AppFormix::CompositeAlarmis used, Contrail Insights Heat plug-in configures a composite alarm in Contrail Insights Platform using the Contrail Insights REST API. The user can define multiple individual alarms in the composite alarm. The state of the composite alarm is a combination of the states of the individual alarms. The user can define weights for the individual alarms and a threshold for the composite alarm. The composite alarm is active when the sum of the weights of the active alarms equals or exceeds the user-defined threshold (see “Example: Heat Autoscaling with OS::AppFormix::Alarm”). A notification will be generated every 60 seconds for as long as the composite alarm is active.When the alarm or composite alarm triggers, a notification is delivered by HTTP/HTTPS POST to an endpoint specified in the
notification_urlproperty of the alarm. To enable auto-scaling, a Heat template can specify thesignal_urlof a HeatScalingPolicyresource as thenotification_url. In that case, the notification is sent to Heat for processing.
To make the OS::AppFormix::Alarm and OS::AppFormix::CompositeAlarm resource types available
to Heat templates, the Contrail Insights Heat plug-in must be installed
and configured on the OpenStack controller host(s). See the following
section, “Install Contrail Insights Heat Plug-in.”
For more information about the extensible design of Heat resources using plug-ins, refer to Heat documentation.
Install Contrail Insights Heat Plug-In
To install and configure the Heat plug-in on the OpenStack controller host:
Copy
appformix-openstackpackage.Copy the
appformix_openstackPython package from the release bundle to the OpenStack controller host on which Heat service is running. This package is provided as either a Python wheel or an RPM package in thepkgdirectory of the release bundle.Install
appformix-openstackpackage.On the OpenStack controller host that runs the Heat service, install the
appformix-openstackpackage. The latest version ofappformix-openstackpackage is 0.6.2.$ pip install appformix_openstack-0.6.2-py2-none-any.whl
By default, this will install the resources in:
/usr/local/lib/python2.7/dist-packages/appformix/heat
If the OpenStack services are running in containers, the resources should be installed in a directory that is accessible to the Heat containers. Use the
--targetoption withpip installto install the resources in a different directory. For example:$ pip install --target=/var/lib/docker/volumes/opt_plugin/_data appformix_openstack-0.6.1-py2-none-any.whl
Modify the Heat configuration file.
Define a variable called
appformix_controller_urlin the[DEFAULT]section and set it to the base URL of the Contrail Insights Platform.Add the installation directory to the list of plug-in directories. Look for the
plugin_dirsentry in the[DEFAULT]section and add the installation directory to the end of the list. If the OpenStack services are running in containers, specify the mount path of the installation directory inside the Heat containers.If desired, define variables called
appformix_task_num_iterationsandappformix_task_wait_millisecondsto control how many times and how frequently the Heat plug-in checks the status of an Contrail Insights API request before declaring that the operation has timed out. Both variables accept Integer values. If these variables are undefined, they default to the following values:appformix_task_num_iterations = 10 appformix_task_wait_milliseconds = 200
This is what the
heat.conffile should look like after modification:[DEFAULT] ... appformix_controller_url = <base URL, e.g., http://appformix_platform_host:9000> plugin_dirs = [...], <e.g. /usr/local/lib/python2.7/dist-packages/appformix/heat> appformix_task_num_iterations = 10 appformix_task_wait_milliseconds = 200If the OpenStack services are running in containers, make sure the changes are made in the
heat.conffiles in all the Heat containers.Restart all the OpenStack Heat services.
service heat-api restart service heat-api-cfn restart service heat-engine restart
If the OpenStack services are running in containers, restart all the Heat containers.
docker restart heat_engine docker restart heat_api docker restart heat_api_cfn
OS::AppFormix::Alarm Configuration in Heat Template
The OS::AppFormix::Alarm resource
type can be used in Heat templates to create a Contrail Insights alarm.
The resource type has the following input parameters:
Parameter |
Description |
|---|---|
alarm_name |
A name that identifies the alarm. |
alarm_metric |
Metric to evaluate in the alarm. To see a list of choices, use the Contrail Insights API endpoint
Use the value in the |
threshold |
Value by which to compare a metric measurement. Units for the
threshold depend on the value of |
aggregation_function |
Operation to use for combining measured values before comparison. Choices are:
|
comparison_function |
Operation to use for comparing measured values to the threshold. Choices are:
|
duration |
Number of seconds for which sample values will be collected before being combined. |
num_intervals |
Number of intervals of length |
num_exception_intervals |
Number of intervals of length |
project_id |
(Optional) ID of a project that contains the instances on which the alarm should be evaluated. |
aggregate_id |
(Optional) ID of an aggregate that contains the instances on which the alarm should be evaluated. Use the following syntax to indicate that alarm should be evaluated on instances in the current Heat stack: aggregate_id: { get_param: "OS::stack_id" }Either |
notification_url |
URL to which a notification will be sent when the alarm is active. This is any URL prepared to
receive notification from Contrail Insights. For Heat templates that use Contrail
Insights Alarms to trigger autoscaling, this URL should be set to the
|
- Example: Heat Autoscaling with OS::AppFormix::Alarm
- Create a Heat Stack for the Auto-Scaling Template using OS::AppFormix::Alarm
Example: Heat Autoscaling with OS::AppFormix::Alarm
With Contrail Insights Heat plug-in, Contrail Insights Alarms
can be used in Heat Autoscaling templates in place of Ceilometer Alarms.
The following Heat template uses OS::AppFormix::Alarm to automatically scale the number of running instances based on
CPU utilization:
heat_template_version: 2014-10-16
description: Example auto scale group, policy and alarm
resources:
scaleup_group:
type: OS::Heat::AutoScalingGroup
properties:
cooldown: 60
desired_capacity: 1
max_size: 5
min_size: 1
resource:
type: OS::Nova::Server
properties:
key_name: heat_key
image: 8e571a43-25c7-4eb1-bbb6-13e446e99e8a
flavor: m1.tiny
name: "test_vm"
networks:
- network: afx-net
scaleup_policy:
type: OS::Heat::ScalingPolicy
properties:
adjustment_type: change_in_capacity
auto_scaling_group_id: { get_resource: scaleup_group }
cooldown: 60
scaling_adjustment: 1
scaledown_policy:
type: OS::Heat::ScalingPolicy
properties:
adjustment_type: change_in_capacity
auto_scaling_group_id: { get_resource: scaleup_group }
cooldown: 60
scaling_adjustment: -1
cpu_alarm_high:
type: OS::AppFormix::Alarm
properties:
alarm_name: 'cpu_alarm_high'
alarm_metric: 'cpu.usage'
aggregation_function: 'average'
comparison_function: 'above'
duration: 60
num_intervals: 1
num_exception_intervals: 1
threshold: 80
aggregate_id: { get_param: "OS::stack_id" }
notification_url: { get_attr: [scaleup_policy, signal_url] }
cpu_alarm_low:
type: OS::AppFormix::Alarm
properties:
alarm_name: 'cpu_alarm_low'
alarm_metric: 'cpu.usage'
aggregation_function: 'average'
comparison_function: 'below’
duration: 300
num_intervals: 1
num_exception_intervals: 1
threshold: 10
aggregate_id: { get_param: "OS::stack_id" }
notification_url: { get_attr: [scaledown_policy, signal_url] }
The following sequence describes what happens when a Heat stack is created from the “Heat Autoscaling with OS::AppFormix::Alarm” template.
When a heat stack is deployed using this template, a single instance
test_vmis initially created.The two Contrail Insights alarms
cpu_alarm_highandcpu_alarm_loware used to monitor CPU utilization on the instance. They can be defined to monitor any metric that Contrail Insights collects.When the CPU utilization on the instance goes above 80 percent, the alarm
cpu_alarm_highis triggered. This results in the execution of thescaleup_policywhich increases the number of running instances by 1 every 60 seconds, for as long as the alarm is active. Thescaleup_policystops executing when the number of running instances equals the value inmax_size.When the CPU utilization on the instance drops below 10 percent, the alarm
cpu_alarm_lowis triggered. This results in the execution of thescaledown_policywhich decreases the number of running instances by 1 every 300 seconds, for as long as the alarm is active. Thescaledown_policystops executing when the number of running instances equals the value inmin_size.
Create a Heat Stack for the Auto-Scaling Template using OS::AppFormix::Alarm
Now, let's create a Heat stack from the template “Example: Heat Autoscaling with OS::AppFormix::Alarm,” and observe what happens when we add CPU load on the VM to trigger the Contrail Insights Alarm.
Save the template defined in “Example: Heat Autoscaling with OS::AppFormix::Alarm” in a file named
appformix_autoscaling.yaml. Enter appropriate values in theproject_idoraggregate_idfields. Then create a Heat stack using the template:$ heat stack-create -f appformix_autoscaling.yaml stack1 +--------------------------------------+------------+--------------------+---------------------+--------------+ | id | stack_name | stack_status | creation_time | updated_time | +--------------------------------------+------------+--------------------+---------------------+--------------+ | 753e8bfd-047e-4297-aaef-3d1a68d36b24 | stack1 | CREATE_IN_PROGRESS | 2017-09-10T19:08:34 | None | +--------------------------------------+------------+--------------------+---------------------+--------------+ $ heat stack-list +--------------------------------------+------------+-----------------+---------------------+--------------+ | id | stack_name | stack_status | creation_time | updated_time | +--------------------------------------+------------+-----------------+---------------------+--------------+ | eb9b7dd3-c1a6-4f5d-9039-8c5968b88775 | stack1 | CREATE_COMPLETE | 2017-09-10T19:17:28 | None | +--------------------------------------+------------+-----------------+---------------------+--------------+
Check that there is a single
test_vminstance running.$ nova list +--------------------------------------+------------+--------+------------+-------------+----------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+------------+--------+------------+-------------+----------------------+ | 11b00a5b-fa62-407d-a155-e3b65b2436ca | test_vm | ACTIVE | - | Running | afx-net=192.168.10.3 | +--------------------------------------+------------+--------+------------+-------------+----------------------+
Generate some load on
test_vm. Watch for thecpu_alarm_highalarm to become active on the Contrail Insights Dashboard.When the alarm is active, check the running instances on the cluster. There should now be two running instances called
test_vm.$ nova list +--------------------------------------+------------+--------+------------+-------------+----------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+------------+--------+------------+-------------+----------------------+ | 0389529f-ae05-4677-99c9-fb79d27eb9e9 | test_vm | ACTIVE | - | Running | afx-net=192.168.10.4 | | 11b00a5b-fa62-407d-a155-e3b65b2436ca | test_vm | ACTIVE | - | Running | afx-net=192.168.10.3 | +--------------------------------------+------------+--------+------------+-------------+----------------------+
Stop the load generator on
test_vm. Watch for thecpu_alarm_lowalarm to become active on the Contrail Insights Dashboard.When the alarm is active, check the running instances on the cluster. There should now be a single
test_vminstance running.
OS::AppFormix::CompositeAlarm Configuration in Heat Template
You can define multiple individual Alarms and combine them in a Composite Alarm. The state of the Composite Alarm is a combination of the states of the individual Alarms. You can define weights for the individual Alarms and a threshold for the Composite Alarm. The Composite Alarm is active when the sum of the weights of the active Alarms equals or exceeds the user-defined threshold.
The OS::AppFormix::CompositeAlarm resource type can be used in Heat templates to create a Contrail
Insights Composite Alarm. The resource type has the following input
parameters:
Parameter |
Description |
|---|---|
composite_alarm_name |
A name that identifies the composite alarm. |
project_id |
(Optional) ID of a project that contains the instances on which the composite alarm should be evaluated. |
aggregate_id |
(Optional) ID of an aggregate that contains the instances on which the composite alarm should be evaluated. Use the following syntax to indicate that the alarm should be evaluated on instances in the current Heat stack: aggregate_id: { get_param: "OS::stack_id" }Either |
notification_url |
URL to which a notification will be sent when the alarm is active. This is any URL prepared to
receive notification from Contrail Insights. For Heat templates that use Contrail
Insights Alarms to trigger autoscaling, this URL should be set to the
|
composite_alarm_rules |
List of individual alarms that make up the composite alarm.
Each individual alarm has the same input parameters as the |
composite_alarm_threshold |
Value between 0 and 1 used to determine if composite alarm is active. If the sum of weights of all active rules equals or exceeds the threshold, the composite alarm is determined to be active. |
- Example: Heat Autoscaling with OS::AppFormix::CompositeAlarm
- Create a Heat Stack for the Auto-Scaling Template using OS::AppFormix::CompositeAlarm
Example: Heat Autoscaling with OS::AppFormix::CompositeAlarm
With Contrail Insights Heat plug-in, Contrail Insights Composite
Alarms can be used in Heat Autoscaling templates in place of Ceilometer
Alarms. The following Heat template uses OS::AppFormix::CompositeAlarm to automatically scale the number of running instances based on
CPU utilization and memory utilization.
heat_template_version: 2014-10-16
description: Example auto scale group, policy and alarm
resources:
scaleup_group:
type: OS::Heat::AutoScalingGroup
properties:
cooldown: 120
desired_capacity: 1
max_size: 5
min_size: 1
resource:
type: OS::Nova::Server
properties:
key_name: heat_key
image: 8e571a43-25c7-4eb1-bbb6-13e446e99e8a
flavor: m1.tiny
name: "test_vm"
networks:
- network: afx-net
scaleup_policy:
type: OS::Heat::ScalingPolicy
properties:
adjustment_type: change_in_capacity
auto_scaling_group_id: { get_resource: scaleup_group }
cooldown: 120
scaling_adjustment: 1
scaledown_policy:
type: OS::Heat::ScalingPolicy
properties:
adjustment_type: change_in_capacity
auto_scaling_group_id: { get_resource: scaleup_group }
cooldown: 120
scaling_adjustment: -1
composite_alarm_high:
type: OS::AppFormix::CompositeAlarm
properties:
composite_alarm_name: 'composite_alarm_high'
aggregate_id: { get_param: "OS::stack_id" }
notification_url: { get_attr: [scaleup_policy, signal_url] }
composite_alarm_threshold: 0.5
composite_alarm_rules:
- alarm_name: 'rule1'
alarm_metric: 'cpu.usage'
aggregation_function: 'average'
comparison_function: 'above'
duration: 180
num_intervals: 1
num_exception_intervals: 1
threshold: 80
alarm_weight: 0.5
- alarm_name: 'rule2'
alarm_metric: 'memory.usage'
aggregation_function: 'average'
comparison_function: 'above'
duration: 180
num_intervals: 1
num_exception_intervals: 1
threshold: 80
alarm_weight: 0.5
composite_alarm_low:
type: OS::AppFormix::CompositeAlarm
properties:
composite_alarm_name: 'composite_alarm_low'
aggregate_id: { get_param: "OS::stack_id" }
notification_url: { get_attr: [scaledown_policy, signal_url] }
composite_alarm_threshold: 1.0
composite_alarm_rules:
- alarm_name: 'rule3'
alarm_metric: 'cpu.usage'
aggregation_function: 'average'
comparison_function: 'below'
duration: 300
num_intervals: 1
num_exception_intervals: 1
threshold: 10
alarm_weight: 0.5
- alarm_name: 'rule4'
alarm_metric: 'memory.usage'
aggregation_function: 'average'
comparison_function: 'below'
duration: 300
num_intervals: 1
num_exception_intervals: 1
threshold: 10
alarm_weight: 0.5
The following sequence describes what happens when a Heat stack is created from the above template.
When a heat stack is deployed using this template, a single instance
test_vmis initially created.The two Contrail Insights composite alarms
composite_alarm_highandcomposite_alarm_loware used to monitor resource utilization on the instances in the Heat stack. They are comprised of two individual alarms to monitor CPU utilization and memory utilization on the instances. They can be defined to monitor any metric that Contrail Insights collects.The individual alarms in
composite_alarm_highare defined with weights of 0.5 each and the composite alarm is defined with a threshold of 0.5. This means that the composite alarm will be considered active when any of the individual alarms is active.The individual alarms in
composite_alarm_loware defined with weights of 0.5 each and the composite alarm is defined with a threshold of 1.0. This means that the composite alarm will be considered active only when both the individual alarms are active.When either the CPU utilization or memory utilization on the instance
test_vmgoes above 80 percent, the composite alarmcomposite_alarm_highis triggered. This results in the execution of thescaleup_policywhich increases the number of running instances by 1 every 60 seconds for as long as the composite alarm is active. Thescaleup_policystops executing when the number of running instances equals the value inmax_size.When both the CPU utilization and memory utilization on the instance
test_vmdrops below 10 percent, the composite alarmcomposite_alarm_lowis triggered. This results in the execution of thescaledown_policywhich decreases the number of running instances by 1 every 60 seconds, for as long as the composite alarm is active. Thescaledown_policystops executing when the number of running instances equals the value inmin_size.
Create a Heat Stack for the Auto-Scaling Template using OS::AppFormix::CompositeAlarm
Now, let's create a Heat stack from the template “Example: Heat Autoscaling with OS::AppFormix::CompositeAlarm” and observe what happens when we add CPU load on the VM to trigger the Contrail Insights Composite Alarm.
Save the template defined in “Example: Heat Autoscaling with OS::AppFormix::CompositeAlarm” in a file called
appformix_composite_autoscaling.yaml. Enter appropriate values in theproject_idoraggregate_idfields. Then create a Heat stack using the template:$ heat stack-create -f appformix_composite_autoscaling.yaml composite1 +--------------------------------------+------------+--------------------+----------------------+--------------+ | id | stack_name | stack_status | creation_time | updated_time | +--------------------------------------+------------+--------------------+----------------------+--------------+ | f2bc3282-1d8b-4230-a3ef-a589f3527188 | composite1 | CREATE_IN_PROGRESS | 2018-10-30T03:12:08Z | None | +--------------------------------------+------------+--------------------+----------------------+--------------+ $ heat stack-list +--------------------------------------+------------+-----------------+----------------------+--------------+ | id | stack_name | stack_status | creation_time | updated_time | +--------------------------------------+------------+-----------------+----------------------+--------------+ | f2bc3282-1d8b-4230-a3ef-a589f3527188 | composite1 | CREATE_COMPLETE | 2018-10-30T03:12:08Z | None | +--------------------------------------+------------+-----------------+----------------------+--------------+
Check that there is a single
test_vminstance running.$ nova list +--------------------------------------+---------+--------+------------+-------------+------------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+---------+--------+------------+-------------+------------------------+ | e9bc550d-084e-410c-8154-4c590f504a69 | test_vm | ACTIVE | - | Running | private-2=192.168.27.3 | +--------------------------------------+---------+--------+------------+-------------+------------------------+
Generate some load on
test_vm. Watch for thecomposite_alarm_highcomposite alarm to become active on the Contrail Insights Dashboard.When the composite alarm is active, check the running instances on the cluster. There should now be two running instances called
test_vm.$ nova list +--------------------------------------+---------+--------+------------+-------------+------------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+---------+--------+------------+-------------+------------------------+ | e9bc550d-084e-410c-8154-4c590f504a69 | test_vm | ACTIVE | - | Running | private-2=192.168.27.3 | | f7feb43b-221d-4738-9092-476fa2e4b3aa | test_vm | ACTIVE | - | Running | private-2=192.168.27.8 | +--------------------------------------+---------+--------+------------+-------------+------------------------+
Stop the load generator on
test_vm. Watch for thecomposite_alarm_lowcomposite alarm to become active on the Contrail Insights Dashboard.When the alarm is active, check the running instances on the cluster. There should now be a single
test_vminstance running.
Troubleshooting
For debugging, enable the verbose and debug options by adding them to the [DEFAULT] section in heat.conf.
verbose = True debug = True
Then restart the Heat services or containers. Detailed logs
will appear in /var/log/heat/heat-engine.log.