Paragon Insights Pull-Model Ingest Methods
Paragon Insights currently supports the following pull-model sensors:
Server Monitoring Ingest
Starting with Paragon Insights Release 4.1.0, the Server Monitoring sensor collects data from servers and virtual machines on which you host the Paragon application. The sensor uses the third-party plug-in, Node Exporter. The Node Exporter plug-in is pre-installed in all server clusters of Paragon Insights. In the GUI, the default servers and virtual machines deployed in the Paragon Insights cluster are represented as devices that are automatically added to the Paragon-Cluster device group. The sensor collects data from servers and virtual machines to track CPU, memory, network, traffic, disk, and filesystem metrics. It writes the output to a time series database.
Paragon Insights has the following pre-configured playbooks to monitor server data.
CPU utilization
Disk reads and writes
Errors, available bytes, and utilized bytes in filesystem
Utilized bytes and available bytes in memory
Received and transmitted total packet size, errors in received and transmitted packets, total received and transmitted multicast packets in network
When you configure a rule using Server Monitoring ingest, you can use the some of the sensor paths listed in Table 1.
Sensor Path |
Description |
|---|---|
/node/boot/time/seconds |
Boot time in each server node. |
/node/cpu/seconds/total |
The total time (in seconds) the CPU stays in idle, system, user, and nice modes. |
/node/disk/read/bytes/total |
The total number of bytes read successfully. |
/node/disk/read/errors/total |
The total number of read errors in nodes. |
/node/disk/read/retries/total |
The number of times the ingest tries to read from the disk if there is a failure. |
/node/disk/read/sectors/total |
The total number of sectors read successfully. |
/node/disk/read/time/seconds/total |
The total time taken to complete reads successfully per node. |
/node/disk/reads/completed/total |
The total number of reads completed successfully. |
/node/disk/write/errors/total |
The total number of errors in writes. |
/node/disk/write/retries/total |
The number of times the ingest tries to write on the disk if there is a failure. |
/node/disk/write/time/seconds/total |
The total time taken to complete all writes. |
/node/disk/writes/completed/total |
The total number of writes completed per node. |
/node/disk/written/bytes/total |
The total number of bytes written successfully. |
/node/disk/written/sectors/total |
The total number of sectors written successfully. |
/node/exporter/build/info |
A metric that has the value '1' and has version, revision, go version, and branch from which node exporter is built. |
/node/filesystem/avail/bytes |
The filesystem size available to non-root users. |
/node/filesystem/device/error |
The number of I/O errors that occur when collecting data from a filesystem. |
/node/filesystem/files |
The total number of index nodes permitted in a node. |
/node/filesystem/files/free |
The number of index nodes that are free for use in a node. |
/node/filesystem/free/bytes |
The free space (in bytes) available for the user, excluding reserved blocks. |
/node/filesystem/readonly |
Data that shows if the filesystem in a node is mounted as read-only. |
/node/filesystem/size/bytes |
The size of all files in bytes. |
/node/load1 |
Load on each server/host node captured every 1 minute. |
/node/load15 |
Load on each server/host node captured every 15 minutes. |
/node/load5 |
Load on each server/host node captured every 5 minutes. |
/node/memory/active/bytes |
Size of memory in bytes that are actively used by processes. |
/node/memory/compressed/bytes |
Total size of compressed memory. |
/node/memory/free/bytes |
Total memory in bytes that is free for use in a node. |
/node/memory/inactive/bytes |
Memory bytes that are not actively used by processes. |
/node/memory/swap/total/bytes |
Total size of memory swapped in all nodes. |
/node/memory/swap/used/bytes |
The size of swapped memory used by nodes. |
/node/memory/swapped/in/bytes/total |
Size of memory swapped back to RAM in all nodes. |
/node/memory/swapped/out/bytes/total |
Size of memory swapped out from RAM in all nodes. |
/node/memory/total/bytes |
Total bytes of memory in all nodes. |
/node/memory/wired/bytes |
Memory that cannot be swapped out. |
/node/network/receive/bytes/total |
Total size of packets received by a device. |
/node/network/receive/errs/total |
Total number of errors encountered by a device when receiving packets. |
/node/network/receive/multicast/total |
Total number of multicast packets received by a device. |
/node/network/receive/packets/total |
Total number of packets received by a device. |
/node/network/transmit/bytes/total |
Total size of packets received by a device. |
/node/network/transmit/errs/total |
Total number of errors encountered by a device when receiving packets. |
/node/network/transmit/multicast/total |
Total number of multicast packets transmitted by a device. |
node/network/transmit/packets/total |
Total number of packets transmitted by a device. |
/node/scrape/collector/duration/seconds |
Time taken by each collector to scrape metrics. |
/node/scrape/collector/success |
Number of times Node Exporter collector successfully scraped targets. |
/node/textfile/scrape/error |
Errors encountered by Node Exporter when scraping targets using textfile scripts. |
/node/time/seconds |
Displays system time in seconds in the node since epoch (1970). |
/node/uname/info |
Name of the node from which Node Exporter collects metrics. |
The following tags such as mode, device etc. can be used as key fields applicable to all metrics listed under main metrics (/node/cpu or /node/disk). When you configure a key field in a rule, you can mention only the key field name in Path field.
/node/cpu/
cpu: The number of cores available in CPU.
mode: The type of CPU usage in a node such as idle, system, user, and nice.
/node/disk/
device: Name of disks such as disk0, disk1, sda, sdb, or sdc.
/node/filesystem/
device: Disk paths such as /dev/sda1, /dev/sda2, and /dev/sdb1
fstype: Type of partition formatting such as ext4, NTFS (New Technology File System), and APFS (Apple File System).
mountpoint: Mount paths in the device.
/node/network/
device: Interface names of the device such as wlan0, en0, or docker0.
Configure a Rule Using Server Monitoring Sensor
In the following example, you can use server monitoring sensor to collect disk read data from servers. You can configure fields for total disk read size, time taken to perform the reads, and name of the device that has the disk (key field). You can also calculate rate of disk read and configure a trigger alert when the total disk read exceeds a preset threshold.
To collect data on metrics, you must add the rule to a Playbook and apply a Playbook instance to device or network groups.
When you start collecting server metrics, you can see the logs by providing the pod IDs for the ingest.
Log in to the Paragon Insights management CLI.
Type the command
healthbot k logs server-monitoring pod id.
Understanding kube-state-metrics Service
kube-state-metrics service is a Beta feature as of Paragon Insights Release 4.2.0.
kube-state-metrics is a third-party metrics monitoring service that generates metrics based on the current state of Kubernetes clusters. You can use kube-state-metrics to monitor the health of Kubernetes cluster and services. With Release 4.2.0, kube-state-metrics service is supported as a beta-only feature. kube-state-metrics runs as a cluster service, and is installed automatically when you install Paragon Insights. Once this service is installed, you can enable this service to generate, monitor, and expose metrics of various objects within a Kubernetes cluster.
kube-state-metrics service provides metrics on pods, DaemonSets, deployments, persistent volume, endpoints, ingress, job, lease, and configmap objects that are part of a Kubernetes cluster.
The following is a list of some metrics that are exposed:
Pods running in a namespace
Pods that are available
Information on successful/failed deployments
State of persistent volumes
Information on currently running, successful, and failed jobs
Pods that are in error state
Health of deployment and DaemonSets
Status and condition of Kubernetes nodes
Enable kube-state-metrics
You can enable kube-state-metrics from the CLI as well as from the Paragon Insights UI.
The following is an overview of steps to enable kube-state-metrics from the UI:
Create a device.
The device represents the cluster. The device hostname must be the kube-state-metrics service IP. For example,
kube-state-metrics.healthbot.svc.cluster.local.Add the device to a device group.
Create rules with server-monitoring sensor type and relevant kube-state-metrics sensor path(s).
Apply playbooks.
Sample Rules and Playbooks
check-daemonset-status.rule
healthbot {
topic kube-metrics {
rule check-daemonset-status {
keys [ daemonset namespace ];
synopsis "";
description "Checks daemon set unavailable status";
sensor daemonset-status {
description "Checks daemon set unavailable status";
server-monitoring {
sensor-name /kube/daemonset;
frequency 60s;
}
}
field daemonset {
sensor daemonset-status {
path daemonset;
}
type string;
description "Checks status for demonset key";
}
field daemonset_status {
sensor daemonset-status {
path /kube/daemonset/status/number/unavailable;
}
type float;
description "Field to check condition";
}
field namespace {
sensor daemonset-status {
path namespace;
}
type string;
description "Checks status for namespace key";
}
trigger daemonset-status {
frequency 1offset;
term available {
when {
equal-to "$daemonset_status" 0;
}
then {
status {
color green;
message "Unavailable demons set is 0 for $namespace $daemonset";
}
}
}
term notavailable {
then {
status {
color red;
message "Unavailable demons set is not 0 for $namespace $daemonset";
}
}
}
}
}
}
}
check-deployment-status-condition.rule
healthbot {
topic kube-metrics {
rule check-deployment-status-condition {
keys [ condition deployment namespace ];
synopsis "";
description "Checks kube metrics deployment status condition";
sensor deployment-status {
description "Checks kube metrics deployment status condition";
server-monitoring {
sensor-name /kube/deployment;
frequency 60s;
}
}
field condition {
sensor deployment-status {
path condition;
}
type string;
description "Deployment condition";
}
field deployment {
sensor deployment-status {
path deployment;
}
type string;
description "Deployment pod name";
}
field namespace {
sensor deployment-status {
path namespace;
}
type string;
description "Deployment namespace";
}
field status {
sensor deployment-status {
path status;
}
type string;
description "Checks for true or false condition";
}
trigger deployment-status {
frequency 1offset;
term available {
when {
matches-with "$status" true;
}
then {
status {
color green;
message "Deployment status for $deployment is $status";
}
}
}
term notavailable {
then {
status {
color red;
message "Deployment status for $deployment is $status";
}
}
}
}
}
}
} check-deployment-status-replicas.rule
healthbot {
topic kube-metrics {
rule check-deployment-status-replicas {
keys [ deployment namespace ];
synopsis "";
description "Checks kube metrics deployment replica status";
sensor deployment-status {
description "Checks kube metrics deployment replica status";
server-monitoring {
sensor-name /kube/deployment;
frequency 60s;
}
}
field deployment {
sensor deployment-status {
path deployment;
}
type string;
description "Deployment pod name";
}
field deployment_status {
sensor deployment-status {
path /kube/deployment/status/replicas;
}
type float;
description "Field to check 0 or other values";
}
field namespace {
sensor deployment-status {
path namespace;
}
type string;
description "Namespace key";
}
trigger deployment-status {
frequency 1offset;
term available {
when {
not-equal-to "$deployment_status" 0;
}
then {
status {
color green;
message "Deployment status for replicaset is $deployment_status for $namespace $deployment";
}
}
}
term notavailable {
then {
status {
color red;
message "Deployment status for replicaset is $deployment_status for $namespace $deployment";
}
}
}
}
}
}
} check-pod-container-restarts.rule
healthbot {
topic kube-metrics {
rule check-pod-container-restarts {
keys [ container namespace pod uid ];
synopsis "";
description "Checks pod container status restarts";
sensor pod-init-container-status {
description "Checks pod container status restarts";
server-monitoring {
sensor-name /kube/pod/container;
frequency 60s;
}
}
field container {
sensor pod-init-container-status {
path container;
}
type string;
description "container name";
}
field namespace {
sensor pod-init-container-status {
path namespace;
}
type string;
description "namespace of the pod";
}
field pod {
sensor pod-init-container-status {
path pod;
}
type string;
description "pod name";
}
field restart-value {
sensor pod-init-container-status {
path /kube/pod/container/status/restarts/total;
}
type integer;
description "restart status value";
}
field uid {
sensor pod-init-container-status {
path uid;
}
type string;
description "Id ";
}
trigger restart-total-status {
frequency 1offset;
term less-than-ten {
when {
less-than-or-equal-to "$restart-value" 10;
}
then {
status {
color green;
message "$namespace pod $pod container $container restart total is $restart-value ";
}
}
}
term between-ten-and-twenty {
when {
range "$restart-value" {
min 10;
max 20;
}
}
then {
status {
color yellow;
message "$namespace pod $pod container $container restart total is $restart-value ";
}
}
}
term more-than-twenty {
then {
status {
color red;
message "$namespace pod $pod container $container restart total is $restart-value";
}
}
}
}
}
}
} check-pod-init-container-status.rule
healthbot {
topic kube-metrics {
rule check-pod-init-container-status {
keys [ container namespace pod uid ];
synopsis "";
description "Checks pod init container status waiting";
sensor pod-init-container-status {
description "Checks pod init container status waiting";
server-monitoring {
sensor-name /kube/pod/init/container;
frequency 60s;
}
}
field container {
sensor pod-init-container-status {
path container;
}
type string;
description "container name";
}
field namespace {
sensor pod-init-container-status {
path namespace;
}
type string;
description "namespace of the pod";
}
field pod {
sensor pod-init-container-status {
path pod;
}
type string;
description "pod name";
}
field status {
sensor pod-init-container-status {
path /kube/pod/init/container/status/waiting;
}
type integer;
description "Statius value (0/1)";
}
field uid {
sensor pod-init-container-status {
path uid;
}
type string;
description "Id ";
}
trigger waiting-status {
frequency 1offset;
term matches-zero {
when {
equal-to "$status" 0 {
time-range 10offset;
}
}
then {
status {
color green;
message "$namespace $pod $container status is $status";
}
}
}
term is-not-zero {
then {
status {
color red;
message "$namespace $pod $container status is $status";
}
}
}
}
}
}
} check-node-status.rule
healthbot {
topic kube-metrics {
rule check-node-status {
keys [ condition node ];
synopsis "";
description "Checks node status";
sensor node-status {
description "Checks node status";
server-monitoring {
sensor-name /kube/node;
frequency 60s;
}
}
field condition {
sensor node-status {
path condition;
}
type string;
description "Node condition";
}
field node {
sensor node-status {
path node;
}
type string;
description "Node name";
}
field node_status {
constant {
value "{{value}}";
}
type integer;
description "Field to check condition";
}
field status {
sensor node-status {
path status;
}
type string;
description "Status of the node";
}
trigger node-status {
frequency 1offset;
term available {
when {
matches-with "$status" false {
ignore-case;
}
}
then {
status {
color green;
message "$condition for $node is $status";
}
}
}
term notavailable {
then {
status {
color red;
message "$condition for $node is $status.";
}
}
}
}
variable value {
value 0;
description "Variable to match true(0) condition.";
type int;
}
}
}
} kube-metrics.playbook
healthbot {
playbook kube-metrics {
rules [ kube-metrics/check-daemonset-status kube-metrics/check-deployment-status-replicas kube-metrics/check-node-status kube-metrics/check-pod-container-restarts kube-metrics/check-pod-init-container-status kube-metrics/check-deployment-status-condition ];
description "Rules to check for kube-metrics ";
synopsis "Rules to check for kube-metrics ";
}
} iAgent (CLI/NETCONF)
For all the benefits of the ’push’ data collection methods, some operational and state information is available only through CLI/VTY commands. iAgent fills this gap by taking advantage of NETCONF/SSH functionality to provide Paragon Insights with the ability to connect to a device, run commands, and capture the output.
iAgent sensors use NETCONF/SSH and YAML-based PyEZ tables and views to fetch the necessary data. Both structured (XML) and unstructured (VTY commands and CLI output) data is supported.
With iAgent, the Paragon Insights server initiates SSH requests over any available network interface, whether in-band or out-of band; and the device responds (when properly configured) with the requested data.
At minimum, iAgent (NETCONF) requires:
Junos OS Version: 11.4 or later
Minimum required device configuration:
set system services netconf ssh
Starting in HealthBot Release 3.1.0, iAgent functionality is extended to third party devices. When adding a device, you can choose Other Vendor from the Vendor pull-down. This adds the Vendor Name text field below the Vendor pull-down. Then you fill in the iAgent Port Number, Vendor Name, and OS name fields highlighted in Figure 1 to allow iAgent connections to non-Juniper devices.
Refer to vendor documentation to understand how to configure third-party vendor devices to allow these connections.
Using Netmiko, Paragon Insights makes persistent SSH connections over the selected port to the third-party device. To gather device information, Paragon Insights sends CLI commands over SSH and receives string blobs back as output. The string blobs are then parsed through TextFSM, using ntc-templates into JSON format and then stored in the database. Default templates are located at /srv/salt/_textfsm. A repository of ntc-templates for network devices is available here: NTC Templates. For advanced users who need a template which does not exist, you can create your own templates and upload them to Paragon Insights using the Upload Rule Files button on the Configuration > Rules page. User defined templates are stored at /jfit/_textfsm. The files must end with the .textfsm suffix.
TextFSM is integrated into PyEZ’s table/view feature which is an integral part of iAgent.
- Example: PaloAlto Panos– Show Running Security Policy
- Outbound SSH (Device-Initiated)
- iAgent - vCenter/ESXi Server Monitor
Example: PaloAlto Panos– Show Running Security Policy
To see the running security policy on a Panos device, we need to:
Define a table/view for it
Gather the output by sending the needed CLI to the device over SSH
Generate JSON to store in Paragon Insights database
Define PyEZ Table/View
We need to define a PyEZ table that is used by the iAgent rule
assigned to the Panos device. The following table definition lacks
a view definition. Because of this, the entire output from the show running security-policy ends up getting stored in the
database after processing.
--- PanosSecurityPolicyTable: command: show running security-policy platform: paloalto_panos key: NAME use_textfsm: True
(Optional) To store only a portion of the received data in Paragon Insights, you can define a view in the same file. The view tells Paragon Insights which fields to pay attention to.
---
PanosSecurityPolicyTable:
command: show running security-policy
platform: paloalto_panos
key: NAME
use_textfsm: True
view: TrafficAndActionView
TrafficAndActionView:
fields:
source: SOURCE
destination: DESTINATION
application_service: APPLICATION_SERVICE
action: ACTION
Gather Output from Device
Using an iAgent rule that references the PyEZ table (or table/view)
defined above, Paragon Insights sends the command show running
security-policy to the device which produces the following output:
"intrazone-default; index: 1" {
from any;
source any;
source-region none;
to any;
destination any;
destination-region none;
category any;
application/service 0:any/any/any/any;
action allow;
icmp-unreachable: no
terminal yes;
type intrazone;
}
"interzone-default; index: 2" {
from any;
source any;
source-region none;
to any;
destination any;
destination-region none;
category any;
application/service 0:any/any/any/any;
action deny;
icmp-unreachable: no
terminal yes;
type interzone;
}
dynamic url: no
Generate JSON for Use in Paragon Insights Database
Since the device configuration specifies Palo Alto Networks as the vendor and Panos OS as the operating system, the TextFSM template used for this example would look like this:
Value Key,Filldown NAME (.*?)
Value Required FROM (\S+)
Value SOURCE (\S+)
Value SOURCE_REGION (\S+)
Value TO (\S+)
Value DESTINATION ([\S+\s+]+)
Value DESTINATION_REGION (\S+)
Value USER (\S+)
Value CATEGORY (\S+)
Value APPLICATION_SERVICE ([\S+\s+]+)
Value ACTION (\S+)
Value ICMP_UNREACHABLE (\S+)
Value TERMINAL (\S+)
Value TYPE (\S+)
Start
^${NAME}\s+\{
^\s+from\s+${FROM};
^\s+source\s+${SOURCE};
^\s+source-region\s+${SOURCE_REGION};
^\s+to\s+${TO};
^\s+destination\s+${DESTINATION};
^\s+destination-region\s+${DESTINATION_REGION};
^\s+user\s+${USER};
^\s+category\s+${CATEGORY};
^\s+application/service\s+${APPLICATION_SERVICE};
^\s+action\s+${ACTION};
^\s+icmp-unreachable:\s+${ICMP_UNREACHABLE}
^\s+terminal\s+${TERMINAL};
^\s+type\s+${TYPE};
^} -> Record
When the template above is used by Paragon Insights to parse the output shown previously, the resulting JSON looks like:
{'"interzone-default; index: 2"': {'ACTION': 'deny',
'APPLICATION_SERVICE': '0:any/any/any/any',
'CATEGORY': 'any',
'DESTINATION': 'any',
'DESTINATION_REGION': 'none',
'FROM': 'any',
'ICMP_UNREACHABLE': 'no',
'SOURCE': 'any',
'SOURCE_REGION': 'none',
'TERMINAL': 'yes',
'TO': 'any',
'TYPE': 'interzone',
'USER': ''},
'"intrazone-default; index: 1"': {'ACTION': 'allow',
'APPLICATION_SERVICE': '0:any/any/any/any',
'CATEGORY': 'any',
'DESTINATION': 'any',
'DESTINATION_REGION': 'none',
'FROM': 'any',
'ICMP_UNREACHABLE': 'no',
'SOURCE': 'any',
'SOURCE_REGION': 'none',
'TERMINAL': 'yes',
'TO': 'any',
'TYPE': 'intrazone',
'USER': ''}}
Outbound SSH (Device-Initiated)
Starting with Release 3.2.0, HealthBot can use iAgent connections that are device-initiated using outbound SSH. This configuration makes Paragon Insights act as the client to the device making the connection. This type of connection is useful in environments in which the remote devices cannot accept incoming connections. All existing iAgent rules can be used when outbound SSH is configured in Junos OS devices.
In the 3.2.0 release, outbound SSH is only supported on Junos OS devices.
At minimum, iAgent (outbound-ssh) requires:
Junos OS Version: 11.4 or later
Minimum required device configuration:
set system services outbound-ssh client client_name device-id <device-name-in-healthbot> set system services outbound-ssh client client_name keep-alive retry 30 set system services outbound-ssh client client_name keep-alive timeout 35 set system services outbound-ssh client client_name services netconf set system services outbound-ssh client client_name 10.1x.x0.1 port 2222
Note:In the configuration above, client_name can be anything that makes sense. The IP address and port shown are examples representing Paragon Insights’s IP address and a unique port number used for the devices in 1 device-group.
Outbound SSH is disabled by default and can be enabled at the device-group level. Once enabled, all devices in the group use outbound SSH unless specifically configured not to. Users can disable outbound SSH in a device through management CLI.
When you enable outbound SSH within the device-group, you must select a TCP port number over which all the member devices initiate their NETCONF connections to Paragon Insights. This port must be unique across the Paragon Insights installation. Figure 2shows the outbound SSH section of the add/edit device group window.
To disable an individual device in a device-group from using outbound SSH, you must edit the device and select disable in the outbound SSH configuration section. If you later change your mind and want that device to use outbound SSH, edit the device and set outbound SSH to reset.Figure 3 is an edited (shortened) device add/edit window with the outbound SSH section shown.
- Configure TCP Port for Outbound SSH in Ingest
- Connect a Device in Multiple Device Groups via Outbound SSH
Configure TCP Port for Outbound SSH in Ingest
When you configure outbound SSH connections for device groups, you must configure TCP ports for each device group. To avoid opening multiple TCP ports, Paragon Insights 4.1.0 Release supports configuration of a single TCP port as ingest configuration for all outbound SSH connections across device groups that use iAgent (NETCONF).
To configure iAgent (NETCONF) TCP port in ingest:
Go to Settings > Ingest.
Select the SSH tab on the Ingest Settings page.
In the OutboundSSH page, enter the TCP port number.
You can use the toggle button to enable or disable the Port field.
Do one of the following:
Click Save and Deploy to save and commit the configuration in your existing network configurations.
Click Save to only save the configuration and not deploy it with the existing network.
Note:If you configure outbound SSH port for iAgent in device groups, that configuration takes precedence over the ingest configuration.
Connect a Device in Multiple Device Groups via Outbound SSH
You can connect a device that is managed in different device groups through outbound SSH by configuring multiple clients, where each client has the same port. In this case, you must create as many copies of the device as there are device groups. Each device must have the same port number.
As an example, consider device r0 (10.1.1.1) configured for device groups dg1 and dg2. To connect 10.1.1.1 to both device groups via the same outbound SSH port, you can create one more device r1 (10.1.1.1) with the same IP and deploy it in dg2.
You must configure Paragon Insights for these ports in the respective device-groups. Figure 4 is an example device group configuration.
Using the following sample client configurations, device 10.1.1.1 can connect to two device groups using two outbound SSH clients with the same port.
set system services outbound-ssh client outbound-ssh1 device-id r0 set system services outbound-ssh client outbound-ssh1 10.1.1.1 port 2020 set system services outbound-ssh client outbound-ssh2 device-id r1 set system services outbound-ssh client outbound-ssh2 10.1.1.1 port 2020
The 10.1.1.1 in the example denotes Paragon Insights (host) IP address.
iAgent - vCenter/ESXi Server Monitor
Since release 2.0.0, Paragon Insights has supported user-defined functions (UDFs) within fields. A user-defined function relies on python tables and views, and YAML configuration files to allow a user to define their own functions for processing telemetry data from Junos OS devices.
Starting with Release 3.2.0, Paragon Insights can use UDFs to process streamed data from VMWare’s vCenter and ESXi server products.
We implement this feature by making use of VMWare’s PyVmomi API and persistent SaltStack connections to that API. We use UDFs to process the vCenter/ESXi server data because:
these servers do not respond to RPC with XML formatted text.
it is extremely difficult to construct a table and view from the response data that these servers provide
SNMP
SNMP is a widely known and accepted network management protocol that many network device manufacturers, including Juniper Networks, provide for use with their devices. It is a polling type protocol where network devices that are properly configured make configuration, diagnostic, and event information available to collectors, which must also be properly configured and authenticated. The collectors poll devices by sending specifically structured requests, called get requests, to retrieve data.
Paragon Insights supports SNMP as a sensor type, using standard get requests to gather statistics from the device. Paragon Insights makes requests over any available interface, whether in-band or out-of-band, and the device responds (when configured) with the requested data.
For information about SNMP as used on Junos OS devices, see Understanding SNMP Implementation in Junos OS.
The sections below delve deeper into SNMP ingest configuration and all of the steps needed for Paragon Insights to successfully ingest SNMP data from a device or devices in a device group.
- SNMP in Paragon Insights
- Example: Creating a Rule using SNMP Ingest
- CONFIGURE NETWORK DEVICES
- CREATE RULE, APPLY PLAYBOOK
- Monitor the Devices
SNMP in Paragon Insights
Paragon Insights supports three methods of collecting telemetry data using SNMP. The ingest, also known as request-response, is a pull mode method in which Paragon Insights requests for telemetry data from the devices. The trap and inform notifications are push mode methods in which the devices notify Paragon Insights about key performance indicator events that prevents the devices from functioning as expected.
Paragon Insights Release 4.0.0 supports SNMPv3 alongside the current SNMPv2c as an ingest method. Users with sp-admin role can select any version of SNMP in the Paragon Insights GUI.
SNMPv3 ingest provides you with an option to set authentication and privacy credentials to leverage the following features:
Authentication — Identifies and verifies the origin of an SNMPv3 message.
Privacy — Prevents packet analyzers from snooping the content of messages by encrypting them.
Integrity — Ensures that the content of SNMP messages is not altered while in transit without authorization.
Table 2 lists the supported authentication and privacy algorithms in SNMPv3 protocol.
Feature |
Algorithm |
|---|---|
Supported authentication algorithms |
MD5 SHA SHA224 SHA256 SHA384 SHA512 |
Supported privacy algorithms |
DES AES AES192 AES256 AES192C AES256C |
Paragon Insights also supports scalar object instances along with tabular objects in SNMP.
The SNMP object can be scalar, tabular, or a combination of both in rules.
When you create a rule using SNMP ingest, you can add:
Only scalar fields.
A combination of tabular and scalar fields.
A tabular column along with the index queried as a scalar object.
A tabular column queried as a scalar comes with the limitation that the index number does not refer to the same Object across all the devices when you configure the tabular field in rule.
For example, IF-MIB::ifAdminStatus.16. The ifAdminStatus is a column in IF-MIB table and IF-MIB::ifAdminStatus.16 refers to the table column with index 16.
Only tabular fields.
A scalar object is identified by its MIB name (for example, JUNIPER-MIB::scalarObjectName) or as an OID.
Paragon Insights validates a given scalar by checking the MAX-ACCESS property in the MIB definition.
If you find MAX-ACCESS in the MIB definition set to read-only, read-create, or read-write, then that object can be queried as a scalar.
When you query a table column with index as a scalar, you must
enter the complete path to the scalar object as MIB-Name::table column name:index number.
For example:
IF-MIB::ifInOctets:16 You can enter complete paths for more than one scalar object in the same query.
Configure a Rule Using SNMP Scalar Fields
The following example configuration of a rule has an SNMP tabular column as a scalar field and a second field with a scalar not indexed in the SNMP table.
To configure a rule with scalar fields:
Go to Configuration > Rules.
Click +Add Rule on top of the Rules page.
Fill in the topic and rule name, description, and synopsis.
By default, the new rule is saved in topic external unless you specify a topic name here.
Click +Add Sensor and enter a name for the sensor.
Select SNMP as Sensor Type and 30s as Frequency.
Click Add Scalar and enter IF-MIB::ifNumber in the scalar field.
You can also enter the OID of the scalar object.
Click Add Scalar again and enter IF-MIB::ifAdminStatus.16 in the scalar field.
Note:The index number is different from one device to another and from one system to another.
Before you configure a rule for a device, get the index number of the scalar object in that device.
For more information, see SNMP index number.
Do one of the following:
Save — Save your configuration changes but do not deploy the updated configuration. You can use this option when, for example, you are making several changes and want to deploy all your updates at the same time later.
Save & Deploy — Save the rule configuration in the GUI and deploy the configuration on your production environment. The ingest starts collecting telemetry data based on the configuration changes.
SNMP Ingest Configurations
The SNMPv3 ingest can be set at the device or device group level, with device configuration taking precedence if the ingest is configured at both levels. The configuration of SNMPv3 and SNMPv2c is mutually exclusive.
If a device is not configured for SNMP ingest, Paragon Insights uses SNMP v2c with SNMP Community set to public as the default settings.
In Paragon Insights, the SNMPv2c and SNMPv3 ingest and trap configurations share the same workflow.
To configure SNMP ingest at the device level:
Click the Configuration > Device option in the left navigation bar.
Click the add device button (+).
Enter the necessary values in the text boxes and select the appropriate options for the device.
The following table describes the attributes in the Add a Device window:
Attributes
Description
Name
Name of the device. Default is hostname. (Required)
Hostname / IP Address / Range
Hostname or IP address of a single device. If you are providing a range of IP addresses, enter the IP address for the device that marks the start and end of the address range. (Required)
System ID to use for JTI
Unique system identifier required for JTI native sensors. Junos devices use the following format: <host_name>:<jti_ip_address>
When a device has dual routing engines (REs), it might send different system IDs depending on which RE is primary. You can use a regular expression to match both system IDs.
Flow Source IPs
Enter the IP address(es) that this device uses to send NetFlow data.
OpenConfig Port Number
Port number required for JTI OpenConfig sensors. The default value is 32767.
iAgent Port Number
Port number required for iAgent. The default value is 830.
Vendor
Lists the vendor or supplier of the device you are using.
The operating system you can select from the OS drop-down list depends on the vendor you select from the Vendor drop-down list.
The following are the list of options you can choose from.
Select Juniper from the vendor drop-down list to select either Junos or Junos Evolved operating systems from the OS drop-down list.
Select CISCO from the Vendor drop-down list to select either IOSXR or NXOS operating systems from the OS drop-down list.
With Paragon Insights Release 4.0.0, NXOS OS is also supported.
Note:If you plan to use Cisco IOS XR devices, you must first configure the telemetry. For more information, see Paragon Insights Installation Requirements
Select Arista from the Vendor drop-down list to select EOS operating system from the OS drop-down list.
Select Paloalto from the Vendor drop-down list to select PANOS operating system from the OS drop-down list.
Select Linux from the Vendor drop-down list and you can enter the name of the operating system in the OS field.
If you select Other Vendor, the Vendor Name and OS Name fields are enabled. You can enter the name of the vendor of your choice in the Vendor Name field and the corresponding operating system for the vendor in the OS field.
Consider the following example. If the operating system of a vendor (listed in the Vendor drop-down list) is not listed (in the OS drop-down list), you can select the Other Vendor option to enter name of the vendor and operating system of your choice.
Starting with Release 4.0.0, Paragon Insights supports Arista Networks, Paloalto Networks, and Linux vendors.
Timezone
Timezone for this device, specified as + or -hh:mm. For example, +07:00
Syslog Source IPs
List of IP addresses for the device sending syslog messages to Paragon Insights. For example,
10.10.10.23,192.168.10.100.Syslog Hostnames
List of hostnames for the device sending syslog messages to Paragon Insights. For example, router1.example.com.
SNMP
SNMP Port Number
Port number required for SNMP ingest (request-response) messages. The port number is set to the standard value of 161.
SNMP Version
Select either v2c or v3 in the drop-down menu.
SNMP Community
Enter an SNMP Community string for SNMPv2c ingest.
In SNMPv2c, Community string is used to verify the authenticity of the ingest (request-response) message issued by the SNMP agent (devices such as routers, switches, servers, and so on).
SNMPv3 Username
Enter a username for SNMPv3 ingest (request-response).
Authentication None
This field appears if you selected v3 in SNMP Version field.
Enable this option if you want to set SNMPv3 authentication to None.
Privacy None
This field appears if you selected v3 in SNMP Version field.
Enable this option if you want to set SNMPv3 privacy protocol to None.
SNMPv3 Authentication Protocol
This field appears if you selected v3 in SNMP Version field and disabled Authentication None.
Select an authentication protocol from the drop-down menu.
SNMP authentication protocol hashes the SNMP username with the passphrase you enter. The hashed output is sent along with the ingest message. Insights again hashes the username with the passphrase you entered for authentication. If the output matches, the ingest message is further processed.
SNMPv3 Authentication Passphrase
This field appears if you selected v3 in SNMP Version field and disabled Privacy None.
Enter a passphrase for SNMPv3 authentication.
SNMPv3 Privacy Protocol
Select a privacy protocol from the drop-down menu.
Privacy algorithm encrypts the ingest message with the protocol passphrase so that the message cannot be read by an unauthorized application in the network.
SNMPv3 Privacy Passphrase
This field appears if you selected v3 in SNMP Version field and disabled Privacy None.
Enter a passphrase to encrypt the ingest message.
Authentication (Required either here or at Device Group level)
Password
Username Authentication username.
Password Authentication password.
SSL
Server Common Name Server name protected by the SSL certificate.
CA Profile* Choose the applicable CA profile(s) from the drop-down list.
Local Certificate* Choose the applicable local certificate profile(s) from the drop-down list.
SSH
SSH Key Profile* Choose the applicable SSH key profile(s) from the drop-down list.
Username Authentication username.
*To edit or view details about saved security profiles, go to the Security page under the Settings menu in the left navigation bar.
Click Save to save the configuration or click Save and Deploy to save and deploy the configuration. For information on how to use the Devices table, see Monitor Device and Network Health.
To configure SNMP ingest at the device-group level:
Click the Configuration > Device Group option in the left-nav bar.
Click the add group button (+).
Enter the necessary values in the text boxes and select the appropriate options for the device group.
The following table describes the attributes in the Add a Device Group window:
Attributes
Description
Name
Name of the device group. (Required)
Description
Description for the device group.
Devices
Add devices to the device group from the drop-down list. (Required)
Starting in Paragon Insights 4.0.0, you can add more than 50 devices per device group. However, the actual scale of the number of devices you can add depends on the available system resources.
For example, consider that you want to create a device group of 120 devices. In releases earlier than release 4.0.0, it is recommended that you create three device groups of 50, 50, and 20 devices respectively. With Paragon Insights Release 4.0.0, you just create one device group.
Native Ports
(Native GPB sensors only) List the port numbers on which the Junos Telemetry Interface (JTI) native protocol buffers connections are established.
Flow Ports
(NetFlow sensors only) List the port numbers on which the NetFlow data is received by Paragon Insights. The port numbers must be unique across the entire Paragon Insights installation.
Syslog Ports
Specify the UDP port(s) on which syslog messages are received by Paragon Insights.
Retention Policy
Select a retention policy from the drop-down list for time series data used by root cause anaylsis (RCA). By default, the retention policy is 7 days.
Reports
In the Reports field, select one or more health report profile names from the drop-down list to generate reports fo the device group. Reports include alarm statistics, device health data, as well as device-specific information (such as hardware and software specifications).
To edit or view details about saved health report profiles, go to the System page under the Settings menu in the left-nav bar. The report profiles are listed under Report Settings.
For more information, see Alerts and Notifications.
SNMP
SNMP Port Number
Port number required for SNMP notifications. The port number is set to the standard value of 161.
SNMP Version
Select either v2c or v3 in the drop-down menu.
If you select v2c, the SNMP Community name field appears. The string used in v2c authentication is set to public by default. It is recommended that users change the community string.
If you select v3, you are given an option to set username, authentication and privacy methods, and authentication and privacy secrets.
SNMP Community
Enter an SNMP Community string for SNMPv2c ingest.
In SNMPv2c, Community string is used to verify the authenticity of the trap message issued by the SNMP agent.
SNMPv3 Username
Enter a username for SNMPv3 ingest messages.
Authentication None
This field appears if you selected v3 in SNMP Version field.
Enable this option on if you want to set SNMPv3 authentication to None.
Privacy None
This field appears if you selected v3 in SNMP Version field.
Enable this option on if you want to set SNMPv3 privacy protocol to None.
SNMPv3 Authentication Protocol
This field appears if you selected v3 in SNMP Version field and disabled Authentication None.
Select an authentication protocol from the drop-down menu.
SNMP authentication protocol hashes the SNMP username with the passphrase you enter. The hashed output is sent along with the trap notification message. Paragon Insights again hashes the username with the passphrase you entered for authentication. If the output matches, the trap notification is further processed.
SNMPv3 Authentication Passphrase
This field appears if you selected v3 in SNMP Version field and disabled Privacy None.
Enter a passphrase to authenticate SNMPv3 ingest messages.
SNMPv3 Privacy Protocol
Select a privacy protocol from the drop-down menu.
Privacy algorithm encrypts the ingest message with the protocol passphrase so that the message cannot be read by an unauthorized application in the network.
SNMPv3 Privacy Passphrase
This field appears if you selected v3 in SNMP Version field and disabled Privacy None.
Enter a passphrase to encrypt the trap notification.
Summarization
To improve the performance and disk space utilization of the Paragon Insights time series database, you can configure data summarization methods to summarize the raw data collected by Paragon Insights. Use these fields to configure data summarization:
Time Span The time span (in minutes) for which you want to group the data points for data summarization.
Summarization Profiles Choose the data summarization profiles from the drop-down list for which you want to apply to the ingest data. To edit or view details about saved data summarization profiles, go to the Data Summarization Profiles page under the Settings menu in the left-nav bar.
For more information, see Configure Data Summarization.
Ingest Frequency
Select existing Ingest Frequency Profiles to override rule or sensor frequency settings.
Authentication(Required here or at Device level)
Password
Username Authentication user name.
Password Authentication password.
SSL
Server Common Name Server name protected by the SSL certificate.
CA Profile* Choose the applicable CA profile(s) from the drop-down list.
Local Certificate* Choose the applicable local certificate profile(s) from the drop-down list.
SSH
SSH Key Profile* Choose the applicable SSH key profile(s) from the drop-down list.
Username Authentication username.
Notifications
You can use the Alarm Manager feature to organize, track, and manage KPI event alarm notifications received from Paragon Insights devices.
To receive Paragon Insights alarm notifications for KPI events that have occurred on your devices, you must first configure the notification delivery method for each KPI event severity level (Major, Minor, and Normal). Select the delivery method from the drop-down lists.
To edit or view details about saved delivery method profiles, go to the System page under the Settings menu in the left-nav bar. The delivery method profiles are listed under Notification Settings.
For more information, see Alerts and Notifications.
Logging Configuration
You can collect different severity levels of logs for the running Paragon Insights services of a device group. Paragon Insights Release 4.0.0 supports log collection for SNMP notification.
Use these fields to configure which log levels to collect:
Global Log Level From the drop-down list, select the level of the log messages that you want to collect for every running Paragon Insights service for the device group. The level is set to error by default.
Log Level for specific services Select the log level from the drop-down list for any specific service that you want to configure differently from the Global Log Level setting. The log level that you select for a specific service takes precedence over the Global Log Level setting.
For more information, see Logs for Paragon Insights Services.
Publish
You can configure Paragon Insights to publish Paragon Insights sensor and field data for a specific device group:
Destinations Select the publishing profiles that define the notification type requirements (such as authentication parameters) for publishing the data.
To edit or view details about saved publishing profiles, go to the System page under the Settings menu in the left-nav bar. The publishing profiles are listed under Notification Settings.
Field Select the Paragon Insights rule topic and rule name pairs that contain the field data you want to publish.
Sensor (Device group only) Select the sensor paths or YAML tables that contain the sensor data you want to publish. No sensor data is published by default.
*To edit or view details about saved security profiles, go to the Security page under the Settings menu in the left-nav bar.
Click Save to save the configuration or click Save and Deploy to save and deploy the configuration. For information on how to use the device group cards, see Monitor Device and Network Health.
Example: Creating a Rule using SNMP Ingest
To illustrate how to configure and use an SNMP sensor, consider a scenario where you want to:
Monitor Routing Engine CPU, CPU average, and memory utilization for a device, using SNMP data
Create a rule with triggers that indicate when utilization for any of the above elements goes above 80%
To implement this scenario, you will need to complete the following activities:
The workflow is as follows:
CONFIGURE NETWORK DEVICES
This example assumes you have already added your devices into Paragon Insights and assigned them to a device group.
If not already done, configure your network device(s) to accept SNMP ingest in Paragon Insights. See SNMP in Paragon Insights for steps to configure SNMP ingest.
CREATE RULE, APPLY PLAYBOOK
- Configure a Rule Using an SNMP Sensor
- Add the Rule to a Playbook
- Apply the playbook to a device group
Configure a Rule Using an SNMP Sensor
You can now create a rule using SNMP as the sensor.
This rule includes multiple elements, as shown below:
An SNMP sensor to ingest data
Five fields extracting specific SNMP data of interest:
CPU utilization, memory utilization
CPU utilization averages - 1min, 5min, 15min
A field representing a static value, used as a threshold
Value provided by a variable
A field representing a description
Value provided by a variable; extracted from the SNMP messages
Five triggers, indicating when CPU, CPU average, and memory utilization is higher than the threshold value
- Click the + Add sensor button and enter the
following parameters to configure the sensor, system-cpu-memory:
Name is user-defined
The sensor is using the Juniper SNMP MIB table jnxOperatingTable
Paragon Insights polls the device group for table data every 60 seconds
- Now move to the Variables tab, click the + Add variable button and enter the following parameters to configure the first
variable, comp-name:
Matches any string that includes “Routing Engine”
Referenced later in field description
- Click the + Add variable buttononce more and
enter the following parameters to configure the second variable, static-threshold:
Represents a (default) static value of “80”; in this case, 80%
Referenced later in field threshold
- Now move to the Fields tab, click the + Add field button and enter the following parameters to configure the first
field, cpu-15min-avg:
Field names are user-defined
Extracts jnxOperating15MinLoadAvg value from SNMP table configured in the sensor
jnxOperating15MinLoadAvg - CPU Load Average (as a % value) over the last 15 minutes
- Click the + Add field button again and enter
the following parameters to configure the second field, cpu-1min-avg:
Extracts jnxOperating1MinLoadAvg value from SNMP table
jnxOperating1MinLoadAvg - CPU Load Average (as a % value) over the last 1 minute
- Click the + Add field button again and enter
the following parameters to configure the third field, cpu-5min-avg:
Extracts jnxOperating5MinLoadAvg value from SNMP table
jnxOperating5MinLoadAvg - CPU Load Average (as a % value) over the last 5 minutes
- Click the + Add field button again and enter
the following parameters to configure the fourth field, description:
Extracts jnxOperatingDescr value from SNMP table
jnxOperatingDescr - name or description; for example, ”Routing Engine 0”, “FPC 0”, etc.
The expression references the variable comp-name; filters the data further to retain only the values that include the string “Routing Engine”
Matching values will act as keys; each key gets a colored block in device health view
- Click the + Add field button again and enter
the following parameters to configure the fifth field, system-buffer-memory:
Extracts jnxOperatingBuffer value from SNMP table
jnxOperatingBuffer - buffer pool utilization (as a % value)
- Click the + Add field button again and enter
the following parameters to configure the sixth field, system-cpu:
Extracts jnxOperatingCPU value from SNMP table
jnxOperatingCPU - CPU utilization (as a % value)
- Click the + Add field button once more and
enter the following parameters to configure the seventh field, threshold:
The expression references the variable static-threshold, giving this field the (default) integer value “80”
Referenced later in triggers
- Now move to the Triggers tab, click the + Add trigger button and enter the following parameters to configure the first
trigger, system-buffer:
Trigger names are user-defined
Trigger logic runs every 90 seconds
Evaluate terms in sequence; when a term’s conditions are met, show its color and message on the device health pages
When system memory buffer utilization (the value in field system-buffer-memory) is greater than 80 (the value in field threshold), set color to red and show related message
Otherwise, set color to green and show related message
- Click the click the + Add trigger button again
and enter the following parameters to configure the second trigger, system-cpu:
Trigger logic runs every 90 seconds
When CPU utilization (the value in field system-cpu) is greater than 80 (the value in field threshold), set color to red and show related message
Otherwise, set color to green and show related message
- Click the click the + Add trigger button again
and enter the following parameters to configure the third trigger, system-cpu-15min-average:
Trigger logic runs every 90 seconds
When CPU 15min utilization average (the value in field cpu-15min-avg) is greater than or equal to 80 (the value in field threshold), set color to red and show related message
Otherwise, set color to green and show related message
- Click the click the + Add trigger button again
and enter the following parameters to configure the fourth trigger, system-cpu-1min-average:
Trigger logic runs every 90 seconds
When CPU 1min utilization average (the value in field cpu-1min-avg) is greater than or equal to 80 (the value in field threshold), set color to red and show related message
Otherwise, set color to green and show related message
- Click the click the + Add trigger button once
more and enter the following parameters to configure the fifth trigger,
called system-cpu-5min-average:
Trigger logic runs every 90 seconds
When CPU 5min utilization average (the value in field cpu-5min-avg) is greater than or equal to 80 (the value in field threshold), set color to red and show related message
Otherwise, set color to green and show related message
Add the Rule to a Playbook
With the rule created, you can now add it to a playbook. For this example, create a new playbook to hold the new rule.
- On the page that appears, enter the following parameters:
Apply the playbook to a device group
To make use of the playbook, apply it to a device group.
- On the page that appears:
Enter a playbook instance name
Select the desired device group
(Optional) If desired, you can adjust the variables for this playbook instance to use different values than the defaults configured in the rule
Click Run Instance
Monitor the Devices
With the playbook applied, you can begin to monitor the devices.
Click Monitor > Device Group Health in the left-nav bar.
Select the device group to which you applied the playbook from the Device Group pull-down menu.
Select one or more of the devices to monitor.
In the Tile View, hover your mouse over one of the external tiles.
external is the topic name under which the rule was created
Each colored block represents a key and its related values
The mouse-over window shows information related to the given key, with the triggers listed inside
In the Table View, try out the various filters and sorting options.
Each trigger is listed as a KPI
