Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

 
 

Extensible Telemetry Guide

Extensible Telemetry Overview

Install Apstra device drivers and telemetry collectors to collect additional telemetry that can be used in analytics probes. The device drivers enable Apstra to connect to a NOS and collect telemetry. Apstra ships with drivers for EOS, NX-OS, Ubuntu, and CentOS. To add a driver for an operating system not listed here, contact Juniper Support.

Telemetry collectors are Python modules that help collect extended telemetry. The following sections describe the pipeline for creating telemetry collectors and extending Apstra with new collectors. You need familiarity with Python to be able to develop collectors.

Set Up Development Environment

To get access to telemetry collectors (which are housed in the aos_developer_sdk repository) contact Juniper Support. Contribute any new collectors that you develop to the repository.

To keep your system environment intact, we recommend that you use a virtual environment to isolate the required Python packages (for development and testing). You can download the base development environment, aos_developer_sdk.run, from https://support.juniper.net/support/downloads/?p=apstra/. To load the environment, execute:

This command loads the aos_developer_sdk Docker image. After the image load is complete, the command to start the environment is printed. Start the container environment as specified by the command. To install the dependencies, execute:

The environment is now set up for developing and testing the collectors. Apstra SDK packages, such as device drivers and REST client, are also installed in the environment.

Develop Collector

To develop a telemetry collector, specify the following in order.

  1. Service for which the collector is developed - Identify what the service is. For example, the service could be to collect received and transmitted bytes from the switch interfaces. Identify a name for the service. Using service names that are reserved for built-in services (ARP, BGP, interface, hostname, route, MAC, XCVR, LAG, MLAG) is prohibited.
  2. The schema of the data provided to Apstra - Identify how the collector output is to be structured. A collection of key-value pairs should be posted to Apstra. Identify what each item is, that is, what is the key/value syntactically and semantically. For the above mentioned example, key is a string that identifies the interface name. The value is a JSON string, with the JSON having two keys 'rx' and 'tx' both having an integer value.
  3. Network Operating System (NOS) for which the collector is developed - The collector plugins are NOS-specific. Before writing a collector, identify the NOS(s) for which collector(s) are required.
  4. How the required data can be obtained from the device - Identify the commands that can be used in the device to retrieve the required information. For example, 'show interfaces' command gives received and transmitted bytes from an Arista EOS device.
  5. Storage Schema Path - The type of key and value in each item determines the storage schema path. The type of collector selected determines the storage schema for the application. The storage schema defines the high level structure of the data returned by the service. The storage schema path for your collector can be determined using the following table:
    Table 1: Determining Storage Schema Path
    Key Type Value Type Storage Schema Path
    String String aos.sdk.telemetry.schemas.generic
    String Dict aos.sdk.telemetry.schemas.generic
    Dict String aos.sdk.telemetry.schemas.iba_string_data
    Dict Integer aos.sdk.telemetry.schemas.iba_integer_data
  6. Application Schema - Application schema defines the schema for each item posted to the framework. Application schema is expressed using draft 4 version of json schema. Each item is comprised of a key and value. The following table specifies two sample items.
    Table 2: Sample item with its storage schema path
    Storage Schema Path Sample Item
    aos.sdk.telemetry.schemas.generic
    {
        "identity": "eth0",
        "value": "up",
    }
    aos.sdk.telemetry.schemas.iba_string_data
    {
        "key": {
            "source_ip": "10.1.1.1",
            "dest_ip": "10.1.1.2",
        },
        "value": "up",
    }
    Note:

    * An item returned by collectors with generic storage schema should specify the key value using the key 'identity' and the value using the key 'value'.

    * An item returned by collectors with IBA-based schemas should specify the key value using the key 'key' and the value using the key 'value'.

    Using this information, you can write the JSON schema. The following table maps the sample item specified above to its corresponding JSON schema.

    Table 3: Sample Application Schema
    Sample Item Application Schema
    {
        "identity": "eth0",
        "value": "up",
    }
    {
        "type": "object",
        "properties": {
            "identity": {
                "type": "string",
            },
            "value": {
                "type": "string",
            }
        }
    }
    {
    {
        "key": {
            "source_ip": "10.1.1.1",
            "dest_ip": "10.1.1.2",
        },
        "value": "up",
    }
    {
        "type": "object",
        "properties": {
            "key": {
                "type": "object",
                "properties": {
                    "source_ip": {
                        "type": "string",
                        "format": "ipv4"
                    },
                    "dest_ip": {
                        "type": "string",
                        "format": "ipv4"
                    },
                    "required": ["source_ip", "dest_ip"],
                }
            },
            "value": {
                "type": "string",
            }
        }
    }
    {

    You can specify more complex schema using the constructs available in JSON schema. Update the schema in the file aos_developer_sdk/aosstdcollectors/aosstdcollectors/json_schemas/<service_name>.json

    Note:

    As of Apstra version 4.0.1, you can import the service schema via the GUI.

Write Collector

Collector is a class that must derive from aos.sdk.system_agent.base_telemetry_collector.BaseTelemetryCollector. Override the collect method of the collector with the logic to:

Collect Data from Device

The device driver instance inside the collector provides methods to execute commands against the devices. For example, most Apstra device drivers provide methods get_json and get_text to execute commands and return the output.

Note:

The device drivers for aos_developer_sdk environment are preinstalled. You can explore the methods available to collect data. For example:

Parse Data

The collected data needs to be parsed and re-formatted per the Apstra framework and the service schema identified above. Collectors with generic storage schema follow the following structure:

Collectors with IBA-based schema follow the following structure:

In the structures above, the data posted has multiple items. Each item has a key and a value. For example, to post interface specific information, there would be an identity/key-value pair for each interface you want to post to the framework.

Note:

In the case when you want to use a third party package to parse data obtained from a device, list the Python package and version in the path.

<aos_developer_sdk>/aosstdcollectors/requirements_<NOS>.txt. The packages installed by the dependency do not conflict with packages that Apstra software uses. The Apstra-installed packages are available at /etc/aos/python_dependency.txt in the development environment.

Post Data to Framework

When data is collected and parsed as per the required schema, post the data to the framework. You can use the post_data method available in the collector. It accepts one argument, and that is the data that should be posted to the framework.

The folder aos_developer_sdk/aosstdcollectors/aosstdcollectors in the repository contains folders for each NOS. Add your collector to the folder that matches the NOS. Cumulus is no longer supported as of Apstra version 4.1.0, although this example remains for illustrative purposes. For example, to write a collector for Cumulus, add the collector to aos_developer_sdk/aosstdcollectors/aosstdcollectors/cumulus, and name the file after the service name. For example, if the service name is interface_in_out_bytes, then name the file interface_in_out_bytes.py.

In addition to defining the collector class, define the function collector_plugin in the collector file. The function takes one argument and returns the collector class that is implemented.

For example, a generic storage schema based collector looks like:

An IBA storage schema based collector looks like:

Unit Test Collector

The folder aos_developer_sdk/aosstdcollectors/test in the repository contains folders based on the NOS. Add your test to the folder that matches the NOS. For example, a test to a collector for Cumulus is added to aos_developer_sdk/aosstdcollectors/test/cumulus. We recommend that you name the unit test with the prefix test_.

The existing infrastructure implements a Pytest fixture collector_factory that is used to mock the device driver command response. The general flow for test development is as follows.

  1. Use the collector factory to get a collector instance and mocked Apstra framework. The collector factory takes the collector class that you have written as input.
  2. Mock the device response.
  3. Invoke collect method.
  4. Validate the data posted to the mocked Apstra framework.

For example, a test looks like:

To run the test, execute:

This command executes all the tests in the repository.

Package Collector

All the collectors are packaged based on the NOS. To generate all packages, execute make at aos_develop_sdk. You can find the build packages at aos_developer_sdk/dist. The packages build can be broadly classified as:

Package Description
Built-In Collector Packages These packages have the prefix aosstdcollectors_builtin_. To collect telemetry from a device per the reference design, Apstra requires services as listed in the <deviceblah> section. Built-In collector packages contain collectors for these services. The packages are generated on a per NOS basis.
Custom Collector Packages

These package have the prefix aosstdcollectors_custom_ in their names. The packages are generated on a per NOS basis. The package named aosstdcollectors_custom_<NOS>-0.1.0-py2-none-any.whl contains the developed collector.

Apstra SDK Device Driver Packages

These packages have a prefix apstra_devicedriver_. These packages are generated on a per NOS basis. Packages are generated for NOS that are not available by default in Apstra.

Upload Packages

If the built-in collector packages and the Apstra SDK Device Driver for your Device Operating System (NOS) were not provided with the Apstra software, you must upload them to the Apstra server.

If you are using an offbox solution and your NOS is not EOS, you must upload the built-in collector package.

Upload the package containing your collector(s) and assign them to a Device System Agent or System Agent Profile.

Use Telemetry Collector

Set up Telemetry Service Registry

The registry maps the service to its application schema and the storage schema path. You can manage the telemetry service registry with the REST endpoint /api/telemetry-service-registry. You can't enable the collector for a service without adding a registry entry for the particular service. The registry entry for a service cannot be modified while the service is in use.

Note:

When executing make, all application schemas are packaged together to a tar file (json_schemas.tgz) in the dist folder. With apstra-cli, you have the option of importing all the schemas in the .tgz file.

Start Collector

To start a service, use the POST API /api/systems/<system_id>/services with the following three arguments:

Arguments  
Input_data The data provided as input to the collector. Defaults to None.
Interval Interval at which to run the service. Defaults to 120 seconds.
Name Name of the service.
Note:

You can also manage collectors via the apstra-cli utility.

Delete Collector

To delete a service, use the DELETE API /api/systems/<system_id>/services/<service_name>.

Get Collected Data

To retrieve collected data, use the GET API /api/systems/<system_id>/services/<service_name>/data. Only the data collected in the last iteration is saved. Data does not persist over Apstra restart.

List Running Collector Services

To retrieve the list of services enabled on a device, use the GET API /api/systems/<system_id>/services.