Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

 
ON THIS PAGE
 

Common Options

SUMMARY This topic describes the common configuration options for Apstra Flow Data.

Licensing

SUMMARY The following sections describe the licensing API configuration options for Flow Data.

EF_JUNIPER_APSTRA_API_HOSTNAME

Use this setting to define the hostname or IP address where the Apstra server provides its API services. This setting is the same IP address or hostname you use to access the Apstra GUI. Note that this value must start with http:// or https://.

  • Example: http://localhost
  • Default value: ''

EF_JUNIPER_APSTRA_API_PORT

Use this setting to specify the port number on which the Apstra server exposes its API services. The most commonly used ports are port 80 and port 443.

  • Example: 80
  • Default value: ''

EF_JUNIPER_APSTRA_API_TLS_SKIP_VERIFICATION

Set this value to true to bypass TLS verification, only if necessary.

Note:

While this action might be necessary under certain testing conditions, it also carries inherent security risks.
  • Valid values: true, false
  • Default value: false (uses TLS verification)

EF_JUNIPER_APSTRA_API_USERNAME

Use this setting to input the username associated with your Apstra server. This setting is the same username you use to access the Apstra GUI.

  • Default value: ''

EF_JUNIPER_APSTRA_API_PASSWORD

Use this setting to enter the password corresponding to your Apstra server. This password is the same password you use to access the Apstra GUI.

  • Default value: ''

Logging

SUMMARY The following sections describe the logging configuration options for Flow Data.

EF_LOGGER_LEVEL

Use this setting to specify the output level for logging.

  • Valid values: debug, info, warn, error, panic, fatal
  • Default value: info

EF_LOGGER_ENCODING

Use this setting to specify the output format of the produced logs.

  • Valid values: console, json
  • Default: json

EF_LOGGER_FILE_LOG_ENABLE

Set to true to enable writing logs to a file.

  • Valid values: true, false
  • Default value: false

EF_LOGGER_FILE_LOG_FILENAME

Use this setting to specify the path to the file where the logs are written. When you enable file logging, EF_LOGGER_FILE_LOG_ENABLE is set to true.

  • Default path: /var/log/juniper/flowcoll/flowcoll.log

EF_LOGGER_FILE_LOG_MAX_SIZE

Use this setting to specify the maximum size, in MBs, of the log file before it is rotated.

  • Valid values: Any integer greater than 1.
  • Minimum value: 1
  • Default value: 100 megabytes

EF_LOGGER_FILE_LOG_MAX_AGE

Use this setting to specify the maximum number of days to retain old log files based on the timestamp encoded in the filenames. Because a day is defined as 24 hours, this value might not correspond to calendar days due to daylight savings, leap seconds, and so on.

  • Valid values: Any integer greater than or equal to 0.
  • Default: '' ( Does not remove old log files based on age).

EF_LOGGER_FILE_LOG_MAX_BACKUPS

Use this setting to specify the maximum number of old log files to retain. The default is to retain 4 old log files.

Note:

You can remove log files due to age (see EF_LOGGER_FILE_LOG_MAX_AGE) even if the maximum number of backups is not reached.

  • Valid values: Any integer greater than or equal to 0.
  • Default value: 4

EF_LOGGER_FILE_LOG_COMPRESS

Use this setting to enable compression of log files. Set this value to true to enable compression.

  • Valid values: true, false
  • Default: false

API

SUMMARY The following sections describe the API configuration options for Flow Data.

The Flow Data collector exposes an API that includes a Prometheus-compatible metrics endpoint and various endpoints for administrative tasks. These endpoints are described in the following sections:

EF_INSTANCE_NAME

Use this setting to configure the name of the collector instance.

  • Default name: default

EF_API_IP

Use this setting to define the IP address on which the collector listens for API requests.

  • Default IP address: 0.0.0.0

EF_API_PORT

Use this setting to define the port the Flow Data collector listens for API requests.

  • Default port number: 8080

EF_API_TLS_ENABLE

Use this setting to enable or disable TLS connections to the API endpoint.

  • Valid values: true, false
  • Default value: false

EF_API_TLS_CERT_FILEPATH

Use this setting to specify the path to the certificate to use for TLS connections to the API endpoint.

  • Default: ''

EF_API_TLS_KEY_FILEPATH

Use this setting to specify the path to the key to use for TLS connections to the API endpoint.

  • Default: ''

EF_API_BASIC_AUTH_ENABLE

Use this setting to enable or disable basic authentication protection of API endpoints.

  • Default: false

EF_API_BASIC_AUTH_USERNAME

Use this setting to specify the username to use to connect to basic authentication protection of API endpoints.

  • Default: ''

EF_API_BASIC_AUTH_PASSWORD

Use this setting to specify the password to use to connect to basic authentication protection of API endpoints.

  • Default: ''

Processor

SUMMARY The following sections describe the processor configuration options for Flow Data.

EF_PROCESSOR_POOL_SIZE

Use this setting to specify the number of record processors to start. You will need at least one processor for every 2000 records/second. Increasing the number of processors enables the collector to better handle a high volume of high latency enrichment tasks such as DNS lookups for IP addresses.

Note:

While increasing the number of processors can be beneficial, there are diminishing returns at higher processor counts. Especially when the number of processors exceeds the number of available CPU threads (real cores + SMT threads) or vCPUs. If you require more than 64 processors, and have an Apstra standard or premium License, it might be more beneficial to use multiple collector instances.

  • Default: 4 * the number of license units

EF_PROCESSOR_DECODE_IPFIX_ENABLE

Set to true to enable decoding of IPFIX records.

  • Valid values: true, false
  • Default value: true

EF_PROCESSOR_DECODE_NETFLOW1_ENABLE

Set to true to enable decoding of Netflow v1 records.

  • Valid values: true, false
  • Default value: true

EF_PROCESSOR_DECODE_NETFLOW5_ENABLE

Set to true to enable decoding of Netflow v5 records.

  • Valid values: true, false
  • Default value: true

EF_PROCESSOR_DECODE_NETFLOW6_ENABLE

Set to true to enable decoding of Netflow v6 records.

  • Valid values: true, false
  • Default value: true

EF_PROCESSOR_DECODE_NETFLOW7_ENABLE

Set to true to enable decoding of Netflow v7 records.

  • Valid values: true, false
  • Default value: true

EF_PROCESSOR_DECODE_NETFLOW9_ENABLE

Set to true to enable decoding of Netflow v9 records.

  • Valid values: true, false
  • Default value: true

EF_PROCESSOR_DECODE_SFLOW5_ENABLE

Set to true to enable decoding of sFlow v5 records.

  • Valid values: true, false
  • Default value: true

EF_PROCESSOR_DECODE_SFLOW_FLOWS_ENABLE

Set to true to enable decoding of sFlow flow_sample and flow_sample_expanded records.

  • Valid values: true, false
  • Default value: true

EF_PROCESSOR_DECODE_SFLOW_FLOWS_KEEP_SAMPLES

When set to true, the packet data from an sFlow sampled_header record is stored in l2.section.sample as a hex-encoded string.

  • Valid values: true, false
  • Default value: false

EF_PROCESSOR_DECODE_SFLOW_COUNTERS_ENABLE

Set to true to enable decoding of sFlow counters_sample and counters_sample_expanded records.

  • Valid values: true, false
  • Default value: true

EF_PROCESSOR_DECODE_MAX_RECORDS_PER_PACKET

Corrupt packets can cause issues decoding records. You avoid this from happenign by limiting the number of records to be decoded from a packet. When the network between the device and collector has an MTU larger than 1500, the default value can be exceeded by normal packets. This configuration option enables you to increase the threshold when necessary.

  • Default value: 64

EF_PROCESSOR_TRANSLATE_KEEP_IDS

Use this setting to specify which identifier values to be included in the final dataset.

  • Valid values:
    • none: All identifiers are removed from the final dataset.
    • default: Most identifiers are removed from the final dataset. Note that some identifiers that are required for common use-cases, such as raw protocol port values, are included.
    • all: All identifiers are included in the final dataset.
  • Default value: default

EF_PROCESSOR_DURATION_PRECISION

  • Valid values:
    • sec: seconds
    • ds: deciseconds
    • cs: centiseconds
    • ms: millseconds
    • us: microseconds
    • ns : nanoseconds
  • Default value: ms
Note:

For most data sources, this value is specified in millseconds (ms).

EF_PROCESSOR_TIMESTAMP_PRECISION

Use this setting to specify the desired precision of timestamp values. Values received at a different precision than specified are converted to the desired precision.

  • Valid values:
    • sec: seconds
    • ds: deciseconds
    • cs: centiseconds
    • ms: millseconds
    • us: microseconds
    • ns : nanoseconds
  • Default value: ms

EF_PROCESSOR_PERCENT_NORM

The desired representation of percentages. Values received with a different representation than specified are converted to the desired representation.

  • Valid values:
    • 1: values are based on a scale of 0 to 1.
    • 100: values are based on a scale of 0 to 100.
  • Default value: 100

EF_PROCESSOR_KEEP_CPU_TICKS

For telemetry sources that provide CPU usage, such as timeticks, utilization percentages are calculated. When this setting is set to false (default value), the timetick values are removed from the final dataset. If this setting is set to true, both the timetick values and utilization values are kept.

  • Valid values: true, false
  • Default value: false

EF_PROCESSOR_DROP_FIELDS

Use this setting to remove a comma-separated list of fields from all records.

Note:

The conversion from the default CODEX schema to alternate schemas happens within the respective outputs as fields are dropped before the outputs. You must use CODEX field names to configure this option.

  • Valid values:
    • any CODEX-schema field names, comma-separated. For example: flow.export.sysuptime,flow.export.version.ver,flow.start.sysuptime,flow.end.sysuptime,flow.seq_num
  • Default value: ''

EF_PROCESSOR_ENRICH_ASN_PREF

If enrichment with autonomous system (AS) attributes is enabled, but the AS is already indicated directly in the flow record data, use this setting to specify which source is preferred. If the preferred source is not available for a given record, the decoder will fall-back to the alternate option.

  • Valid values:
    • lookup: The AS determined by lookup.
    • flow: The AS is indicated directly in the flow record data.
  • Default value: lookup

EF_PROCESSOR_ENRICH_JOIN_ASN

Some features require that related values from separate fields are stored as an array in a single field. An attribute join of AS related fields is enabled when this setting is set to true.

  • Valid values: true, false
  • Default value: true

EF_PROCESSOR_ENRICH_JOIN_GEOIP

Some features require that related values from separate fields are stored as an array in a single field. An attribute joinof GeoIP related fields is enabled when this setting is set to true.

  • Valid values: true, false
  • Default value: true

EF_PROCESSOR_ENRICH_JOIN_NETATTR

Some features require that related values from separate fields are stored as an array in a single field. An attribute join of network attribute related fields is enabled when this setting is true.

  • Valid values: true, false
  • Default value: true

EF_PROCESSOR_ENRICH_JOIN_SUBNETATTR

Some features require that related values from separate fields are stored as an array in a single field. An attribute join of IP subnetwork related fields is enabled when this setting is set to true.

  • Valid values: true, false
  • Default value: true

EF_PROCESSOR_ENRICH_JOIN_SEC

Some features require that related values from separate fields are stored as an array in a single field. An attribute join of security attribute related fields is enabled when this setting is set to true.

  • Valid values: true, false
  • Default value: true

EF_PROCESSOR_EXPAND_CLISRV

The Flow Data collector infers the client/server relationship of two source/destination endpoints. Use this setting to enable or disable inference. The default value is true.

  • Valid values: true, false
  • Default value: true

EF_PROCESSOR_EXPAND_CLISRV_NO_L4_PORTS

For flow records related to protocols that include no layer-4 ports, the collector infers the client/server relationship of the two source/destination endpoints using the order of the IP addresses. Use this setting to enable or disable inference. The default value is true.

  • Valid values: true, false
  • Default value: true

EF_PROCESSOR_IFA_ENABLE

  • Valid values: true, false
  • Default value: false

EF_PROCESSOR_IFA_WORKER_SIZE

Use this setting to specify the number of IFA Hop record processors to start.

  • Default value: 4 * the number of license units

Outputs

SUMMARY The following sections describe the stdout configuration options for Flow Data.

stdout

The stdout output is used to output JSON-formatted records to a standard output. This output is useful during the initial installation or when troubleshooting issues to see Flow Data collector output directly in the terminal or logs.

Note:

The stdout output is used primarily for manual testing. This is because, at more than a few flow records per second, the data scrolls too fast to be useful.

EF_OUTPUT_STDOUT_ENABLE

Use this setting to enable or disable the stdout. The default value is false.

  • Valid values: true, false
  • Default value: false

EF_OUTPUT_STDOUT_FORMAT

Use this setting to specify how JSON documents are formatted. The default value is json_pretty.

  • Valid values:
    • json: Outputs a single JSON-formatted record per line.
    • json_pretty: Outputs each record as a "pretty" formatted JSON document ("pretty" refers to whitespace added to the document for easier human-readability).
  • Default value: json_pretty

Monitor

SUMMARY The following sections describe the monitor output configuration options for Flow Data.

EF_OUTPUT_MONITOR_ENABLE

The monitor output generates a log message containing the rate of records received and decoded by the Flow Data collector over the past interval (see EF_OUTPUT_MONITOR_INTERVAL) This output is useful for sizing or troubleshooting. To enable this option, set EF_OUTPUT_MONITOR_ENABLE to true.

  • Valid values: true, false
  • Default value: false

EF_OUTPUT_MONITOR_INTERVAL

Use this setting to specify the interval, in seconds, at which the rate of records is calculated and logged.

  • Default value: 300 (5 minutes)

OpenSearch

The following sections describe the OpenSearch output configuration options.
Note:

You can use the OpenSearch output to send records to OpenSearch, Open Distro for OpenSearch and Amazon OpenSearch Service.

EF_OUTPUT_OPENSEARCH_ENABLE

Use this setting to enable or disable OpenSearch output. The default value is false.

  • Valid values: true, false
  • Default value: false

EF_OUTPUT_OPENSEARCH_BATCH_DEADLINE

Use this setting to specify the maximum time (in ms) to wait for a batch of records to fill up before the records are sent to the OpenSearch bulk API.

  • Default value: 2000 ms.

EF_OUTPUT_OPENSEARCH_BATCH_MAX_BYTES

Use this setting to specify the maximum size of batch of records that can be sent to the OpenSearch bulk API.

  • Default value: 8388608 bytes.

EF_OUTPUT_OPENSEARCH_TIMESTAMP_SOURCE

Use this setting to specify the timestamp source used to set the @timestamp field. The recommended setting is end. If your device is behaving poorly or is misconfigured, we suggest you use the collect option instead.

  • Valid timestamp values:
    • start: The flow.start.timestamp indicates the flow start time.
    • end: The flow.end.timestamp is the last reported flow end time.
    • export: The flow.export.timestamp indicates time received from the flow record header.
    • collect: The flow.collect.timestamp indicates the time the Flow Data collector processes the flow record.
  • Default timestamp value: collect

EF_OUTPUT_OPENSEARCH_INDEX_PERIOD

Use this setting to specify how often new indexes are created (daily, weekly, monthly) and how to create and delete indexes.

  • Valid values:
    • daily : Indices are created each day. Specify this time period suffix as: -yyyy.MM.dd.
    • weekly: Indices are created each week. Specify this time period suffix as: -yyyy.'w'ww.
    • monthly: Indices are created each month. Specify this time period suffix as: -yyyy.MM.
    • ilm (Index Lifecycle Management): Use to create and delete indices.
  • Default value: daily

EF_OUTPUT_OPENSEARCH_INDEX_SUFFIX

Use this setting to specify a suffix to the index. This setting is useful if you have separate indices for different environments, locations or other organizational units.

  • Default value: ''

EF_OUTPUT_OPENSEARCH_INDEX_TEMPLATE_ENABLE

Use this setting to specify the output attempts to add the required index template to OpenSearch.

  • Valid values: true, false
  • Default value: true

EF_OUTPUT_OPENSEARCH_INDEX_TEMPLATE_OVERWRITE

Use this setting to determine if the index template should be overwritten or if it already exists. If the output is configured to add the index template to OpenSearch, set EF_OUTPUT_OPENSEARCH_INDEX_TEMPLATE_ENABLE to true.

  • Valid values: true, false
  • Default value: false

EF_OUTPUT_OPENSEARCH_INDEX_TEMPLATE_SHARDS

Use this setting to indicate the number of shards in which the index is created. As a general rule, additional shards increases ingest performance, assuming there are sufficient data nodes across in which the shards can be distributed.

  • Recommended number of shards: equal to the number of OpenSearch data nodes to which data to which the data is indexed.
  • Default number of shards: 3
Note:

This setting configures the index template sent to OpenSearch. It does not change any existing indexes.

EF_OUTPUT_OPENSEARCH_INDEX_TEMPLATE_REPLICAS

Use this setting to specify the number of replicas created for each shard.

In general, additional replicas increases query performance assuming there are sufficient data nodes across which the replicas can be distributed.

If you are using a multinode cluster and data redundancy is desired, this value must be at least 1.

  • Recommended number of replicas:
    • Use 1 if indexing data to a multi-node cluster.
    • Use 0 for a single-node.
  • Default value: 1
Note:

This setting configures the index template sent to OpenSearch. It does not change any existing indexes.

EF_OUTPUT_OPENSEARCH_INDEX_TEMPLATE_REFRESH_INTERVAL

Use this setting to specify the period for the refresh interval. TThis setting indicates the time that newly ingested documents are added to a segment, before the segment is added to the index. Only after the refresh interval ends and the segment is added to the index, do the documents become searchable.

  • Recommended refresh intervals:
    • 5s: Use this value for the data to become available for queries more quickly. Note that shorter refresh intervals might negatively impact ingest performance.
    • 30s (or longer): Use this value if maximizing ingest performance is your highest priority. Note that longer refresh intervals negatively impact the real-time accessibility of new records.
    • 10s or 15s: Use these values for most network traffic analytic use-cases. These interval numbers are a reasonable compromise between ingest performance and data accessibility.
  • Default value: 10s
Note:

This setting configures the index template that is sent to OpenSearch. It does not change any existing indexes.

EF_OUTPUT_OPENSEARCH_INDEX_TEMPLATE_CODEC

Use this setting to determine the level of compression used for stored values.

  • Valid values:
    • default: stored values are compressed using LZ4.
    • best_compression: stored values are compressed using the DEFLATE value. This value reduces disk capacity requirements with the trade-off of slightly higher CPU utilization.
  • Default value: best_compression
Note:

This setting configures the index template sent to OpenSearch. It does not change any existing indices.

EF_OUTPUT_OPENSEARCH_INDEX_TEMPLATE_ISM_POLICY

If data is being stored to an Open Distro for an OpenSearch cluster, this setting specifies the Index State Management (ISM) policy ID that is applied to the indexes. The default value is ''.

Note:

You must configure the ISM policy separately in OpenSearch.

  • Default value: ''

EF_OUTPUT_OPENSEARCH_INDEX_TEMPLATE_PIPELINE_DEFAULT

Use this setting to specify the name of the OpenSearch default pipeline or to process the OpenSearch ingest pipeline before the pipeline is indexed.

  • Default name: _none

EF_OUTPUT_OPENSEARCH_INDEX_TEMPLATE_PIPELINE_FINAL

Use this setting to specify the name of the OpenSearch final pipeline or to process the OpenSearch ingest pipeline before the pipeline is indexed.

  • Default value: _none

EF_OUTPUT_OPENSEARCH_ADDRESSES

Use this setting to specify the OpenSearch servers to which the output should connect. This value is a comma-separated list of OpenSearch nodes, including port number. Do not include http:// or https:// in the value.

  • Default value: 127.0.0.1:9200
Note:

You can enable or disable TLS communications using the EF_OUTPUT_OPENSEARCH_TLS_ENABLE option.

EF_OUTPUT_OPENSEARCH_USERNAME

Use this setting to specify the username to connect to the OpenSearch server.

  • Default value: admin

EF_OUTPUT_OPENSEARCH_PASSWORD

Use this setting to specify the password to connect to the OpenSearch server.

  • Default value: admin

EF_OUTPUT_OPENSEARCH_CLIENT_CA_CERT_FILEPATH

Use this setting to specify the path to the Certificate Authority (CA) certificate used for client PKI authentication.

  • Default value: ''

EF_OUTPUT_OPENSEARCH_CLIENT_CERT_FILEPATH

Use this setting to specify the path to the client certificate used for client PKI authentication.

  • Default value: ''

EF_OUTPUT_OPENSEARCH_CLIENT_KEY_FILEPATH

Use this setting to specify the path to the client key used for client PKI authentication.

  • Default value: ''

EF_OUTPUT_OPENSEARCH_TLS_ENABLE

Use this setting to enable or disable TLS connections to the OpenSearch server. The default value is false.

  • Valid values: true, false
  • Default value: false

EF_OUTPUT_OPENSEARCH_TLS_SKIP_VERIFICATION

Use this setting to enable or disable TLS verification of the OpenSearch server. The default value is false.

  • Valid values: true, false
  • Default value: false

EF_OUTPUT_OPENSEARCH_TLS_CA_CERT_FILEPATH

Use this setting to specify the path to the Certificate Authority (CA) certificate used tp verify the OpenSearch server connection.

  • Default value: ''

EF_OUTPUT_OPENSEARCH_RETRY_ENABLE

Use this setting to specify whether to retry connecting to the OpenSearch server after a connection has failed.

  • Valid values: true, false
  • Default: true

EF_OUTPUT_OPENSEARCH_RETRY_ON_TIMEOUT_ENABLE

Use this setting to specify whether to retry bulk indexing requests that timed-out.

  • Valid values: true, false
  • Default: true

EF_OUTPUT_OPENSEARCH_MAX_RETRIES

Use this setting to specify the number of times to retry bulk indexing requests which have timed-out.

  • Default value: 3 times

EF_OUTPUT_OPENSEARCH_RETRY_BACKOFF

Use this setting to specify the number of milliseconds (ms) you want the output to backoff before retrying a failed bulk request.

  • Default value: 1000 ms

EF_OUTPUT_OPENSEARCH_DROP_FIELDS

Use this setting to create a comma-separated list of fields to be removed from all records.

Note:

Fields are dropped if you add any output specific fields and dropped after any schema conversion. Make sure you use the same field names as the names that appear in the Apstra GUI.

  • Valid values: Any field names related to the enabled schema, comma-separated. For example: flow.export.sysuptime,flow.export.version.ver,flow.start.sysuptime,flow.end.sysuptime,flow.seq_num
  • Default value: ''

EF_OUTPUT_OPENSEARCH_ALLOWED_RECORD_TYPES

Use this setting to create a comma-separated list of record types. This list is particularly useful when used with multiple namespaced outputs, such as sending flow records to one datastore and telemetry to another.

  • Valid values: as_path_hop, flow_option, flow , telemetry, ifa_hop
  • Default values: 'as_path_hop,flow_option,flow,telemetry,ifa_hop '

See Also