Common Options

EF_JUNIPER_APSTRA_API_HOSTNAME

Use this setting to define the hostname or IP address where the Apstra server provides its API services. This setting is the same IP address or hostname you use to access the Apstra GUI. Note that this value must start with http:// or https://.

• Example: http://localhost
• Default value: ''

EF_JUNIPER_APSTRA_API_PORT

Use this setting to specify the port number on which the Apstra server exposes its API services. The most commonly used ports are port 80 and port 443.

• Example: 80
• Default value: ''

EF_JUNIPER_APSTRA_API_TLS_SKIP_VERIFICATION

Set this value to true to bypass TLS verification, only if necessary.

• Valid values: true, false
• Default value: false (uses TLS verification)

EF_JUNIPER_APSTRA_API_USERNAME

Use this setting to input the username associated with your Apstra server. This setting is the same username you use to access the Apstra GUI.

• Default value: ''

EF_JUNIPER_APSTRA_API_PASSWORD

Use this setting to enter the password corresponding to your Apstra server. This password is the same password you use to access the Apstra GUI.

• Default value: ''

EF_LOGGER_LEVEL

Use this setting to specify the output level for logging.

• Valid values: debug, info, warn, error, panic, fatal
• Default value: info

EF_LOGGER_ENCODING

Use this setting to specify the output format of the produced logs.

• Valid values: console, json
• Default: json

EF_LOGGER_FILE_LOG_ENABLE

Set to true to enable writing logs to a file.

• Valid values: true, false
• Default value: false

EF_LOGGER_FILE_LOG_FILENAME

Use this setting to specify the path to the file where the logs are written. When you enable file logging, EF_LOGGER_FILE_LOG_ENABLE is set to true.

• Default path: /var/log/flowdata/flowcoll/flowcoll.log

EF_LOGGER_FILE_LOG_MAX_SIZE

Use this setting to specify the maximum size (MB) of the log file before it is rotated.

• Valid values: Any integer greater than 1.
• Minimum value: 1
• Default value: 100 megabytes

EF_LOGGER_FILE_LOG_MAX_AGE

Use this setting to specify the maximum number of days to retain old log files based on the timestamp encoded in the filenames. Because a day is defined as 24 hours, this value might not correspond to calendar days due to daylight savings, leap seconds, and so on.

• Valid values: Any integer greater than or equal to 0.
• Default: '' ( Does not remove old log files based on age).

EF_LOGGER_FILE_LOG_MAX_BACKUPS

Use this setting to specify the maximum number of old log files to retain. The default is to retain 4 old log files.

• Valid values: Any integer greater than or equal to 0.
• Default value: 4

EF_LOGGER_FILE_LOG_COMPRESS

Use this setting to enable compression of log files. Set the value to true to enable compression.

• Valid values: true, false
• Default: false

EF_INSTANCE_NAME

Use this setting to configure the name of the collector instance.

• Default name: default

EF_API_IP

Use this setting to define the IP address on which the collector listens for API requests.

• Default IP address: 0.0.0.0

EF_API_PORT

Use this setting to define the port the Apstra Flow collector listens for API requests.

• Default port number: 8080

EF_API_TLS_ENABLE

Use this setting to enable or disable TLS connections to the API endpoint.

• Valid values: true, false
• Default value: false

EF_API_TLS_CERT_FILEPATH

Use this setting to specify the path to the certificate to use for TLS connections to the API endpoint.

• Default: ''

EF_API_TLS_KEY_FILEPATH

Use this setting to specify the path to the key to use for TLS connections to the API endpoint.

• Default: ''

EF_API_BASIC_AUTH_ENABLE

Use this setting to enable or disable basic authentication protection of API endpoints.

• Default: false

EF_API_BASIC_AUTH_USERNAME

Use this setting to specify the username to use to connect to basic authentication protection of API endpoints.

• Default: ''

EF_API_BASIC_AUTH_PASSWORD

Use this setting to specify the password to use to connect to basic authentication protection of API endpoints.

• Default: ''

EF_PROCESSOR_POOL_SIZE

Use this setting to specify the number of record processors to start. You will need at least one processor for every 2000 records/second. Increasing the number of processors enables the collector to better handle a high volume of high latency enrichment tasks such as DNS lookup for IP addresses.

• Default: 4 * the number of license units

EF_PROCESSOR_DECODE_IPFIX_ENABLE

Set to true to enable decoding of IPFIX records.

• Valid values: true, false
• Default value: true

EF_PROCESSOR_DECODE_NETFLOW1_ENABLE

Set to true to enable decoding of Netflow v1 records.

• Valid values: true, false
• Default value: true

EF_PROCESSOR_DECODE_NETFLOW5_ENABLE

Set to true to enable decoding of Netflow v5 records.

• Valid values: true, false
• Default value: true

EF_PROCESSOR_DECODE_NETFLOW6_ENABLE

Set to true to enable decoding of Netflow v6 records.

• Valid values: true, false
• Default value: true

EF_PROCESSOR_DECODE_NETFLOW7_ENABLE

Set to true to enable decoding of Netflow v7 records.

• Valid values: true, false
• Default value: true

EF_PROCESSOR_DECODE_NETFLOW9_ENABLE

Set to true to enable decoding of Netflow v9 records.

• Valid values: true, false
• Default value: true

EF_PROCESSOR_DECODE_SFLOW5_ENABLE

Set to true to enable decoding of sFlow v5 records.

• Valid values: true, false
• Default value: true

EF_PROCESSOR_DECODE_SFLOW_FLOWS_ENABLE

Set to true to enable decoding of sFlow flow_sample and flow_sample_expanded records.

• Valid values: true, false
• Default value: true

EF_PROCESSOR_DECODE_SFLOW_FLOWS_KEEP_SAMPLES

When set to true, the packet data from an sFlow sampled_header record is stored in l2.section.sample as a hex-encoded string.

• Valid values: true, false
• Default value: false

EF_PROCESSOR_DECODE_SFLOW_COUNTERS_ENABLE

Set to true to enable decoding of sFlow counters_sample and counters_sample_expanded records.

• Valid values: true, false
• Default value: true

EF_PROCESSOR_DECODE_MAX_RECORDS_PER_PACKET

Corrupt packets can cause issues decoding records. To prevent this, you can limit the number of records to be decoded from a packet. When the network between the device and collector has an MTU larger than 1500, the default value can be exceeded by normal packets. This configuration option enables you to increase the threshold when necessary.

• Default value: 64

EF_PROCESSOR_TRANSLATE_KEEP_IDS

Use this setting to specify which ID values to be included in the final dataset.

• Valid values:
  • none: All identifiers are removed from the final dataset.
  • default: Most identifiers are removed from the final dataset. Note that some identifiers that are required for common use-cases, such as raw protocol port values, are included.
  • all: All identifiers are included in the final dataset.
• Default value: default

EF_PROCESSOR_DURATION_PRECISION

• Valid values:
  • sec: seconds
  • ds: deciseconds
  • cs: centiseconds
  • ms: millseconds
  • us: microseconds
  • ns : nanoseconds
• Default value: ms

EF_PROCESSOR_TIMESTAMP_PRECISION

Use this setting to specify the desired precision of timestamp values. Values received at a different precision than specified are converted to the desired precision.

• Valid values:
  • sec: seconds
  • ds: deciseconds
  • cs: centiseconds
  • ms: millseconds
  • us: microseconds
  • ns : nanoseconds
• Default value: ms

EF_PROCESSOR_PERCENT_NORM

The desired representation of percentages. Values received with a different representation than specified are converted to the desired representation.

• Valid values:
  • 1: values are based on a scale of 0 to 1.
  • 100: values are based on a scale of 0 to 100.
• Default value: 100

EF_PROCESSOR_KEEP_CPU_TICKS

For telemetry sources that provide CPU usage, such as timeticks, utilization percentages are calculated. When this setting is set to false (default value), the timetick values are removed from the final dataset. If this setting is set to true, both the timetick values and utilization values are kept.

• Valid values: true, false
• Default value: false

EF_PROCESSOR_DROP_FIELDS

Use this setting to remove a comma-separated list of fields from all records.

• Valid values:
  • any CODEX-schema field names, comma-separated. For example: flow.export.sysuptime,flow.export.version.ver,flow.start.sysuptime,flow.end.sysuptime,flow.seq_num
• Default value: ''

EF_PROCESSOR_ENRICH_ASN_PREF

If enrichment with AS attributes is enabled, but the AS is referenced directly in the flow record data, use this setting to specify which source is preferred. If the preferred source is not available for a given record, the decoder will fall-back to the alternate option.

• Valid values:
  • lookup: The AS determined by lookup.
  • flow: The AS is indicated directly in the flow record data.
• Default value: lookup

EF_PROCESSOR_ENRICH_JOIN_ASN

Some features require that related values from separate fields are stored as an array in a single field. A join attribute of AS related fields is enabled when this setting is set to true.

• Valid values: true, false
• Default value: true

EF_PROCESSOR_ENRICH_JOIN_GEOIP

Some features require that related values from separate fields are stored as an array in a single field. A join attribute of IP subnetwork related fields is enabled of GeoIP related fields is enabled when this setting is set to true.

• Valid values: true, false
• Default value: true

EF_PROCESSOR_ENRICH_JOIN_NETATTR

Some features require that related values from separate fields are stored as an array in a single field. A join attribute of network attribute related fields is enabled when this setting is true.

• Valid values: true, false
• Default value: true

EF_PROCESSOR_ENRICH_JOIN_SUBNETATTR

Some features require that related values from separate fields are stored as an array in a single field. A join attribute of IP subnetwork related fields is enabled when this setting is set to true.

• Valid values: true, false
• Default value: true

EF_PROCESSOR_ENRICH_JOIN_SEC

Some features require that related values from separate fields are stored as an array in a single field. A join attribute of security attribute related fields is enabled when this setting is set to true.

• Valid values: true, false
• Default value: true

EF_PROCESSOR_EXPAND_CLISRV

The Apstra Flow collector infers the client/server relationship of two source/destination endpoints. Use this setting to enable or disable inference. The default value is true.

• Valid values: true, false
• Default value: true

EF_PROCESSOR_EXPAND_CLISRV_NO_L4_PORTS

For flow records related to protocols that include no layer-4 ports, the collector infers the client/server relationship of the two source/destination endpoints using the order of the IP addresses. Use this setting to enable or disable inference. The default value is true.

• Valid values: true, false
• Default value: true

EF_PROCESSOR_IFA_ENABLE

• Valid values: true, false
• Default value: false

EF_PROCESSOR_IFA_WORKER_SIZE

Use this setting to specify the number of IFA Hop record processors to start.

• Default value: 4 * the number of license units

stdout

The stdout output is used to output JSON-formatted records to a standard output. This output is useful during the initial installation or when troubleshooting issues to see Apstra Flow collector output directly in the terminal or logs.

EF_OUTPUT_STDOUT_ENABLE

Use this setting to enable or disable the stdout. The default value is false.

• Valid values: true, false
• Default value: false

EF_OUTPUT_STDOUT_FORMAT

Use this setting to specify how JSON documents are formatted. The default value is json_pretty.

• Valid values:
  • json: Outputs a single JSON-formatted record per line.
  • json_pretty: Outputs each record as a "pretty" formatted JSON document ("pretty" refers to whitespace added to the document for easier human-readability).
• Default value: json_pretty

EF_OUTPUT_GENERIC_HTTP_ENABLE

Use this setting to specify whether Generic HTTP output is enabled or disabled.

• Valid values: true, false
• Default value: false

EF_OUTPUT_GENERIC_HTTP_ECS_ENABLE

Use this setting to specify whether the data is sent using Elastic Common Schema (ECS).

• Valid values: true, false
• Default value: false

EF_OUTPUT_GENERIC_HTTP_BATCH_DEADLINE

Use this setting to specify the maximum waiting time (ms) for a batch of records to fill before being sent to the HTTP Endpoint.

• Default value: 2000

EF_OUTPUT_GENERIC_HTTP_BATCH_MAX_BYTES

Use this setting to specify the maximum size (in bytes) for a batch of records being sent to the HTTP Endpoint.

• Default value: 8388608

EF_OUTPUT_GENERIC_HTTP_TIMESTAMP_SOURCE

Use this setting to determine the timestamp source used to set the @timestamp field. Typically, end is the recommended setting. However, in the case of poorly behaving or misconfigured devices, collect might be the better option. For this reason the default value is collect because it handles a variety of scenarios.

Valid values:

• start: The flow start time indicated in the flow. Use the timestamp from flow.start.timestamp:

• end: The flow end time (or last reported time). Use the timestamp from flow.end.timestamp.

• export: The time from the flow record header. Use the timestamp from flow.export.timestamp.

• collect: The time that the collector processed the flow records. Use the timestamp from flow.collect.timestamp.

• Default value: collect

EF_OUTPUT_GENERIC_HTTP_ADDRESSES

Specifies the HTTP servers to which the output connects. It is a comma-separated list of HTTP servers, including port number.

IMPORTANT: Do not include http:// or https:// in the provided value. You enable of disable TLS communications by using using EF_OUTPUT_GENERIC_HTTP_TLS_ENABLE.

• • Default value: ``

Default value: 127.0.0.1: 8888

EF_OUTPUT_GENERIC_HTTP_USERNAME

Use this setting to specify the username used to connect to the HTTP endpoint.

• Default value: ``

EF_OUTPUT_GENERIC_HTTP_PASSWORD

Use this setting to specify the password used to connect to the HTTP endpoint.

• Default value: ``

EF_OUTPUT_GENERIC_HTTP_TLS_ENABLE

Use this setting to enable or disable TLS connections to the HTTP server.

• Valid values: true, false
• Default value: false

EF_OUTPUT_GENERIC_HTTP_TLS_SKIP_VERIFICATION

Use this setting to enable or disable TLS verification of the HTTP server to which the output is trying to connect to.

• Valid values: true, false
• Default value: false

EF_OUTPUT_GENERIC_HTTP_TLS_CA_CERT_FILEPATH

Use this setting to specify the path to the CA certificate used to verify the HTTP server to which the output is attempting to connect to.

• Default value: ''

EF_OUTPUT_GENERIC_HTTP_DROP_FIELDS

Use this setting to specify a comma-separated list of fields you want to remove from all records.

EF_OUTPUT_MONITOR_ENABLE

The monitor output generates a log message containing the rate of records received and decoded by the Apstra Flow collector over the past interval (see EF_OUTPUT_MONITOR_INTERVAL). This output is useful for sizing or troubleshooting. To enable this option, set EF_OUTPUT_MONITOR_ENABLE to true.

• Valid values: true, false
• Default value: false

EF_OUTPUT_MONITOR_INTERVAL

Use this setting to specify the interval, in seconds, at which the rate of records is calculated and logged.

• Default value: 300 (5 minutes)

EF_OUTPUT_OPENSEARCH_ENABLE

Use this setting to enable or disable OpenSearch output. The default value is false.

• Valid values: true, false
• Default value: false

EF_OUTPUT_OPENSEARCH_BATCH_DEADLINE

Use this setting to specify the maximum time (in ms) to wait for a batch of records to fill up before the records are sent to the OpenSearch bulk API.

• Default value: 2000 ms.

EF_OUTPUT_OPENSEARCH_BATCH_MAX_BYTES

Use this setting to specify the maximum size of batch of records that can be sent to the OpenSearch bulk API.

• Default value: 8388608 bytes.

EF_OUTPUT_OPENSEARCH_TIMESTAMP_SOURCE

Use this setting to specify the timestamp source used to set the @timestamp field. The recommended setting is end. If your device is behaving poorly or is misconfigured, we suggest you use the collect option instead.

• Valid timestamp values:
  • start: The flow.start.timestamp indicates the flow start time.
  • end: The flow.end.timestamp is the last reported flow end time.
  • export: The flow.export.timestamp indicates time received from the flow record header.
  • collect: The flow.collect.timestamp indicates the time the Apstra Flow collector processes the flow record.
• Default timestamp value: collect

EF_OUTPUT_OPENSEARCH_INDEX_PERIOD

Use this setting to specify how often new indexes are created (daily, weekly, monthly) and how to create and delete indexes.

• Valid values:
  • daily : Indices are created each day. Specify this time period suffix as: -yyyy.MM.dd.
  • weekly: Indices are created each week. Specify this time period suffix as: -yyyy.'w'ww.
  • monthly: Indices are created each month. Specify this time period suffix as: -yyyy.MM.
  • ilm (Index Lifecycle Management): Use to create and delete indices.
• Default value: daily

EF_OUTPUT_OPENSEARCH_INDEX_SUFFIX

Use this setting to specify a suffix to the indexes. This setting is useful if you have separate indexes for different environments, locations or other organizational units.

• Default value: ''

EF_OUTPUT_OPENSEARCH_INDEX_TEMPLATE_ENABLE

Use this setting to specify the output attempts to add the required index template to OpenSearch.

• Valid values: true, false
• Default value: true

EF_OUTPUT_OPENSEARCH_INDEX_TEMPLATE_OVERWRITE

Use this setting to determine if the index template should be overwritten or if it exists. If the output is configured to add the index template to OpenSearch, set EF_OUTPUT_OPENSEARCH_INDEX_TEMPLATE_ENABLE to true.

• Valid values: true, false
• Default value: false

EF_OUTPUT_OPENSEARCH_INDEX_TEMPLATE_SHARDS

Use this setting to indicate the number of shards in which the index is created. As a general rule, additional shards increases ingest performance, assuming there are sufficient data nodes in which the shards can be distributed.

• Recommended number of shards: equal to the number of OpenSearch data nodes to which data to which the data is indexed.
• Default number of shards: 3

EF_OUTPUT_OPENSEARCH_INDEX_TEMPLATE_REPLICAS

Use this setting to specify the number of replicas created for each shard.

In general, additional replicas increases query performance assuming sufficient data nodes exist across which the replicas can be distributed.

If you are using a multinode cluster and data redundancy is desired, this value must be at least 1.

• Recommended number of replicas:
  • Use 1 if indexing data to a multi-node cluster.
  • Use 0 for a single-node.
• Default value: 1

EF_OUTPUT_OPENSEARCH_INDEX_TEMPLATE_REFRESH_INTERVAL

Use this setting to specify the period for the refresh interval. This setting indicates the time that newly ingested documents are added to a segment, before the segment is added to the indexes. Only after the refresh interval ends and the segment is added to the indexes, do the documents become searchable.

• Recommended refresh intervals:
  • 5s: Use this value for the data to become available for queries more quickly. Note that shorter refresh intervals might negatively impact ingest performance.
  • 30s (or longer): Use this value if maximizing ingest performance is your highest priority. Note that longer refresh intervals negatively impact the real-time accessibility of new records.
  • 10s or 15s: Use these values for most network traffic analytic use-cases. These interval numbers are a reasonable compromise between ingest performance and data accessibility.
• Default value: 10s

EF_OUTPUT_OPENSEARCH_INDEX_TEMPLATE_CODEC

Use this setting to determine the level of compression used for stored values.

• Valid values:
  • default: stored values are compressed using LZ4.
  • best_compression: stored values are compressed using the DEFLATE value. This value reduces disk capacity requirements with the trade-off of slightly higher CPU utilization.
• Default value: best_compression

EF_OUTPUT_OPENSEARCH_INDEX_TEMPLATE_ISM_POLICY

If data is being stored to an Open Distro for an OpenSearch cluster, this setting specifies the Index State Management (ISM) policy ID that is applied to the indexes The default value is ''.

• Default value: ''

EF_OUTPUT_OPENSEARCH_INDEX_TEMPLATE_PIPELINE_DEFAULT

Use this setting to specify the name of the OpenSearch default pipeline or to process the OpenSearch ingest pipeline before the pipeline is indexed.

• Default name: _none

EF_OUTPUT_OPENSEARCH_INDEX_TEMPLATE_PIPELINE_FINAL

Use this setting to specify the name of the OpenSearch final pipeline or to process the OpenSearch ingest pipeline before the pipeline is indexed.

• Default value: _none

EF_OUTPUT_OPENSEARCH_ADDRESSES

Use this setting to specify the OpenSearch servers to which the output should connect. This value is a comma-separated list of OpenSearch nodes, including port number. Do not include http:// or https:// in the value.

• Default value: 127.0.0.1:9200

EF_OUTPUT_OPENSEARCH_USERNAME

Use this setting to specify the username to connect to the OpenSearch server.

• Default value: admin

EF_OUTPUT_OPENSEARCH_PASSWORD

Use this setting to specify the password to connect to the OpenSearch server.

• Default value: admin

EF_OUTPUT_OPENSEARCH_CLIENT_CA_CERT_FILEPATH

Use this setting to specify the path to the CA certificate used for client PKI authentication.

• Default value: ''

EF_OUTPUT_OPENSEARCH_CLIENT_CERT_FILEPATH

Use this setting to specify the path to the client certificate used for client PKI authentication.

• Default value: ''

EF_OUTPUT_OPENSEARCH_CLIENT_KEY_FILEPATH

Use this setting to specify the path to the client key used for client PKI authentication.

• Default value: ''

EF_OUTPUT_OPENSEARCH_TLS_ENABLE

Use this setting to enable or disable TLS connections to the OpenSearch server. The default value is false.

• Valid values: true, false
• Default value: false

EF_OUTPUT_OPENSEARCH_TLS_SKIP_VERIFICATION

Use this setting to enable or disable TLS verification of the OpenSearch server. The default value is false.

• Valid values: true, false
• Default value: false

EF_OUTPUT_OPENSEARCH_TLS_CA_CERT_FILEPATH

Use this setting to specify the path to the CA certificate used tp verify the OpenSearch server connection.

• Default value: ''

EF_OUTPUT_OPENSEARCH_RETRY_ENABLE

Use this setting to specify whether to retry connecting to the OpenSearch server after a connection has failed.

• Valid values: true, false
• Default: true

EF_OUTPUT_OPENSEARCH_RETRY_ON_TIMEOUT_ENABLE

Use this setting to specify whether to retry bulk indexing requests that timed-out.

• Valid values: true, false
• Default: true

EF_OUTPUT_OPENSEARCH_MAX_RETRIES

Use this setting to specify the number of times to retry bulk indexing requests which have timed-out.

• Default value: 3 times

EF_OUTPUT_OPENSEARCH_RETRY_BACKOFF

Use this setting to specify the number of milliseconds (ms) you want the output to backoff before retrying a failed bulk request.

• Default value: 1000 ms

EF_OUTPUT_OPENSEARCH_DROP_FIELDS

Use this setting to create a comma-separated list of fields to be removed from all records.

• Valid values: Any field names related to the enabled schema, comma-separated. For example: flow.export.sysuptime,flow.export.version.ver,flow.start.sysuptime,flow.end.sysuptime,flow.seq_num
• Default value: ''

EF_OUTPUT_OPENSEARCH_ALLOWED_RECORD_TYPES

Use this setting to create a comma-separated list of record types. This list is particularly useful when used with multiple namespaced outputs, such as sending flow records to one datastore and telemetry to another.

• Valid values: as_path_hop, flow_option, flow , telemetry, ifa_hop
• Default values: 'as_path_hop,flow_option,flow,telemetry,ifa_hop '