Common Options
SUMMARY This topic describes the common configuration options for Apstra Flow.
Licensing
SUMMARY The following sections describe the licensing API configuration options for Apstra Flow.
- EF_JUNIPER_APSTRA_API_HOSTNAME
- EF_JUNIPER_APSTRA_API_PORT
- EF_JUNIPER_APSTRA_API_TLS_SKIP_VERIFICATION
- EF_JUNIPER_APSTRA_API_USERNAME
- EF_JUNIPER_APSTRA_API_PASSWORD
EF_JUNIPER_APSTRA_API_HOSTNAME
Use this setting to define the hostname or IP address where the Apstra server
provides its API services. This setting is the same IP address or hostname you
use to access the Apstra GUI. Note that this value must start with
http://
or https://
.
- Example:
http://localhost
- Default value:
''
Use the EF_JUNIPER_APSTRA_API_ADDRESS and EF_JUNIPER_APSTRA_API_TLS_ENABLE environment variables over EF_JUNIPER_APSTRA_API_HOSTNAME to create the URI needed to connect to the Apstra license server.
EF_JUNIPER_APSTRA_API_PORT
Use this setting to specify the port number on which the Apstra server exposes its API services. The most commonly used ports are port 80 and port 443.
- Example:
80
- Default value:
''
EF_JUNIPER_APSTRA_API_TLS_SKIP_VERIFICATION
Set this value to true
to bypass TLS verification, only if
necessary.
While this action might be necessary under certain testing conditions, it also carries inherent security risks.
- Valid values:
true
,false
- Default value:
false
(uses TLS verification)
EF_JUNIPER_APSTRA_API_USERNAME
Use this setting to input the username associated with your Apstra server. This setting is the same username you use to access the Apstra GUI.
- Default value:
''
EF_JUNIPER_APSTRA_API_PASSWORD
Use this setting to enter the password corresponding to your Apstra server. This password is the same password you use to access the Apstra GUI.
- Default value:
''
Logging
SUMMARY The following sections describe the logging configuration options for Apstra Flow.
- EF_LOGGER_LEVEL
- EF_LOGGER_ENCODING
- EF_LOGGER_FILE_LOG_ENABLE
- EF_LOGGER_FILE_LOG_FILENAME
- EF_LOGGER_FILE_LOG_MAX_SIZE
- EF_LOGGER_FILE_LOG_MAX_AGE
- EF_LOGGER_FILE_LOG_MAX_BACKUPS
- EF_LOGGER_FILE_LOG_COMPRESS
EF_LOGGER_LEVEL
Use this setting to specify the output level for logging.
- Valid values:
debug
,info
,warn
,error
,panic
,fatal
- Default value:
info
EF_LOGGER_ENCODING
Use this setting to specify the output format of the produced logs.
- Valid values:
console
,json
- Default:
json
EF_LOGGER_FILE_LOG_ENABLE
Set to true
to enable writing logs to a file.
- Valid values:
true
,false
- Default value:
false
EF_LOGGER_FILE_LOG_FILENAME
Use this setting to specify the path to the file where the logs are written.
When you enable file logging, EF_LOGGER_FILE_LOG_ENABLE
is set
to true
.
- Default path:
/var/log/flowdata/flowcoll/flowcoll.log
EF_LOGGER_FILE_LOG_MAX_SIZE
Use this setting to specify the maximum size (MB) of the log file before it is rotated.
- Valid values: Any integer greater than
1
. - Minimum value:
1
- Default value:
100
megabytes
EF_LOGGER_FILE_LOG_MAX_AGE
Use this setting to specify the maximum number of days to retain old log files based on the timestamp encoded in the filenames. Because a day is defined as 24 hours, this value might not correspond to calendar days due to daylight savings, leap seconds, and so on.
- Valid values: Any integer greater than or equal to
0
. - Default:
''
( Does not remove old log files based on age).
EF_LOGGER_FILE_LOG_MAX_BACKUPS
Use this setting to specify the maximum number of old log files to retain. The default is to retain 4 old log files.
You can remove log files due to age (see EF_LOGGER_FILE_LOG_MAX_AGE
) even if the maximum number of backups is
not reached.
- Valid values: Any integer greater than or equal to
0
. - Default value:
4
EF_LOGGER_FILE_LOG_COMPRESS
Use this setting to enable compression of log files. Set the value to
true
to enable compression.
- Valid values:
true
,false
- Default:
false
API
SUMMARY The following sections describe the API configuration options for Apstra Flow.
- EF_INSTANCE_NAME
- EF_API_IP
- EF_API_PORT
- EF_API_TLS_ENABLE
- EF_API_TLS_CERT_FILEPATH
- EF_API_TLS_KEY_FILEPATH
- EF_API_BASIC_AUTH_ENABLE
- EF_API_BASIC_AUTH_USERNAME
- EF_API_BASIC_AUTH_PASSWORD
EF_INSTANCE_NAME
Use this setting to configure the name of the collector instance.
- Default name:
default
EF_API_IP
Use this setting to define the IP address on which the collector listens for API requests.
- Default IP address:
0.0.0.0
EF_API_PORT
Use this setting to define the port the Apstra Flow collector listens for API requests.
- Default port number:
8080
EF_API_TLS_ENABLE
Use this setting to enable or disable TLS connections to the API endpoint.
- Valid values:
true
,false
- Default value:
false
EF_API_TLS_CERT_FILEPATH
Use this setting to specify the path to the certificate to use for TLS connections to the API endpoint.
- Default:
''
EF_API_TLS_KEY_FILEPATH
Use this setting to specify the path to the key to use for TLS connections to the API endpoint.
- Default:
''
EF_API_BASIC_AUTH_ENABLE
Use this setting to enable or disable basic authentication protection of API endpoints.
- Default:
false
EF_API_BASIC_AUTH_USERNAME
Use this setting to specify the username to use to connect to basic authentication protection of API endpoints.
- Default:
''
EF_API_BASIC_AUTH_PASSWORD
Use this setting to specify the password to use to connect to basic authentication protection of API endpoints.
- Default:
''
Processor
SUMMARY The following sections describe the processor configuration options for Apstra Flow.
- EF_PROCESSOR_POOL_SIZE
- EF_PROCESSOR_DECODE_IPFIX_ENABLE
- EF_PROCESSOR_DECODE_NETFLOW1_ENABLE
- EF_PROCESSOR_DECODE_NETFLOW5_ENABLE
- EF_PROCESSOR_DECODE_NETFLOW6_ENABLE
- EF_PROCESSOR_DECODE_NETFLOW7_ENABLE
- EF_PROCESSOR_DECODE_NETFLOW9_ENABLE
- EF_PROCESSOR_DECODE_SFLOW5_ENABLE
- EF_PROCESSOR_DECODE_SFLOW_FLOWS_ENABLE
- EF_PROCESSOR_DECODE_SFLOW_FLOWS_KEEP_SAMPLES
- EF_PROCESSOR_DECODE_SFLOW_COUNTERS_ENABLE
- EF_PROCESSOR_DECODE_MAX_RECORDS_PER_PACKET
- EF_PROCESSOR_TRANSLATE_KEEP_IDS
- EF_PROCESSOR_DURATION_PRECISION
- EF_PROCESSOR_TIMESTAMP_PRECISION
- EF_PROCESSOR_PERCENT_NORM
- EF_PROCESSOR_KEEP_CPU_TICKS
- EF_PROCESSOR_DROP_FIELDS
- EF_PROCESSOR_ENRICH_ASN_PREF
- EF_PROCESSOR_ENRICH_JOIN_ASN
- EF_PROCESSOR_ENRICH_JOIN_GEOIP
- EF_PROCESSOR_ENRICH_JOIN_NETATTR
- EF_PROCESSOR_ENRICH_JOIN_SUBNETATTR
- EF_PROCESSOR_ENRICH_JOIN_SEC
- EF_PROCESSOR_EXPAND_CLISRV
- EF_PROCESSOR_EXPAND_CLISRV_NO_L4_PORTS
- EF_PROCESSOR_IFA_ENABLE
- EF_PROCESSOR_IFA_WORKER_SIZE
EF_PROCESSOR_POOL_SIZE
Use this setting to specify the number of record processors to start. You will need at least one processor for every 2000 records/second. Increasing the number of processors enables the collector to better handle a high volume of high latency enrichment tasks such as DNS lookup for IP addresses.
While increasing the number of processors can be beneficial, you might see diminishing returns at higher processor counts. Especially when the number of processors exceeds the number of available CPU threads (real cores + SMT threads) or vCPUs. If you require more than 64 processors, and have an Apstra standard or premium License, it might be more beneficial to use multiple collector instances.
- Default:
4 * the number of license units
EF_PROCESSOR_DECODE_IPFIX_ENABLE
Set to true
to enable decoding of IPFIX records.
- Valid values:
true
,false
- Default value:
true
EF_PROCESSOR_DECODE_NETFLOW1_ENABLE
Set to true
to enable decoding of Netflow v1 records.
- Valid values:
true
,false
- Default value:
true
EF_PROCESSOR_DECODE_NETFLOW5_ENABLE
Set to true
to enable decoding of Netflow v5 records.
- Valid values:
true
,false
- Default value:
true
EF_PROCESSOR_DECODE_NETFLOW6_ENABLE
Set to true
to enable decoding of Netflow v6 records.
- Valid values:
true
,false
- Default value:
true
EF_PROCESSOR_DECODE_NETFLOW7_ENABLE
Set to true
to enable decoding of Netflow v7 records.
- Valid values:
true
,false
- Default value:
true
EF_PROCESSOR_DECODE_NETFLOW9_ENABLE
Set to true
to enable decoding of Netflow v9 records.
- Valid values:
true
,false
- Default value:
true
EF_PROCESSOR_DECODE_SFLOW5_ENABLE
Set to true
to enable decoding of sFlow v5 records.
- Valid values:
true
,false
- Default value:
true
EF_PROCESSOR_DECODE_SFLOW_FLOWS_ENABLE
Set to true
to enable decoding of sFlow
flow_sample
and flow_sample_expanded
records.
- Valid values:
true
,false
- Default value:
true
EF_PROCESSOR_DECODE_SFLOW_FLOWS_KEEP_SAMPLES
When set to true
, the packet data from an sFlow
sampled_header
record is stored in
l2.section.sample
as a hex-encoded string.
- Valid values:
true
,false
- Default value:
false
EF_PROCESSOR_DECODE_SFLOW_COUNTERS_ENABLE
Set to true
to enable decoding of sFlow
counters_sample
and
counters_sample_expanded
records.
- Valid values:
true
,false
- Default value:
true
EF_PROCESSOR_DECODE_MAX_RECORDS_PER_PACKET
Corrupt packets can cause issues decoding records. To prevent this, you can limit
the number of records to be decoded from a packet. When the network between the
device and collector has an MTU larger than 1500
, the default
value can be exceeded by normal packets. This configuration option enables you
to increase the threshold when necessary.
- Default value:
64
EF_PROCESSOR_TRANSLATE_KEEP_IDS
Use this setting to specify which ID values to be included in the final dataset.
- Valid values:
none
: All identifiers are removed from the final dataset.default
: Most identifiers are removed from the final dataset. Note that some identifiers that are required for common use-cases, such as raw protocol port values, are included.all
: All identifiers are included in the final dataset.
- Default value:
default
EF_PROCESSOR_DURATION_PRECISION
- Valid values:
sec
: secondsds
: decisecondscs
: centisecondsms
: millsecondsus
: microsecondsns
: nanoseconds
- Default value:
ms
For most data sources, this value is specified in milliseconds
(ms
).
EF_PROCESSOR_TIMESTAMP_PRECISION
Use this setting to specify the desired precision of timestamp values. Values received at a different precision than specified are converted to the desired precision.
- Valid values:
sec
: secondsds
: decisecondscs
: centisecondsms
: millsecondsus
: microseconds- ns : nanoseconds
- Default value:
ms
EF_PROCESSOR_PERCENT_NORM
The desired representation of percentages. Values received with a different representation than specified are converted to the desired representation.
- Valid values:
1
: values are based on a scale of 0 to 1.100
: values are based on a scale of 0 to 100.
- Default value:
100
EF_PROCESSOR_KEEP_CPU_TICKS
For telemetry sources that provide CPU usage, such as timeticks, utilization
percentages are calculated. When this setting is set to false
(default value), the timetick values are removed from the final dataset. If this
setting is set to true
, both the timetick values and
utilization values are kept.
- Valid values:
true
,false
- Default value:
false
EF_PROCESSOR_DROP_FIELDS
Use this setting to remove a comma-separated list of fields from all records.
The conversion from the default CODEX schema to alternate schemas happens within the respective outputs as fields are dropped before the outputs. You must use CODEX field names to configure this option.
- Valid values:
- any CODEX-schema field names, comma-separated. For example:
flow.export.sysuptime,flow.export.version.ver,flow.start.sysuptime,flow.end.sysuptime,flow.seq_num
- any CODEX-schema field names, comma-separated. For example:
- Default value:
''
EF_PROCESSOR_ENRICH_ASN_PREF
If enrichment with AS attributes is enabled, but the AS is referenced directly in the flow record data, use this setting to specify which source is preferred. If the preferred source is not available for a given record, the decoder will fall-back to the alternate option.
- Valid values:
lookup
: The AS determined by lookup.flow
: The AS is indicated directly in the flow record data.
- Default value:
lookup
EF_PROCESSOR_ENRICH_JOIN_ASN
Some features require that related values from separate fields are stored as an
array in a single field. A join attribute of AS related fields is enabled when
this setting is set to true
.
- Valid values:
true
,false
- Default value:
true
EF_PROCESSOR_ENRICH_JOIN_GEOIP
Some features require that related values from separate fields are stored as an
array in a single field. A join attribute of IP subnetwork related fields is
enabled of GeoIP related fields is enabled when this setting is set to
true
.
- Valid values:
true
,false
- Default value:
true
EF_PROCESSOR_ENRICH_JOIN_NETATTR
Some features require that related values from separate fields are stored as an
array in a single field. A join attribute of network attribute related fields is
enabled when this setting is true
.
- Valid values:
true
,false
- Default value:
true
EF_PROCESSOR_ENRICH_JOIN_SUBNETATTR
Some features require that related values from separate fields are stored as an
array in a single field. A join attribute of IP subnetwork related fields is
enabled when this setting is set to true
.
- Valid values:
true
,false
- Default value:
true
EF_PROCESSOR_ENRICH_JOIN_SEC
Some features require that related values from separate fields are stored as an
array in a single field. A join attribute of security attribute related fields
is enabled when this setting is set to true
.
- Valid values:
true
,false
- Default value:
true
EF_PROCESSOR_EXPAND_CLISRV
The Apstra Flow collector infers the client/server relationship of two
source/destination endpoints. Use this setting to enable or disable inference.
The default value is true
.
- Valid values:
true
,false
- Default value:
true
EF_PROCESSOR_EXPAND_CLISRV_NO_L4_PORTS
For flow records related to protocols that include no layer-4 ports, the
collector infers the client/server relationship of the two source/destination
endpoints using the order of the IP addresses. Use this setting to enable or
disable inference. The default value is true
.
- Valid values:
true
,false
- Default value:
true
EF_PROCESSOR_IFA_ENABLE
- Valid values:
true
,false
- Default value:
false
EF_PROCESSOR_IFA_WORKER_SIZE
Use this setting to specify the number of IFA Hop record processors to start.
- Default value:
4 * the number of license units
STDOUT Output
stdout
The stdout
output is used to output JSON-formatted records to a
standard output. This output is useful during the initial installation or when
troubleshooting issues to see Apstra Flow collector output directly in the
terminal or logs.
The stdout
output is used primarily for manual testing. This
is because (at more than a few flow records per second), the data scrolls
too fast to be useful.
EF_OUTPUT_STDOUT_ENABLE
Use this setting to enable or disable the stdout
. The default
value is false
.
- Valid values:
true
,false
- Default value:
false
EF_OUTPUT_STDOUT_FORMAT
Use this setting to specify how JSON documents are formatted. The default value
is json_pretty
.
- Valid values:
json
: Outputs a single JSON-formatted record per line.json_pretty
: Outputs each record as a "pretty" formatted JSON document ("pretty" refers to whitespace added to the document for easier human-readability).
- Default value:
json_pretty
Generic HTTP Output
SUMMARY Use the Generic HTTP output option to send records to an HTTP endpoint.
- EF_OUTPUT_GENERIC_HTTP_ENABLE
- EF_OUTPUT_GENERIC_HTTP_ECS_ENABLE
- EF_OUTPUT_GENERIC_HTTP_BATCH_DEADLINE
- EF_OUTPUT_GENERIC_HTTP_BATCH_MAX_BYTES
- EF_OUTPUT_GENERIC_HTTP_TIMESTAMP_SOURCE
- EF_OUTPUT_GENERIC_HTTP_ADDRESSES
- EF_OUTPUT_GENERIC_HTTP_USERNAME
- EF_OUTPUT_GENERIC_HTTP_PASSWORD
- EF_OUTPUT_GENERIC_HTTP_TLS_ENABLE
- EF_OUTPUT_GENERIC_HTTP_TLS_SKIP_VERIFICATION
- EF_OUTPUT_GENERIC_HTTP_TLS_CA_CERT_FILEPATH
- EF_OUTPUT_GENERIC_HTTP_DROP_FIELDS
EF_OUTPUT_GENERIC_HTTP_ENABLE
Use this setting to specify whether Generic HTTP output is enabled or disabled.
- Valid values:
true
,false
- Default value:
false
EF_OUTPUT_GENERIC_HTTP_ECS_ENABLE
Use this setting to specify whether the data is sent using Elastic Common Schema (ECS).
- Valid values:
true
,false
- Default value:
false
EF_OUTPUT_GENERIC_HTTP_BATCH_DEADLINE
Use this setting to specify the maximum waiting time (ms) for a batch of records to fill before being sent to the HTTP Endpoint.
- Default value:
2000
EF_OUTPUT_GENERIC_HTTP_BATCH_MAX_BYTES
Use this setting to specify the maximum size (in bytes) for a batch of records being sent to the HTTP Endpoint.
- Default value:
8388608
EF_OUTPUT_GENERIC_HTTP_TIMESTAMP_SOURCE
Use this setting to determine the timestamp source used to set the
@timestamp
field. Typically, end
is the
recommended setting. However, in the case of poorly behaving or misconfigured
devices, collect
might be the better option. For this reason
the default value is collect
because it handles a variety of
scenarios.
Valid values:
-
start
: The flow start time indicated in the flow. Use the timestamp fromflow.start.timestamp
: -
end
: The flow end time (or last reported time). Use the timestamp fromflow.end.timestamp
. -
export
: The time from the flow record header. Use the timestamp fromflow.export.timestamp
. -
collect
: The time that the collector processed the flow records. Use the timestamp fromflow.collect.timestamp
.
-
Default value:
collect
EF_OUTPUT_GENERIC_HTTP_ADDRESSES
Specifies the HTTP servers to which the output connects. It is a comma-separated list of HTTP servers, including port number.
IMPORTANT: Do not include http://
or https://
in the provided value. You enable of disable TLS communications by using using
EF_OUTPUT_GENERIC_HTTP_TLS_ENABLE
.
-
-
Default value: ``
-
Default value: 127.0.0.1: 8888
EF_OUTPUT_GENERIC_HTTP_USERNAME
Use this setting to specify the username used to connect to the HTTP endpoint.
-
Default value: ``
EF_OUTPUT_GENERIC_HTTP_PASSWORD
Use this setting to specify the password used to connect to the HTTP endpoint.
-
Default value: ``
EF_OUTPUT_GENERIC_HTTP_TLS_ENABLE
Use this setting to enable or disable TLS connections to the HTTP server.
- Valid values:
true
,false
- Default value:
false
EF_OUTPUT_GENERIC_HTTP_TLS_SKIP_VERIFICATION
Use this setting to enable or disable TLS verification of the HTTP server to which the output is trying to connect to.
- Valid values:
true
,false
- Default value:
false
EF_OUTPUT_GENERIC_HTTP_TLS_CA_CERT_FILEPATH
Use this setting to specify the path to the CA certificate used to verify the HTTP server to which the output is attempting to connect to.
- Default value:
''
EF_OUTPUT_GENERIC_HTTP_DROP_FIELDS
Use this setting to specify a comma-separated list of fields you want to remove from all records.
Fields are dropped after any output specific fields are added and after any schema conversion. This means that you must use the field names shown in the Apstra Flow UI.
-
Valid values: any field names that are related to the enabled schema, comma-separated. For example:
flow.export.sysuptime,flow.export.version.ver,flow.start.sysuptime,flow.end.sysuptime,flow.seq_num
-
Default value:
''
Monitor
SUMMARY The following sections describe the monitor output configuration options for Apstra Flow.
EF_OUTPUT_MONITOR_ENABLE
The monitor output generates a log message containing the rate of records
received and decoded by the Apstra Flow collector over the past interval (see
EF_OUTPUT_MONITOR_INTERVAL).
This output is useful for sizing or
troubleshooting. To enable this option, set
EF_OUTPUT_MONITOR_ENABLE
to true
.
- Valid values:
true
,false
- Default value:
false
EF_OUTPUT_MONITOR_INTERVAL
Use this setting to specify the interval, in seconds, at which the rate of records is calculated and logged.
- Default value:
300
(5 minutes)
OpenSearch
You can use the OpenSearch output to send records to OpenSearch, Open Distro for OpenSearch and Amazon OpenSearch Service.
- EF_OUTPUT_OPENSEARCH_ENABLE
- EF_OUTPUT_OPENSEARCH_BATCH_DEADLINE
- EF_OUTPUT_OPENSEARCH_BATCH_MAX_BYTES
- EF_OUTPUT_OPENSEARCH_TIMESTAMP_SOURCE
- EF_OUTPUT_OPENSEARCH_INDEX_PERIOD
- EF_OUTPUT_OPENSEARCH_INDEX_SUFFIX
- EF_OUTPUT_OPENSEARCH_INDEX_TEMPLATE_ENABLE
- EF_OUTPUT_OPENSEARCH_INDEX_TEMPLATE_OVERWRITE
- EF_OUTPUT_OPENSEARCH_INDEX_TEMPLATE_SHARDS
- EF_OUTPUT_OPENSEARCH_INDEX_TEMPLATE_REPLICAS
- EF_OUTPUT_OPENSEARCH_INDEX_TEMPLATE_REFRESH_INTERVAL
- EF_OUTPUT_OPENSEARCH_INDEX_TEMPLATE_CODEC
- EF_OUTPUT_OPENSEARCH_INDEX_TEMPLATE_ISM_POLICY
- EF_OUTPUT_OPENSEARCH_INDEX_TEMPLATE_PIPELINE_DEFAULT
- EF_OUTPUT_OPENSEARCH_INDEX_TEMPLATE_PIPELINE_FINAL
- EF_OUTPUT_OPENSEARCH_ADDRESSES
- EF_OUTPUT_OPENSEARCH_USERNAME
- EF_OUTPUT_OPENSEARCH_PASSWORD
- EF_OUTPUT_OPENSEARCH_CLIENT_CA_CERT_FILEPATH
- EF_OUTPUT_OPENSEARCH_CLIENT_CERT_FILEPATH
- EF_OUTPUT_OPENSEARCH_CLIENT_KEY_FILEPATH
- EF_OUTPUT_OPENSEARCH_TLS_ENABLE
- EF_OUTPUT_OPENSEARCH_TLS_SKIP_VERIFICATION
- EF_OUTPUT_OPENSEARCH_TLS_CA_CERT_FILEPATH
- EF_OUTPUT_OPENSEARCH_RETRY_ENABLE
- EF_OUTPUT_OPENSEARCH_RETRY_ON_TIMEOUT_ENABLE
- EF_OUTPUT_OPENSEARCH_MAX_RETRIES
- EF_OUTPUT_OPENSEARCH_RETRY_BACKOFF
- EF_OUTPUT_OPENSEARCH_DROP_FIELDS
- EF_OUTPUT_OPENSEARCH_ALLOWED_RECORD_TYPES
EF_OUTPUT_OPENSEARCH_ENABLE
Use this setting to enable or disable OpenSearch output. The default value is
false
.
- Valid values:
true
,false
- Default value:
false
EF_OUTPUT_OPENSEARCH_BATCH_DEADLINE
Use this setting to specify the maximum time (in ms) to wait for a batch of records to fill up before the records are sent to the OpenSearch bulk API.
- Default value:
2000
ms.
EF_OUTPUT_OPENSEARCH_BATCH_MAX_BYTES
Use this setting to specify the maximum size of batch of records that can be sent to the OpenSearch bulk API.
- Default value:
8388608
bytes.
EF_OUTPUT_OPENSEARCH_TIMESTAMP_SOURCE
Use this setting to specify the timestamp source used to set the
@timestamp
field. The recommended setting is
end
. If your device is behaving poorly or is misconfigured,
we suggest you use the collect
option instead.
- Valid timestamp values:
start
: Theflow.start.timestamp
indicates the flow start time.end
: Theflow.end.timestamp
is the last reported flow end time.export
: Theflow.export.timestamp
indicates time received from the flow record header.collect
: Theflow.collect.timestamp
indicates the time the Apstra Flow collector processes the flow record.
- Default timestamp value:
collect
EF_OUTPUT_OPENSEARCH_INDEX_PERIOD
Use this setting to specify how often new indexes are created (daily, weekly, monthly) and how to create and delete indexes.
- Valid values:
daily
: Indices are created each day. Specify this time period suffix as:-yyyy.MM.dd
.weekly
: Indices are created each week. Specify this time period suffix as:-yyyy.'w'ww
.monthly
: Indices are created each month. Specify this time period suffix as:-yyyy.MM
.ilm
(Index Lifecycle Management): Use to create and delete indices.
- Default value:
daily
EF_OUTPUT_OPENSEARCH_INDEX_SUFFIX
Use this setting to specify a suffix to the indexes. This setting is useful if you have separate indexes for different environments, locations or other organizational units.
- Default value:
''
EF_OUTPUT_OPENSEARCH_INDEX_TEMPLATE_ENABLE
Use this setting to specify the output attempts to add the required index template to OpenSearch.
- Valid values:
true
,false
- Default value:
true
EF_OUTPUT_OPENSEARCH_INDEX_TEMPLATE_OVERWRITE
Use this setting to determine if the index template should be overwritten or if
it exists. If the output is configured to add the index template to OpenSearch,
set EF_OUTPUT_OPENSEARCH_INDEX_TEMPLATE_ENABLE
to true
.
- Valid values:
true
,false
- Default value:
false
EF_OUTPUT_OPENSEARCH_INDEX_TEMPLATE_SHARDS
Use this setting to indicate the number of shards in which the index is created. As a general rule, additional shards increases ingest performance, assuming there are sufficient data nodes in which the shards can be distributed.
- Recommended number of shards: equal to the number of OpenSearch data nodes to which data to which the data is indexed.
- Default number of shards:
3
This setting configures the index template that is sent to OpenSearch. It does not change any existing indexes.
EF_OUTPUT_OPENSEARCH_INDEX_TEMPLATE_REPLICAS
Use this setting to specify the number of replicas created for each shard.
In general, additional replicas increases query performance assuming sufficient data nodes exist across which the replicas can be distributed.
If you are using a multinode cluster and data redundancy is desired, this value
must be at least 1
.
- Recommended number of replicas:
- Use
1
if indexing data to a multi-node cluster. - Use
0
for a single-node.
- Use
- Default value:
1
This setting configures the index template sent to OpenSearch. It does not change any existing indexes.
EF_OUTPUT_OPENSEARCH_INDEX_TEMPLATE_REFRESH_INTERVAL
Use this setting to specify the period for the refresh interval. This setting indicates the time that newly ingested documents are added to a segment, before the segment is added to the indexes. Only after the refresh interval ends and the segment is added to the indexes, do the documents become searchable.
- Recommended refresh intervals:
5s
: Use this value for the data to become available for queries more quickly. Note that shorter refresh intervals might negatively impact ingest performance.30s
(or longer): Use this value if maximizing ingest performance is your highest priority. Note that longer refresh intervals negatively impact the real-time accessibility of new records.10s
or15s
: Use these values for most network traffic analytic use-cases. These interval numbers are a reasonable compromise between ingest performance and data accessibility.
- Default value:
10s
This setting configures the indexes template that is sent to OpenSearch. It does not change any existing indexes.
EF_OUTPUT_OPENSEARCH_INDEX_TEMPLATE_CODEC
Use this setting to determine the level of compression used for stored values.
- Valid values:
default
: stored values are compressed using LZ4.best_compression
: stored values are compressed using theDEFLATE
value. This value reduces disk capacity requirements with the trade-off of slightly higher CPU utilization.
- Default value:
best_compression
This setting configures the indexes template that is sent to OpenSearch. It does not change any existing indexes.
EF_OUTPUT_OPENSEARCH_INDEX_TEMPLATE_ISM_POLICY
If data is being stored to an Open Distro for an OpenSearch cluster, this setting
specifies the Index State Management (ISM) policy ID that is applied to the
indexes The default value is ''
.
You must configure the ISM policy separately in OpenSearch.
- Default value:
''
EF_OUTPUT_OPENSEARCH_INDEX_TEMPLATE_PIPELINE_DEFAULT
Use this setting to specify the name of the OpenSearch default pipeline or to process the OpenSearch ingest pipeline before the pipeline is indexed.
- Default name:
_none
EF_OUTPUT_OPENSEARCH_INDEX_TEMPLATE_PIPELINE_FINAL
Use this setting to specify the name of the OpenSearch final pipeline or to process the OpenSearch ingest pipeline before the pipeline is indexed.
- Default value:
_none
EF_OUTPUT_OPENSEARCH_ADDRESSES
Use this setting to specify the OpenSearch servers to which the output should
connect. This value is a comma-separated list of OpenSearch nodes, including
port number. Do not include http://
or
https://
in the value.
- Default value:
127.0.0.1:9200
You can enable or disable TLS communications using the EF_OUTPUT_OPENSEARCH_TLS_ENABLE
option.
EF_OUTPUT_OPENSEARCH_USERNAME
Use this setting to specify the username to connect to the OpenSearch server.
- Default value:
admin
EF_OUTPUT_OPENSEARCH_PASSWORD
Use this setting to specify the password to connect to the OpenSearch server.
- Default value:
admin
EF_OUTPUT_OPENSEARCH_CLIENT_CA_CERT_FILEPATH
Use this setting to specify the path to the CA certificate used for client PKI authentication.
- Default value:
''
EF_OUTPUT_OPENSEARCH_CLIENT_CERT_FILEPATH
Use this setting to specify the path to the client certificate used for client PKI authentication.
- Default value:
''
EF_OUTPUT_OPENSEARCH_CLIENT_KEY_FILEPATH
Use this setting to specify the path to the client key used for client PKI authentication.
- Default value:
''
EF_OUTPUT_OPENSEARCH_TLS_ENABLE
Use this setting to enable or disable TLS connections to the OpenSearch server.
The default value is false
.
- Valid values:
true
,false
- Default value:
false
EF_OUTPUT_OPENSEARCH_TLS_SKIP_VERIFICATION
Use this setting to enable or disable TLS verification of the OpenSearch server.
The default value is false
.
- Valid values:
true
,false
- Default value:
false
EF_OUTPUT_OPENSEARCH_TLS_CA_CERT_FILEPATH
Use this setting to specify the path to the CA certificate used tp verify the OpenSearch server connection.
- Default value:
''
EF_OUTPUT_OPENSEARCH_RETRY_ENABLE
Use this setting to specify whether to retry connecting to the OpenSearch server after a connection has failed.
- Valid values:
true
,false
- Default:
true
EF_OUTPUT_OPENSEARCH_RETRY_ON_TIMEOUT_ENABLE
Use this setting to specify whether to retry bulk indexing requests that timed-out.
- Valid values:
true
,false
- Default:
true
EF_OUTPUT_OPENSEARCH_MAX_RETRIES
Use this setting to specify the number of times to retry bulk indexing requests which have timed-out.
- Default value:
3
times
EF_OUTPUT_OPENSEARCH_RETRY_BACKOFF
Use this setting to specify the number of milliseconds (ms) you want the output to backoff before retrying a failed bulk request.
- Default value:
1000
ms
EF_OUTPUT_OPENSEARCH_DROP_FIELDS
Use this setting to create a comma-separated list of fields to be removed from all records.
Fields are dropped if you add any output specific fields and dropped after any schema conversion. Make sure you use the same field names as the names that appear in the Apstra GUI.
- Valid values: Any field names related to the enabled schema,
comma-separated. For example:
flow.export.sysuptime,flow.export.version.ver,flow.start.sysuptime,flow.end.sysuptime,flow.seq_num
- Default value:
''
EF_OUTPUT_OPENSEARCH_ALLOWED_RECORD_TYPES
Use this setting to create a comma-separated list of record types. This list is particularly useful when used with multiple namespaced outputs, such as sending flow records to one datastore and telemetry to another.
- Valid values:
as_path_hop
,flow_option
,flow
,telemetry
,ifa_hop
- Default values:
'as_path_hop,flow_option,flow,telemetry,
ifa_hop
'