Operation

The Apstra Flow collector’s log reports the errors, processor to output writer or UDP Server to Flow Decoder are 90 percent full. For example:

{"level":"info","ts":"2023-08-07T08:08:14.301Z","logger":"flowcoll","caller":"flowprocessor/metrics.go:118","msg":"flow processor to output writer is 90% full. This is normal when the collector is starting. If it persists for hours, it may indicate that you are at your license threshold or your system is under-resourced."}

{"level":"info","ts":"2023-08-07T08:08:34.264Z","logger":"flowcoll","caller":"server/metrics.go:125","msg":"UDP Server to Flow Decoder is 90% full. This is normal when the collector is starting. If it persists for hours, it may indicate that you are at your license threshold or your system is under-resourced."}

You might see these logs accompanied by throttle logs.

2023-06-28T21:20:21.821Z        warn    throttle/restricted_throttle.go:105     [throttler]: start burst
2023-06-28T21:20:41.822Z        warn    throttle/restricted_throttle.go:111     [throttler]: stop burst
2023-06-28T21:20:41.822Z        warn    throttle/restricted_throttle.go:117     [throttler]: start recovery
2023-06-28T21:50:42.142Z        warn    throttle/restricted_throttle.go:123     [throttler]: stop recovery

Problem

These messages usually occur when the collector first starts, as various internal processes might not be fully initialized. However, if the messages persist after the first few minutes, one of the following issues might exist:

• ONLY flow processor to output writer: Indicates that the system, which data is being output, lacks sufficient performance to ingest records at the rate being sent by the Apstra Flow collector. This can caused by insufficient CPU, memory, disk space, or excessive disk latency. Insufficient network bandwidth between the collector and target system can also cause the problem.
• BOTH UDP Server to Flow Decoder and flow processor to output writer: This is a further progression of the previous condition. The resulting back pressure from the slow downstream system is now likely causing data to be lost.

• ONLY UDP Server to Flow Decoder: The internal decoder/processor workers cannot keep up with the rate of records being received. This issue can be caused by one of the following conditions:

  • More records are being received than are allowed by the license. If so, throttler messages will also appear in the log.

  • The collector has insufficient resources, primarily CPU cores, to process the rate of records being received.

  • The caches, for IP addresses, interfaces, and so on, have yet to be "warmed up" so the related high latency enrichment tasks are limiting throughput.

Solution

The solution varies depending on the issue, as described in the problem section above.

• • ONLY flow processor to output writer: Increases the performance of the system to which records are being sent.
  • BOTH UDP Server to Flow Decoder and flow processor to output writer: Increases the performance of the system to which records are being sent.
  • ONLY UDP Server to Flow Decoder:
    • Increase the CPU cores available to the collector.
    • If the collector has sufficient CPU resources try increasing the processor pool size by setting the EF_PROCESSOR_POOL_SIZE. This allows great concurrency of high latency enrichment tasks.
      Note:

      If throttler messages appear in the log, contact your Juniper sales representative about subscription options that allows you to collector additional flow records.

Question:

When upgrading the OpenSearch dashboards, is it necessary to also update the Apstra Flow dashboards?

Answer:

The Apstra Flow dashboards should continue to work after upgrading the OpenSearch dashboards. If However, if you want to use the latest dashboards, you can usually overwrite the existing saved objects. You can also delete the existing saved objects before importing the latest.

Question:

When using the OpenSearch outputs, can the names of the indexes be changed?

Answer:

No, changing the names of Apstra Flow-related indexes is not supported. The various components of the Apstra Flow solutions are designed to work together in an integrated manner. Changing the indexes names can potentially break dashboards, index state management (ISM) policies, machine-learning jobs and alerts. Although you can use an ingest pipeline to change the indexes names as records are ingested,Apstra Flow does not support this type of environment.

Often the reason for changing indexes names is to achieve multi-tenancy, separate indexes for different environments, locations or other organizational units. To facilitate this use-case, use the EF_OUTPUT_OPENSEARCH_INDEX_SUFFIX option to add a suffix to the index name.

Consider the index name juniper-flow-codex-2.2-2023.01.01.

For example, if you set the suffix value to staging, the resulting index name will be called: juniper-flow-codex-2.2-staging-2023.01.01. You can then control access to the staging indexes by setting permissions for the *-staging-* index naming pattern.