Custom Rules Notifications for JSA Appliances

Explanation

The custom rules engine (CRE) on an event processor is unable to read a rule to correlate an incoming event. The notification might contain one of the following messages:

• If the CRE was unable to read a single rule, in most cases, a recent rule change is the cause. The payload of the notification message displays the rule or rule of the rule chain that is responsible.

• In rare circumstances, data corruption can cause a complete failure of the rule set. An application error is displayed and the rule editor interface might become unresponsive or generate more errors.

User Response

For a single rule read error, review the following options:

• To locate the rule that is causing the notification, temporarily disable the rule.

• Edit the rule to revert any recent changes.

• Delete and re-create the rule that is causing the error.

For application errors where the CRE failed to read rules, contact Juniper Customer Support.

Explanation

A single rule referred to itself directly or to itself through a series of other rules or building blocks. The error occurs when you deploy a full configuration. The rule set is not loaded.

User Response

Edit the rules that created the cyclic dependency. The rule chain must be broken to prevent a recurring system notification. After the rule chain is corrected, a save automatically reloads the rules and resolves the issue.

Explanation

The custom rules engine (CRE) is a process that validates if an event matches a rule set and then trigger alerts, offenses, or notifications

A user can create a custom rule that has a large scope, uses a regex pattern that is not efficient, includes Payload contains tests, or combines the rule with regular expressions. When this custom rule is used, it negatively impacts performance, which can cause events to be incorrectly routed directly to storage. Events are indexed and normalized but they don't trigger alerts or offenses.

When multiple, expensive, or inefficient rule tests are used, the maximum event throughput rate can be reduced, causing backlogs of events to go through the rules engine. Events might be routed directly to storage, and this warning is displayed.

User Response

Review the following options:

• Review the payload of the notification to determine which expensive rule in the pipeline affects performance.

• For example, the following payload reports the test: "Payload Verification" rule in the pipeline and the EPS rate reported is 787 events per second, potentially reducing the maximum throughput of the rules engine.

• On the Offenses tab, click Rules and use the search window to find and either edit or disable the expensive rule. By editing the rule, you can reduce the amount of data that goes through the rule, by applying a log source or IP address range filter. Expensive tests, such as payload contains, can also be reduced or removed if they are not required. Reference set tests are to be reviewed to ensure that they are not querying a large reference set.

• Use SSH to log in to the Event Processor and verify that parser threads are running for longer than 1500 milliseconds for EPS loads by using the following command:

• Order your log source parsers from the log sources with the most sent events to the least and disable unused parsers.

  /opt/qradar/support/threadTop.sh

  Search the Java thread stack for regex.Pattern.Curly, referenceSet, assets, host profile, and port profile by using the following command:

  /opt/qradar/support/threadTop.sh -p 7799 -s -e ".*CRE Processor.*"

  • If the output contains regex.Pattern.Curly, issues with Payload contains tests are possible.

  • If the output contains referenceSet, issues might occur with tests against large reference sets.

  • If the output contains assets, host profile, and port profile, issues might occur with Host with port open tests or asset tests.

Rules Might Not Be the Issue

This notification can trigger when events are routed to storage around the rules engine. If, when you investigate this notice, the "EPS" rate in the notification is higher than ~20,000 EPS, it can indicate that the issue might be elsewhere. A rule that can process events upwards of 20,000 EPS is fairly optimized. The situation that triggered the 'events routed to storage' might not be a rule, but might be something else. Other items to consider are listed as follows.

• Is the system under higher load for other reasons, such as long-term data searching?

• Is the disk utilization at 85% or higher "on/store", and potentially data compression is affecting storage performance?

• If HA is in use, and event rates are higher than 10,000 EPS, ensure that sufficient bandwidth is between the two HA nodes. For example, a single 1Gbps connection, even in a dedicated crossover, can limit storage performance.

• Is there a separate "/transient/" partition. If not, then temporary data decompression might also use storage resources and contribute to the high storage demands.