Event Processing Performance

Non-optimized Custom Properties

Custom properties are marked as optimized when they are regularly used for JSA rules or for searching and filtering.

Non-optimized custom properties are parsed by the system, which affects search speeds and the loading rate of the web browser.

Rule Tests That Impact Performance

Rules that test for regular expressions in an event payload affect JSA performance because they search the entire payload.

Before you add a payload test to a rule, use rule filters to reduce the number of events. For example, when you search for a specific message in the active directory logs, apply the following filters to the rule:

• Log source type filter

• Log source group or specific log source filter

• An optional source IP address filter

The Host with port open test can impact performance because it compares passive and active ports with the events and flows that are received by JSA. Before you use the test, do a bidirectional check to ensure that the host responds to the communication request.

Global Views

A saved search that is grouped by multiple fields generates a global view that has many unique entries. As the volume of data increases, disk usage, processing times, and search performance can be impacted.

To prevent increasing the volume of data, only aggregate searches on necessary fields. You can reduce the impact on the accumulator by adding a filter to your search criteria.

Identifying DSM and Optimized Custom Property Issues

To help you troubleshoot performance degradation, identify issues with any DSM extensions that were recently installed or custom property that was recently enabled.

A DSM extension creates custom parsing methods by using regex pattern matching to extract event data from unsupported or incomplete log sources. Optimized custom properties use regular expression patterns to extract data from events as they are parsed.

The regex patterns that are used in your DSM extension or optimized custom property can impact event processing in JSA. Inefficient regular expressions can incorrectly route data directly to storage, degrade JSA performance, and affect event processing.

DSM and optimized custom property issues can cause the following system notification:

Performance degradation has been detected in the event pipeline. Events were routed directly to storage.

1. Disable any DSM extension or custom property that is recently installed or enabled.

2. Choose one of the following options:

   • If JSA stops dropping events and you receive a system notification, then review your DSM extensions or custom properties to identify and improve the inefficient regex patterns.

   • If JSA continues dropping events, then multiple DSM extensions or custom properties might be causing a problem with the event pipeline.

3. Use SSH to log in to the JSA Event Processor that is dropping events and type the following command:

   /opt/qradar/support/threadTop.sh –p 7777

   The command displays the data processing engine activity. The following table describes the columns in the output:

   Table 1: Data Processing Engine Columns

   Columns	Description
   Server	Port or process.
   ID	Process ID.
   MSecs	CPU time.
   Name	Process name.

4. If parser threads run longer than 1500 milliseconds, review the Java thread stacks by typing the following command:

   /opt/qradar/support/threadTop.sh -p 7777 -s -e ".*Event Parser.*" | less

If the Java thread stack contains java.util.regex.Pattern$Curly.match, then the performance degradation might be caused by your expensive DSM extensions or custom properties. For more information, see Expensive DSM extensions were found in Log and Log Source Notifications for JSA Appliances or Expensive custom properties found in Event and Flow Notifications for JSA Appliances.

If the Java thread stack doesn't have expensive regular expressions, then your DSM extensions or custom properties might have parsing issues. For more information, see the parsing issues topic in the Configuring DSMs Guide.