Index Management
Use Index Management to control database indexing on event and flow properties. To improve the speed of searches in JSA, narrow the overall data by adding an indexed field in your search query.
An index is a set of items that specify information about data in a file and its location in the file system. Data indexes are built in real-time as data is streamed or are built upon request after data is collected. Searching is more efficient because systems that use indexes don't have to read through every piece of data to locate matches. The index contains references to unique terms in the data and their locations. Because indexes use disk space, storage space might be used to decrease search time.
Use indexing event and flow properties first to optimize your searches. You can enable indexing on any property that is listed in the Index Management window and you can enable indexing on more than one property. When a search starts in JSA, the search engine first filters the data set by indexed properties. The indexed filter eliminates portions of the data set and reduces the overall data volume and number of event or flow logs that must be searched. Without any filters, JSA takes more time to return the results for large data sets.
For example, you might want to find all the logs in the past
six months that match the text: The operation is not
allowed. By default, JSA stores full
text indexing for the past 30 days. Therefore, to complete a search
from the last 6 months, the system must reread every payload value
from every event or flow in that time frame to find matches. Your
results display faster when you search with an indexed value filter
such as a Log Source Type, Event Name, or Source IP.
The Index Management feature also provides statistics, such as:
The percentage of saved searches running in your deployment that include the indexed property
The volume of data that is written to the disk by the index during the selected time frame
To enable payload indexing, you must enable indexing on the Quick Filter property.
Enabling Indexes
The Index Management window lists all event and flow event properties that can be indexed and provides statistics for the properties. Toolbar options allow you to enable and disable indexing on selected event and flow event properties.
Modifying database indexing might decrease system performance. Ensure that you monitor the statistics after you enable indexing on multiple properties.
On the navigation menu (
), click Admin.In the System Configuration section, click Index Management.
Select one or more properties from the Index Management list.
Choose one of the following options:
Situation
Time frame
Action
Reason
The index is disabled and % of Searches Using Property is above 30% and % of Searches Missing Index is above 30%.
24 hours, 7 days, or 30 days
Click Enable Index.
This search property is used often. Enabling an index can improve performance.
The index is enabled and % of Searches Using Property is zero.
30 days
Click Disable Index.
The enabled index is not used in the searches. Disable the indexed property to preserve disk space.
Click Save.
Click OK.
In lists that include event and flow event properties, indexed property names are appended with
the following text: [Indexed]. Examples
of such lists include the search parameters on the
Log Activity and Network Activity
Log Activity
tab search criteria pages and the Add Filter window.
Enabling Payload Indexing to Optimize Search Times
To optimize event and flow search times, enable payload indexing on the Quick Filter property.
Use the Quick Filter feature in the Log Activity and Network Activity tab to search event and flow payloads by using a text string.
Payload indexing increases disk storage requirements and might affect system performance. Enable payload indexing if your deployment meets the following conditions:
The event and flow processors are at less than 70% disk usage.
The event and flow processors are less than 70% of the maximum events per second (EPS) or flows per interface (FPI) rating.
On the navigation menu (
), click Admin.In the System Configuration section, click Index Management.
In the Quick Search field, type Quick Filter.
The Quick Filter property is displayed.
Select the Quick Filter property that you want to index.
In the results table, use the value in the Database column to identify the flows or events Quick Filter property.
On the toolbar, click Enable Index.
A green dot indicates that the payload index is enabled.
If a list includes event or flow properties that are indexed, the property names are appended with the following text:
[Indexed].Click Save.
To manage payload indexes, see Configuring the Retention Period for Payload Indexes.
Configuring the Retention Period for Payload Indexes
By default, JSA sets 30 days for the data retention period of the payload index. You can search for specific values in quick filter indexes beyond 30 days by changing the default retention in JSA.
Your virtual and physical appliances require a minimum of 24 GB of RAM to enable full payload indexing. However, 48 GB of RAM is suggested.
The minimum and suggested RAM values applies to all JSA systems that are processing events or flows.
The retention values reflect the time spans that you are typically searching. The minimum retention period is 1 day and the maximum is 2 years.
Quick Filter searches that use a time frame outside of the Payload Index Retention setting can trigger slow and resource-intensive system responses. For example, if the payload index retention is set for 1 day, and you use a time frame for the last 30 hours in the search.
On the navigation menu (
), click Admin.In the System Configuration section, click System Settings.
In the Database Settings section, select a retention time period from the Payload Index Retention list.
Click Save.
Close the System Settings window.
On the Admin tab menu, click Deploy Changes.
If you retain payload indexes longer than the default value, extra disk space is used. After you select a greater value in the Payload Index Retention field, monitor system notifications to ensure that you do not fill disk space.