Anomaly Detection Rules

Anomaly Rules

Test event and flow traffic for changes in short-term events when you are comparing against a longer timeframe. For example, new services or applications that appear in a network, a web server crashes, firewalls that all start to deny traffic.

Example: You want to be notified when one of your firewall devices is reporting more often than it usually does because your network might be under attack. You want to be notified when you receive twice as many events in 1 hour. You follow these steps:

• Create and save a search that groups by log source, and displays only the count column.

• Apply the saved search to an anomaly rule, and add the rule test, and when the average value (per interval) of count over the last 1 hour is at least 100% different from the average value (per interval) of the same property over the last 24 hours.

Threshold Rules

Test events or flows for activity that is greater than or less than a specified range. Use these rules to detect bandwidth usage changes in applications, failed services, the number of users connected to a VPN, and detecting large outbound transfers.

Example: A user who was involved in a previous incident has large outbound transfer

When a user is involved in a previous offense, automatically set the Rule response to add to the Reference set. If you have a watch list of users, add them to the Reference set. Tune acceptable limits within the Threshold rule.

A reference set, WatchUsers, and Key:username are required for your search.

Complete the following search, and then apply it to a Threshold rule.

select assetuser(sourceip, now()) as ’srcAssetUser’, Applicationname(applicationid)as ’AppName’, long(sum(sourcebytes +destinationbytes)) as ’flowsum’ from flows where flowdirection = ’L2R’ and REFERENCESETCONTAINS(’Watchusers’, username)group by ’srcAssetUser’, applicationid order by ’flowsum’ desc last 24 hours

Behavioral Rules

Test events or flows for volume changes that occur in regular patterns to detect outliers. For example, a mail server that has an open relay and suddenly communicates with many hosts, or an IPS (intrusion protection system) that starts to generate numerous alert activities.

A behavioral rule learns the rate or volume of a property over a pre-defined season. The season defines the baseline comparison timeline for what you’re evaluating. When you set a season of 1 week, the behavior of the property over that 1 week is learned and then you use rule tests to alert you to any significant changes.

After a behavioral rule is set, the seasons adjusts automatically. When the data in the season is learned, it is continually evaluated so that business growth is profiled within the season; you do not have to change your rules. The longer a behavioral rule runs, the more accurate it becomes. You can then adjust the rule responses to capture more subtle changes.

The following table describes the Behavioral rule test parameter options.

Theforecast for value from the (n+1) th interval is calculated by using the following formula:

Fn+1 = Bn + Tn + Tn+1-s

Where F is the predicted value, B is the base value for interval n, T is the trend value for interval n, and T is the trend value for season intervals ago and s is the number of intervals within the season.

The base value is calculated by using the following formula:

Bn+1 = (0.2 + 0.3*(<Current traffic level>100.0))*(valuen+1 – Tn+1-s) + (1 – (0.2 + 0.3*(<Current traffic level> /100.0)))*Tn

The trend value is calculated by using the following formula:

Tn+1 = (0.2 + 0.3*(<Current traffic trend> / 100.0))*(Bn+1 - Bn) + (1 - (0.2 + 0.3*(<Current traffic trend> / 100.0)))*Tn

Smoothed deviation D is calculated by using the following formula:

Dn+1 = (0.2 + 0.3*(<Current traffic level>/ 100.0))*|valuen+1 – Fn+1| + (1 – (0.2 + 0.3*(<Current traffic level> /100.0)))*Dn+1-s

The behavioral rule produces an alert for the interval if the following expression is false:

F – (1 + (sensitivity / 100.0)*3)*D <= value <= F + (1 + (sensitivity / 100.0)*3)*D

During the first season, the behavioral rule learns for future calculations and doesn't produce any alerts.

Creating an Anomaly Detection Rule

Anomaly detection rules test the result of saved flow or event searches to search for unusual traffic patterns that occur in your network. Behavioral rules test event and flow traffic according to "seasonal" traffic levels and trends. Threshold rules test event and flow traffic for activity less than, equal to, or greater than a configured threshold or within a specified range.

To create anomaly detection rules on the Log Activity tab, you must have the Log Activity Maintain Custom Rules role permission.

To create anomaly detection rules on the Network Activity tab, you must have the Network Activity Maintain Custom Rules role permission.

To manage default and previously created anomaly detection rules, use the Rules page on the Offenses tab.

When you create an anomaly detection rule, the rule is populated with a default test stack, based on your saved search criteria. You can edit the default tests or add tests to the test stack. At least one Accumulated Property test must be included in the test stack.

By default, the Test the [Selected Accumulated Property] value of each [group] separately option is selected on the Rule Test Stack Editor page.

An anomaly detection rule tests the selected accumulated property for each event or flow group separately. For example, if the selected accumulated value is UniqueCount(sourceIP), the rule tests each unique source IP address for each event or flow group.

The Test the [Selected Accumulated Property] value of each [group] separately option is dynamic. The [Selected Accumulated Property] value depends on the option that you select for the this accumulated property test field of the default test stack. The [group] value depends on the grouping options that are specified in the saved search criteria. If multiple grouping options are included, the text might be truncated. Move your mouse pointer over the text to view all groups.

1. Click the Log Activity or Network Activity tab.

2. Perform an aggregated search.

   You can add a property to the group by in a new historical search or select a property from the Display list on the current search page.

3. On the search result page, click Configure, and then configure the following options:

   1. Select a property from the Value to Graph list.

   2. Select time series as the chart type from the Value to Graph list

   3. Enable the Capture Time Series Data check box.

   4. Click Save, and then enter a name for your search.

   5. Click OK.

   6. Select last 5 minutes from the Time Range list, while you wait for the time series graph to load.

   You must have time series data for the property that you selected in the Value to Graph list to run a rule test on that accumulated property.

4. From the Rules menu, select the rule type that you want to create.

   • Add Anomaly Rule

   • Add Threshold Rule

   • Add Behavioral Rule

5. On the Rule Test Stack Editor page, in the enter rule name here field, type a unique name that you want to assign to this rule.

6. To apply your rule by using the default test, select the first rule in the anomaly Test Group list.

   You might need to set the accumulated property parameter to the property that you selected from the Value to Graph list that you saved in the search criteria. If you want to see the result sooner, set the percentage to a lower value, such as 10%. Change last 24 hours to a lesser time period, such as 1 hour. Because an anomaly detection tests on aggregated fields in real time to alert you of anomalous network activity, you might want to increase or decrease events or flows in your network traffic.

7. Add a test to a rule.

   1. To filter the options in the Test Group list, type the text that you want to filter for in the Type to filter field.

   2. From the Test Group list, select the type of test that you want to add to this rule.

   3. To identify a test as an excluded test, click and at the beginning of the test in the Rule pane. The and is displayed as and not.

   4. Click the underlined configurable parameters to customize the variables of the test.

   5. From the dialog box, select values for the variable, and then click Submit.

8. To test the total selected accumulated properties for each event or flow group, disable Test the [Selected Accumulated Property] value of each [group] separately.

9. In the groups pane, enable the groups you want to assign this rule to.

10. In the Notes field, type any notes that you want to include for this rule, and then Click Next.

11. On the Rule Responses page, configure the responses that you want this rule to generate.

    Learn more about rule response page parameters for anomaly detection rules:

    The following table provides the Rule Response page parameters if the rule type is Anomaly.

    Table 2: Anomaly Detection Rule Response Page Parameters

    Parameter	Description
    Dispatch New Event	Specifies that this rule dispatches a new event with the original event or flow, which is processed like all other events in the system. By default, this check box is selected and cannot be cleared.
    Offense Naming	If you want the Event Name information to contribute to the name of the offense, select the This information should contribute to the name of the associated offense(s) option. If you want the configured Event Name to contribute to the offense, select the This information should set or replace the name of the associated offense(s). Note: After you replace the name of the offense, the name won't change until the offense is closed. For example, if an offense is associated with more than one rule, and the last event doesn't trigger the rule that is configured to override the name of the offense, the offense's name won't be updated by the last event. Instead, the offense name remains the name that is set by the override rule.
    Severity	The severity level that you want to assign to the event. The range is 0 (lowest) to 10 (highest) and the default is 5. The Severity is displayed in the Annotations pane of the event details.
    Credibility	The credibility that you want to assign to the log source. For example, is the log source noisy or expensive? Using the list boxes, select the credibility of the event. The range is 0 (lowest) to 10 (highest) and the default is 5. Credibility is displayed in the Annotations pane of the event details.
    Relevance	The relevance that you want to assign to the weight of the asset. For example, how much do you care about the asset? Using the list boxes, select the relevance of the event. The range is 0 (lowest) to 10 (highest) and the default is 5. Relevance is displayed in the Annotations pane of the event details.
    Ensure that the dispatched event is part of an offense	As a result of this rule, the event is forwarded to the magistrate. If an offense exists, this event is added. If no offense was created on the Offenses tab, a new offense is created.
    Notify	Events that generate as a result of this rule are displayed in the System Notifications item in the Dashboard tab. If you enable notifications, configure the Response Limiter parameter
    Send to Local SysLog	Select this check box if you want to log the event or flow locally. By default, the check box is clear. Note: Only normalized events can be logged locally on a JSA appliance. If you want to send raw event data, you must use the Send to Forwarding Destinations option to send the data to a remote syslog host.
    Add to Reference Set	Adds events that are generated as a result of this rule to a reference set. You must be an administrator to add data to a reference set. To add data to a reference set, follow these steps: From the first list, select the property of the event or flow that you want to add. From the second list, select the reference set to which you want to add the specified data.
    Add to Reference Data	To use this rule response, you must create the reference data collection.
    Remove from Reference Set	If you want this rule to remove data from a reference set, select this check box. To remove data from a reference set, follow these steps: From the first list, select the property of the event or flow that you want to remove. From the second list, select the reference set from which you want to remove the specified data.
    Remove from Reference Data	To use this rule response, you must have a reference data collection
    Execute Custom Action	You can write scripts that do specific actions in response to network events. For example, you might write a script to create a firewall rule that blocks a particular source IP address from your network in response to repeated login failures. Select this check box and select a custom action from the Custom action to execute list. You add and configure custom actions by using the Define Actions icon on the Admin tab.
    Publish on the IF-MAP Server	If the IF-MAP parameters are configured and deployed in the system settings, select this option to publish the offense information about the IF-MAP server.
    Response Limiter	Select this check box and use the list boxes to configure the frequency with which you want this rule to respond
    Enable Rule	Select this check box to enable this rule. By default, the check box is selected.

    An SNMP notification might resemble:

    "Wed Sep 28 12:20:57 GMT 2005, Custom Rule Engine Notification Rule ’SNMPTRAPTst’ Fired. 172.16.20.98:0 -> 172.16.60.75:0 1, Event Name: ICMP Destination Unreachable Communication with Destination Host is Administratively Prohibited, QID: 1000156, Category: 1014, Notes: Offense description"

    Sep 28 12:39:01 localhost.localdomain ECS: Rule ’Name of Rule’ Fired: 172.16.60.219:12642 -> 172.16.210.126:6666 6, Event Name: SCAN SYN FIN, QID: 1000398, Category: 1011, Notes: Event description

12. Click Next.

13. Click Finish.