Creating a Log Source Extensions Document to get data into JSA

You create log source extensions (LSX) when log sources don't have a supported DSM, or to repair an event that has missing or incorrect information, or to parse an event when the associated DSM fails to produce a result.

When to create a Log Source Extension

For log sources that don't have an official DSM, use a custom log source type to integrate log sources. A log source extension (also known as a device extension) is then applied to the custom log source type to provide the logic for parsing the logs. The LSX is based on Java regular expressions and can be used against any protocol type, such as syslog, JDBC, and Log File. Values can be extracted from the logs and mapped to all common fields within JSA.

When you use log source extensions to repair missing or incorrect content, any new events that are produced by the log source extensions are associated to the log source that failed to parse the original payload. Creating an extension prevents unknown or uncategorized events from being stored as unknown in JSA.

Using the DSM Editor to quickly create a Log Source Extension

For JSA 2014.8 and later, you can use the DSM Editor to create log source extensions. The DSM Editor provides real-time feedback so that you know whether the log source extension that you are creating has problems. Use the DSM Editor to extract fields, define custom properties, categorize events, define new QID definitions, and define your own log source type. For more information about the DSM Editor, see the Juniper Secure Analytics Administration Guide.

Process for manually creating a Log Source Extension

Alternatively, to manually create a log source extension, complete the following steps:

Ensure that a log source is created in JSA.

Use a custom log source type to collect events from a source when the log source type not listed as a JSA supported DSM.

Use the DSM Editor to create the new log source type, and then manually create the log source. You can attach an LSX to a supported log source type, such as Windows, Bluecoat, Cisco, and others that are listed as JSA supported DSMs.
To determine what fields are available, use the Log Activity tab to export the logs for evaluation.
Use the extension document example template to determine the fields that you can use.

It is not necessary to use all of the fields in the template. Determine the values in the log source that can be mapped to the fields in extension document template.
Remove any unused fields and their corresponding Pattern IDs from the log source extension document.
Upload the extension document and apply the extension to the log source.
Map the events to their equivalents in the QIDmap.

This manual action on the Log Activity tab is used to map unknown log source events to known JSA events so that they can be categorized and processed.

Common Regular Expressions

Use regular expressions to match patterns of text in the log source file. You can scan messages for patterns of letters, numbers, or a combination of both. For example, you can create regular expressions that match source and destination IP addresses, ports, MAC addresses, and more.

The following codes shows several common regular expressions:

The escape character, or "\", is used to denote a literal character. For example, "." character means "any single character" and matches A, B, 1, X, and so on. To match the "." characters, a literal match, you must use "\."

Table 1: Common Regex Expressions
Type	Expression
Type	\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}
IP Address	\d{1,5}
Port Number	(?:[0-9a-fA-F]{2}\:){5}[0-9a-fA-F]{2}
Protocol	(TCP\|UDP\|ICMP\|GRE)
Device Time	\w{3}\s\d{2}\s\d{2}:\d{2}:\d{2}
Whitespace	\s
Tab	\t
Match Anything	.*?

Tip:

To ensure that you don't accidentally match another characters, escape any non-digit or non-alpha character.

Building Regular Expression Patterns

To create a log source extension, you use regular expressions (regex) to match strings of text from the unsupported log source.

The following example shows a log entry that is referenced in the steps.

Visually analyze the unsupported log source to identify unique patterns.

These patterns are later translated into regular expressions.
Find the text strings to match.

Tip:
To provide basic error checking, include characters before and after the values to prevent similar values from being unintentionally matched. You can later isolate the actual value from the extra characters.
Develop pseudo-code for matching patterns and include the space character to denote the beginning and end of a pattern.

You can ignore the quotes. In the example log entry, the event names are DROP, PASS, and REJECT. The following list shows the usable event fields.
- EventName: " kernel: VALUE "
- SourceMAC: " MAC=VALUE "
- SourceIp: " SRC=VALUE "
- DestinationIp: " DST=VALUE "
- Protocol: " PROTO=VALUE "
- SourcePort: " SPT=VALUE "
- DestinationPort: " DPT=VALUE "
Substitute a space with the \s regular expression.

You must use an escape character for non-digit or non-alpha characters. For example, = becomes \= and : becomes \:.

Translate the pseduo-code to a regular expression.

Table 2: Translating Pseudo-code to Regular Expressions
Field	Pseudo-code	Regular expression
EventName	" kernel: VALUE "	\skernel\:\s.*?\s
SourceMAC	" MAC=VALUE "	\sMAC\=(?:[0-9a-fA-F]{2}\:){5}[0-9a-fA-F]{2}\s
SourceIP	" SRC=VALUE "	\sSRC\=\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\s
DestinationIp	" DST=VALUE "	\sDST\=\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\s
Protocol	" PROTO=VALUE "	\sPROTO\=(TCP\|UDP\|ICMP\|GRE)\s
SourcePort	" SPT=VALUE "	\sSPT\=\d{1,5}\s
DestinationPort	" DPT=VALUE "	\sDPT\=\d{1,5}\s

Specify capture groups.

A capture group isolates a certain value in the regular expression.

For example, in the SourcePort pattern in the previous example, you can't pass the entire value since it includes spaces and SRC=<code>. Instead, you specify only the port number by using a capture group. The value in the capture group is what is passed to the relevant field in JSA.

Insert parenthesis around the values you that you want capture:

Table 3: Mapping Regular Expressions to Capture Groups for Event Fields
Field	Regular expression	Capture group
EventName	\skernel\:\s.*?\s	\skernel\:\s(.*?)\s
SourceMAC	\sMAC\=(?:[0-9a-fA- F]{2}\:){5}[0-9a-fA-F]{2}\s	\sMAC\=((?:[0-9a-fA- F]{2}\:){5}[0-9a-fA-F]{2})\s
SourceIP	\sSRC\=\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\s	\sSRC\=(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\s
Destination IP	\sDST\=\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\s	\sDST\=(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\s
Protocol	\sPROTO\=(TCP\|UDP\|ICMP\|GRE)\s	\sPROTO\=((TCP\|UDP\|ICMP\|GRE))\s
SourcePort	\sSPT\=\d{1,5}\s	\sSPT\=(\d{1,5})\s
DestinationPort	\sDPT\=\d{1,5}\s	\sDPT\=(\d{1,5})\s

Migrate the patterns and capture groups into the log source extensions document.

The following code snippet shows part of the document that you use.

Uploading Extension Documents to JSA

You can create multiple extension documents and then upload them and associated them to various log source types. The logic from the log source extension (LSX) is then used to parse the logs from the unsupported log source.

Extension documents can be stored anywhere before you upload to JSA.

On the Admin tab, click Log Source Extensions.
Click Add.
Assign a name.
If you want to apply this log source extension to more than one instance of a log source type, select the log source type from the available Log Source Type list and click the add arrow to set it as the default.

Setting the default log source type applies the log source extension to all events of a log source type, including those log sources that are automatically discovered.

Ensure that you test the extension for the log source type first to ensure that the events are parsed correctly.
Click Browse to locate the LSX that you saved and then click Upload.

JSA validates the document against the internal XSD and verifies the validity of the document before the extension document is uploaded to the system.
Click Save and close the window.
Associate the log source extension to a log source.
1. From the Admin tab, click Data Sources >Log Sources.
2. Double-click the log source type that you created the extension document for.
3. From the Log Source Extension list, select the document that you created.
4. Click Save and close the window.