Creating a Log Source Extensions Document to get data into JSA

Common Regular Expressions

Use regular expressions to match patterns of text in the log source file. You can scan messages for patterns of letters, numbers, or a combination of both. For example, you can create regular expressions that match source and destination IP addresses, ports, MAC addresses, and more.

The following codes shows several common regular expressions:

\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3} \d{1,5} 
(?:[0-9a-fA-F]{2}\:){5}[0-9a-fA-F]{2} (TCP|UDP|ICMP|GRE) 
\w{3}\s\d{2}\s\d{2}:\d{2}:\d{2} 
\s \t .*? 

The escape character, or "\", is used to denote a literal character. For example, "." character means "any single character" and matches A, B, 1, X, and so on. To match the "." characters, a literal match, you must use "\."

Building Regular Expression Patterns

To create a log source extension, you use regular expressions (regex) to match strings of text from the unsupported log source.

The following example shows a log entry that is referenced in the steps.

May 20 17:24:59 kernel: DROP MAC=5c:31:39:c2:08:00 
SRC=172.29.255.121 DST=10.43.2.10 LEN=351 TOS=0x00 PREC=0x00 TTL=64 ID=9582 
PROTO=UDP SPT=67 DPT=68 LEN=331
May 20 17:24:59 kernel: PASS MAC=5c:14:ab:c4:12:59 
SRC=192.168.50.10 DST=192.168.10.25
LEN=351 TOS=0x00 PREC=0x00 TTL=64 
ID=9583 PROTO=TCP SPT=1057 DPT=80 LEN=331 
May 20 17:24:59 kernel: REJECT 
MAC=5c:ad:3c:54:11:07 SRC=10.10.10.5 DST=192.168.100.25 LEN=351 
TOS=0x00 PREC=0x00 TTL=64 ID=9584 PROTO=TCP SPT=25212 DPT=6881 LEN=331 

1. Visually analyze the unsupported log source to identify unique patterns.

   These patterns are later translated into regular expressions.

2. Find the text strings to match.

   Tip:

   To provide basic error checking, include characters before and after the values to prevent similar values from being unintentionally matched. You can later isolate the actual value from the extra characters.

3. Develop pseudo-code for matching patterns and include the space character to denote the beginning and end of a pattern.

   You can ignore the quotes. In the example log entry, the event names are DROP, PASS, and REJECT. The following list shows the usable event fields.

   • EventName: " kernel: VALUE "

   • SourceMAC: " MAC=VALUE "

   • SourceIp: " SRC=VALUE "

   • DestinationIp: " DST=VALUE "

   • Protocol: " PROTO=VALUE "

   • SourcePort: " SPT=VALUE "

   • DestinationPort: " DPT=VALUE "

4. Substitute a space with the \s regular expression.

   You must use an escape character for non-digit or non-alpha characters. For example, = becomes \= and : becomes \:.

5. Translate the pseduo-code to a regular expression.

   Table 2: Translating Pseudo-code to Regular Expressions

   Field	Pseudo-code	Regular expression
   EventName	" kernel: VALUE "	\skernel\:\s.*?\s
   SourceMAC	" MAC=VALUE "	\sMAC\=(?:[0-9a-fA-F]{2}\:){5}[0-9a-fA-F]{2}\s
   SourceIP	" SRC=VALUE "	\sSRC\=\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\s
   DestinationIp	" DST=VALUE "	\sDST\=\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\s
   Protocol	" PROTO=VALUE "	\sPROTO\=(TCP|UDP|ICMP|GRE)\s
   SourcePort	" SPT=VALUE "	\sSPT\=\d{1,5}\s
   DestinationPort	" DPT=VALUE "	\sDPT\=\d{1,5}\s

6. Specify capture groups.

   A capture group isolates a certain value in the regular expression.

   For example, in the SourcePort pattern in the previous example, you can't pass the entire value since it includes spaces and SRC=<code>. Instead, you specify only the port number by using a capture group. The value in the capture group is what is passed to the relevant field in JSA.

   Insert parenthesis around the values you that you want capture:

   Table 3: Mapping Regular Expressions to Capture Groups for Event Fields

   Field	Regular expression	Capture group
   EventName	\skernel\:\s.*?\s	\skernel\:\s(.*?)\s
   SourceMAC	\sMAC\=(?:[0-9a-fA- F]{2}\:){5}[0-9a-fA-F]{2}\s	\sMAC\=((?:[0-9a-fA- F]{2}\:){5}[0-9a-fA-F]{2})\s
   SourceIP	\sSRC\=\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\s	\sSRC\=(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\s
   Destination IP	\sDST\=\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\s	\sDST\=(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\s
   Protocol	\sPROTO\=(TCP|UDP|ICMP|GRE)\s	\sPROTO\=((TCP|UDP|ICMP|GRE))\s
   SourcePort	\sSPT\=\d{1,5}\s	\sSPT\=(\d{1,5})\s
   DestinationPort	\sDPT\=\d{1,5}\s	\sDPT\=(\d{1,5})\s

7. Migrate the patterns and capture groups into the log source extensions document.

   The following code snippet shows part of the document that you use.

   content_copy zoom_out_map

   <device-extension xmlns="event_parsing/device_extension"> 
   <pattern id="EventNameFWSM_Pattern" xmlns=""><![CDATA[%FWSM[a-zA-Z\-]*\d-(\d{1,6})]]></pattern>
   <pattern id="SourceIp_Pattern" xmlns=""><![CDATA[gaddr (\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})/([\d]{1,5})]]></pattern> 
   <pattern id="SourceIpPreNAT_Pattern" xmlns=""><![CDATA[gaddr (\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})/([\d]{1,5})]]></pattern>
   <pattern id="SourceIpPostNAT_Pattern" xmlns=""><![CDATA[laddr (\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})/([\d]{1,5})]]></pattern>
   <pattern id="DestinationIp_Pattern" xmlns=""><![CDATA[faddr (\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})/([\d]{1,5})]]></pattern>
   <pattern id="Protocol_Pattern" case-insensitive="true" xmlns=""><![CDATA[(TCP|UDP|ICMP|GRE)]]></pattern>
   <pattern id="Protocol_6_Pattern" case-insensitive="true" xmlns=""><![CDATA[protocol=6]]></pattern> 
   <pattern id="EventNameId_Pattern" xmlns=""><![CDATA[(\d{1,6})]]></pattern>
   

Uploading Extension Documents to JSA

You can create multiple extension documents and then upload them and associated them to various log source types. The logic from the log source extension (LSX) is then used to parse the logs from the unsupported log source.

Extension documents can be stored anywhere before you upload to JSA.

1. On the Admin tab, click Log Source Extensions.

2. Click Add.

3. Assign a name.

4. If you want to apply this log source extension to more than one instance of a log source type, select the log source type from the available Log Source Type list and click the add arrow to set it as the default.

   Setting the default log source type applies the log source extension to all events of a log source type, including those log sources that are automatically discovered.

   Ensure that you test the extension for the log source type first to ensure that the events are parsed correctly.

5. Click Browse to locate the LSX that you saved and then click Upload.

   JSA validates the document against the internal XSD and verifies the validity of the document before the extension document is uploaded to the system.

6. Click Save and close the window.

7. Associate the log source extension to a log source.

   1. From the Admin tab, click Data Sources >Log Sources.

   2. Double-click the log source type that you created the extension document for.

   3. From the Log Source Extension list, select the document that you created.

   4. Click Save and close the window.

You can create multiple extension documents and then upload them and associated them to various log source types. The logic from the log source extension (LSX) is then used to parse the logs from the unsupported log source.

Extension documents can be stored anywhere before you upload to JSA.