Troubleshooting Logical Systems
Use the following features to monitor logical systems and troubleshoot the software issues. For more information, see the following topics:
Understanding Security Logs and Logical Systems
Security logs are system log messages that include security events. If a device is configured for logical systems, security logs generated within the context of a logical system use the name logname_LS (for example, IDP_ATTACK_LOG_EVENT_LS). The logical system version of a log has the same set of attributes as the log for devices that are not configured for logical systems. The logical system log includes logical-system-name as the first attribute.
The following security log shows the attributes for the IDP_ATTACK_LOG_EVENT log for a device that is not configured for logical systems:
IDP_ATTACK_LOG_EVENT {
help "IDP attack log";
description "IDP Attack log generated for attack";
type event;
args timestamp message-type source-address source-port destination-address destination-port protocol-name service-name application-name rule-name rulebase-name policy-name repeat-count action threat-severity attack-name nat-source-address nat-source-port nat-destination-address nat-destination-port elapsed-time inbound-bytes outbound-bytes inbound-packets outbound-packets source-zone-name source-interface-name destination-zone-name destination-interface-name packet-log-id message;
severity LOG_INFO;
flag auditable;
edit "2010/10/01 mvr created";
}The following security log shows the attributes for the IDP_ATTACK_LOG_EVENT_LS log for a device that is configured for logical systems (note that logical-system-name is the first attribute):
IDP_ATTACK_LOG_EVENT_LS {
help "IDP attack log";
description "IDP Attack log generated for attack";
type event;
args logical-system-name timestamp message-type source-address source-port destination-address destination-port protocol-name service-name application-name rule-name rulebase-name policy-name repeat-count action threat-severity attack-name nat-source-address nat-source-port nat-destination-address nat-destination-port elapsed-time inbound-bytes outbound-bytes inbound-packets outbound-packets source-zone-name source-interface-name destination-zone-name destination-interface-name packet-log-id message;
severity LOG_INFO;
flag auditable;
edit "2010/10/01 mvr created";
}If a device is configured for logical systems, log parsing scripts might need to be modified because the log name includes the _LS suffix and the logical-system-name attribute can be used to segregate logs by logical system.
If a device is not configured for logical systems, the security logs remain unchanged and scripts built to parse logs do not need any modification.
Only the primary administrator can configure logging at
the [edit security log] hierarchy level. User logical system
administrators cannot configure logging for their logical systems.
Stream mode is a set of logging services that includes:
Off-box logging (SRX Series)
On-box logging and reporting (SRX300, SRX320, SRX340, SRX345, SRX380, SRX550M, SRX1500, SRX4100, SRX4200, and SRX4600 Series)
Per logical system configuration is supported for the off-box logging and logs are handled based on these configurations. Previously the user logical system logs were generated from root logical system. For off-box logging, the logical system logs can only be generated from logical system interface.
Limitations
Each SPU can only support a maximum of 1000 connections for standalone and 500 connections for cluster on the SRX5400, SRX5600, and SRX5800 devices in the Junos OS 18.2R1 release. If all the connections are used up, some connections for user logical systems might not be established.
The error message will be captured in the System Log Explorer.
Configuring On-Box Reporting for logical Systems
SRX Series Firewalls supports different types of reports for logical system users.
Reports are stored locally on the SRX Series Firewall and there is no requirement for separate devices or tools for logs and reports storage. The on-box reports provides a simple and easy-to-use interface for viewing the security logs.
Before you begin:
Understand how to configure security log for logical systems. See Example: Configure Security Log for logical Systems
To configure on-box reporting for logical system:
By default the report option is disabled. The set logical-systems LSYS1 security log mode stream command
is enabled by default.
Example: Configure Security Log for Logical Systems
This example shows how to configure security logs for a logical system.
Requirements
This example uses the following hardware and software components:
An SRX Series Firewall.
Junos OS Release 18.3R1 and later releases.
Before you begin:
Understand how to configure a logical system.
Understand how to create security profiles for the primary logical system. See Understanding Logical Systems Security Profiles (Primary Administrators Only).
Overview
SRX Series Firewalls have two types of log: system logs and security logs. System logs record control plane events, for example, admin login to the device. Security logs, also known as traffic logs, record data plane events regarding specific traffic handling, for example when a security policy denies certain traffic due to some violation of the policy.
The two types of logs can be collected and saved either on-box or off-box. The procedure below explains how to configure security logs in binary format for off-box (stream-mode) logging.
For off-box logging, security logs for a logical system are
sent from a logical system interface. If the logical system interface
is already configured in a routing instance, then configure routing-instance
routing-instance-name at edit logical-systems logical-system-name
security log stream log-stream-name host hierarchy. If the interface
is not configured in routing instance, then no routing instance should
be configured at edit logical-systems logical-system-name security
log stream log-stream-name host hierarchy.
Configuration
CLI Quick Configuration
To quickly configure this example, copy the
following commands, paste them into a text file, remove any line breaks,
change any details necessary to match your network configuration,
copy and paste the commands into the CLI at the [edit] hierarchy
level, and then enter commit from configuration mode.
set logical-systems LSYS1 security log mode stream set logical-systems LSYS1 security log stream LSYS1_s format binary host 1.3.54.22 set logical-systems LSYS1 security log source-address 2.3.45.66 set logical-systems LSYS1 security log transport protocol tls set logical-systems LSYS1 routing-instances LSYS1_ri instance-type virtual-router set logical-systems LSYS1 routing-instances LSYS_ri interface ge-0/0/3 set logical-systems LSYS1 security log stream LSYS1_s host routing-instance LSYS1_ri set system security-profile p1 security-log-stream-number reserved 1 set system security-profile p1 security-log-stream-number maximum 2 set system security-profile LSYS1_profile logical-system LSYS1
Procedure
Step-by-Step Procedure
The following procedure specifies how to configure security logs for a logical system.
Specify the logging mode and the format for the log file. For off-box, stream-mode logging.
[edit ] user@host# set logical-systems LSYS1 security log mode stream user@host# set logical-systems LSYS1 security log stream LSYS1_s format binary host 1.3.54.22
-
For off-box security logging, specify the source address, which identifies the SRX Series Firewall that generated the log messages. The source address is required.
[edit ] user@host# set logical-systems LSYS1 security log source-address 2.3.45.66
Specify the routing instance and define the interface.
[edit ] user@host# set logical-systems LSYS1 routing-instances LSYS1_ri instance-type virtual-router user@host# set logical-systems LSYS1 routing-instances LSYS_ri interface ge-0/0/3
Define routing instance for a logical system.
[edit ] user@host# set logical-systems LSYS1 security log stream LSYS1_s host routing-instance LSYS1_ri
Specify the security log transport protocol for the device.
[edit ] user@host# set logical-systems LSYS1 security log transport protocol tls
Procedure
Step-by-Step Procedure
The following procedure specifies how to configure a security profile for a logical system.
Configure a security profile and specify the number of maximum and reserved policies.
[edit ] user@host# set system security-profile p1 security-log-stream-number reserved 1 user@host# set system security-profile p1 security-log-stream-number maximum 2
Assign the configured security profile to TSYS1.
[edit ] user@host# set system security-profile LSYS1_profile logical-system LSYS1
Results
From configuration mode, confirm your configuration
by entering the show system security-profile, show
logical-systems LSYS1 security log, and show logical-systems
LSYS1 routing-instances commands. If the output does not display
the intended configuration, repeat the configuration instructions
in this example to correct it.
[edit]
user@host# show system security-profile
LSYS1_profile {
logical-system LSYS1;
}
p1 {
security-log-stream-number {
maximum 2;
reserved 1;
}
}
[edit]
user@host# show logical-systems LSYS1 security log
mode stream;
source-address 2.3.45.66;
transport {
protocol tls;
}
stream LSYS1_s {
format binary;
host {
1.3.54.22;
}
}
[edit]
user@host# show logical-systems LSYS1 routing-instances
LSYS1_ri {
instance-type virtual-router;
interface ge-0/0/3.0;
}
If you are done configuring the device, enter commit from configuration mode.
Verification
Verifying Detailed Output for Security Log
Purpose
Verify that the output displays the resource information for all logical systems.
Action
From operational mode, enter the show system security-profile
security-log-stream-number tenant all command.
logical-system name security profile name usage reserved maximum
root-logical-system Default-Profile 0 0
Meaning
The output displays the resource information for logical systems.
Configuring On-Box Binary Security Log Files for Logical System
SRX Series devices support two types of log: system logs and security logs.
The two types of log are collected and saved either on-box or off-box. The following procedure explains how to configure security logs in binary format for on-box (event-mode) logging for logical system.
The following procedure specifies binary format for event-mode security logging, and defines the log filename, path, and log file characteristics for logical system.
Specify the logging mode and the format for the log file. For on-box, event-mode logging:
[edit] user@host# set logical-systems LSYS1 security log mode event user@host# set logical-systems LSYS1 security log format binary
(Optional) Specify a log filename.
[edit] user@host# set logical-systems LSYS1 security log file name security-binary-log
Note:Security log filename is not mandatory. If security log filename is not configured, by default the file bin_messages is created in the /var/log directory.
Confirm your configuration by entering the
show logical-systems LSYS1command.[edit] user@host# show logical-systems LSYS1 security { log { mode event; format binary; file { name security-binary-log; } } }
The following procedure specifies binary format for stream-mode security logging, and defines the log filename and log file characteristics for logical system.
Specify the logging mode and the format for the log file. For on-box, stream-mode logging:
[edit] user@host# set logical-systems LSYS1 security log mode stream user@host# set logical-systems LSYS1 security log stream s1 format binary
(Optional) Specify a log filename.
[edit] user@host# set logical-systems LSYS1 security log stream s1 file name f1.bin
Confirm your configuration by entering the
show logical-systems LSYS1command.[edit] user@host# show logical-systems LSYS1 security { log { mode stream; stream s1 { format binary; file { name f1.bin; } } } }
Configuring Off-Box Binary Security Log Files for Logical System
SRX Series devices support two types of log: system logs and security logs.
The two types of log can be collected and saved either on-box or off-box. The procedure below explains how to configure security logs in binary format for off-box (stream-mode) logging.
The following procedure specifies binary format for stream-mode security logging, and defines the logging mode, source address, and host name characteristics for logical system.
Specify the logging mode and the format for the log file. For off-box, stream-mode logging:
[edit] user@host# set logical-systems LSYS1 security log mode stream s1 format binary
Specify the source address for off-box security logging.
[edit] user@host# set logical-systems LSYS1 security log source-address 100.0.0.1
Specify the host name.
[edit] user@host# set logical-systems LSYS1 security log stream s1 host 100.0.0.2
Confirm your configuration by entering the
show logical-systems LSYS1command.[edit] user@host#show logical-systems LSYS1 security { log { mode stream; source-address 100.0.0.1; stream s1 { format binary; host { 100.0.0.2; } } } }
Understanding Data Path Debugging for Logical Systems
Data path debugging provides tracing and debugging at multiple processing units along the packet-processing path. Data path debugging can also be performed on traffic between logical systems.
Only the primary administrator can configure data path debugging for logical systems at the [edit security datapath-debug] level. User logical system administrators cannot configure data path debugging for their logical systems.
End-to-end event tracing traces the path of a packet from when it enters the device to when it leaves the device. When the primary administrator configures end-to-end event tracing, the trace output contains logical system information.
The primary administrator can also configure tracing for traffic between logical systems. The trace output shows traffic entering and leaving the logical tunnel between logical systems. When the preserve-trace-order option is configured, the trace message is sorted chronologically. In addition to the trace action, other actions such as packet-dump and packet-summary may be configured for traffic between logical systems.
Data path debugging is supported on SRX1400, SRX3400, SRX3600, SRX5400, SRX5600, and SRX5800.
See Also
Performing Tracing for Logical Systems (Primary Administrators Only)
Only the primary administrator can configure data path debugging for logical systems at the root level.
To configure an action profile for a trace or packet capture:
To capture trace messages for logical systems:
Configure the trace capture file.
[edit security datapath-debug] user@host# set traceoptions file e2e.trace user@host# set traceoptions file size 10m
Display the captured trace in operational mode.
user@host> show log e2e.trace Jul 7 09:49:56 09:49:56.417578:CID-00:FPC-01:PIC-00:THREAD_ID-00:FINDEX:0:IIF:75:SEQ:0:TC:0 PIC History: ->C0/F1/P0 NP ingress channel 0 packet Meta: Src: F1/P0 Dst: F0/P0 IP: saddr 10.1.1.2 daddr 30.1.1.2 proto 6 len 500 Jul 7 09:49:56 09:49:55.1414031:CID-00:FPC-00:PIC-00:THREAD_ID-04:FINDEX:0:IIF:75:SEQ:0:TC:1 PIC History: ->C0/F1/P0->C0/F0/P0 LBT pkt, payload: DATA Meta: Src: F1/P0 Dst: F0/P0 IP: saddr 10.1.1.2 daddr 30.1.1.2 proto 6 len 500 ... (Some trace information omitted) ... .Jul 7 09:49:56 09:49:55.1415649:CID-00:FPC-00:PIC-00:THREAD_ID-05:FINDEX:0:IIF:75:SEQ:0:TC:16 PIC History: ->C0/F1/P0->C0/F0/P0->C0/F0/P0->C0/F0/P0->C0/F0/P0 POT pkt, action: POT_SEND payload: DATA Meta: Src: F0/P0 Dst: F1/P0 IP: saddr 10.1.1.2 daddr 30.1.1.2 proto 6 len 500 Jul 7 09:49:56 09:49:56.419274:CID-00:FPC-01:PIC-00:THREAD_ID-00:FINDEX:0:IIF:75:SEQ:0:TC:17 PIC History: ->C0/F1/P0->C0/F0/P0->C0/F0/P0->C0/F0/P0->C0/F0/P0->C0/F1/P0 NP egress channel 0 packet Meta: Src: F0/P0 Dst: F1/P0 IP: saddr 10.1.1.2 daddr 30.1.1.2 proto 6 len 500
Clear the log.
user@host> clear log e2e.trace
To perform packet capture for logical systems:
Configure the packet capture file.
[edit security datapath-debug] user@host# set capture-file e2e.pcap user@host# set capture-file format pcap user@host# set capture-file size 10m user@host# set capture-file world-readable user@host# set capture-file maximum-capture-size 1500
Enter operational mode to start and then stop the packet capture.
user@host> request security datapath-debug capture start user@host> request security datapath-debug capture stop
Note:Packet capture files can be opened and analyzed offline with tcpdump or any packet analyzer that recognizes the libpcap format. You can also use FTP or the Session Control Protocol (SCP) to transfer the packet capture files to an external device.
Disable packet capture from configuration mode.
Note:Disable packet capture before opening the file for analysis or transferring the file to an external device with FTP or SCP. Disabling packet capture ensures that the internal file buffer is flushed and all the captured packets are written to the file.
[edit forwarding-options] user@host# set packet-capture disable
Display the packet capture.
To display the packet capture with the tcpdump utility:
user@host# tcpdump -nr /var/log/e2e.pcap 09:49:55.1413990 C0/F0/P0 event:11(lbt) SEQ:0 IP 10.1.1.2.23451 > 30.1.1.2.12345: S 0:460(460) win 0 09:49:55.1414154 C0/F0/P0 event:11(lbt) SEQ:0 IP 10.1.1.2.23451 > 30.1.1.2.12345: S 0:460(460) win 0 09:49:55.1415062 C0/F0/P0 event:11(lbt) SEQ:0 IP 10.1.1.2.23451 > 30.1.1.2.12345: S 0:460(460) win 0 09:49:55.1415184 C0/F0/P0 event:11(lbt) SEQ:0 IP 10.1.1.2.23451 > 30.1.1.2.12345: S 0:460(460) win 0 09:49:55.1414093 C0/F0/P0 event:12(pot) SEQ:0 IP 10.1.1.2.23451 > 30.1.1.2.12345: S 0:460(460) win 0 09:49:55.1414638 C0/F0/P0 event:12(pot) SEQ:0 IP 10.1.1.2.23451 > 30.1.1.2.12345: S 0:460(460) win 0 09:49:55.1415011 C0/F0/P0 event:12(pot) SEQ:0 IP 10.1.1.2.23451 > 30.1.1.2.12345: S 0:460(460) win 0 09:49:55.1415129 C0/F0/P0 event:12(pot) SEQ:0 IP 10.1.1.2.23451 > 30.1.1.2.12345: S 0:460(460) win 0 09:49:55.1415511 C0/F0/P0 event:12(pot) SEQ:0 IP 10.1.1.2.23451 > 30.1.1.2.12345: S 0:460(460) win 0 09:49:55.1415649 C0/F0/P0 event:12(pot) SEQ:0 IP 10.1.1.2.23451 > 30.1.1.2.12345: S 0:460(460) win 0 09:49:55.1415249 C0/F0/P0 event:18(jexec) SEQ:0 IP 10.1.1.2.23451 > 30.1.1.2.12345: S 0:460(460) win 0 09:49:55.1415558 C0/F0/P0 event:18(jexec) SEQ:0 IP 10.1.1.2.23451 > 30.1.1.2.12345: S 0:460(460) win 0 09:49:55.1414226 C0/F0/P0 event:18(jexec) SEQ:0 IP 10.1.1.2.23451 > 30.1.1.2.12345: S 0:460(460) win 0 09:49:55.1414696 C0/F0/P0 event:18(jexec) SEQ:0 IP 10.1.1.2.23451 > 30.1.1.2.12345: S 0:460(460) win 0 09:49:55.1414828 C0/F0/P0 event:16(lt-enter) SEQ:0 IP 10.1.1.2.23451 > 30.1.1.2.12345: S 0:460(460) win 0 09:49:55.1414919 C0/F0/P0 event:15(lt-leave) SEQ:0 IP 10.1.1.2.23451 > 30.1.1.2.12345: S 0:460(460) win 0 09:49:56.417560 C0/F1/P0 event:1(np-ingress) SEQ:0 IP 10.1.1.2.23451 > 30.1.1.2.12345: S 0:460(460) win 0 09:49:56.419263 C0/F1/P0 event:2(np-egress) SEQ:0 IP 10.1.1.2.23451 > 30.1.1.2.12345: S 0:460(460) win 0
To display the packet capture from CLI operational mode:
user@host> show security datapath-debug capture Packet 1, len 568: (C0/F0/P0/SEQ:0:lbt) 00 00 00 00 00 00 50 c5 8d 0c 99 4a 00 00 0a 01 01 02 08 00 45 60 01 f4 00 00 00 00 40 06 4e 9f 0a 01 01 02 1e 01 01 02 5b 9b 30 39 00 00 00 00 00 00 00 00 50 02 00 00 f8 3c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ac 7a 00 04 00 00 00 00 b3 e3 15 4e 66 93 15 00 04 22 38 02 38 02 00 00 00 01 00 03 0b 00 00 00 50 d0 1a 08 30 de be bf e4 f3 19 08 Packet 2, len 624: (C0/F0/P0/SEQ:0:lbt) aa 35 00 00 00 00 00 00 00 00 00 00 00 03 00 00 00 0a 00 00 00 00 00 00 05 bd 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 50 c5 8d 0c 99 4a 00 00 0a 01 01 02 08 00 45 60 01 f4 00 00 00 00 40 06 4e 9f 0a 01 01 02 ac 7a 00 04 00 00 00 00 b3 e3 15 4e 0a 94 15 00 04 5a 70 02 70 02 00 00 00 03 00 03 0b 00 00 00 50 d0 1a 08 30 de be bf e4 f3 19 08 ... (Packets 3 through 17 omitted) ... Packet 18, len 568: (C0/F1/P0/SEQ:0:np-egress) 00 00 00 04 00 00 00 00 1e 01 01 02 50 c5 8d 0c 99 4b 08 00 45 60 01 f4 00 00 00 00 3e 06 50 9f 0a 01 01 02 1e 01 01 02 5b 9b 30 39 00 00 00 00 00 00 00 00 50 02 00 00 f8 3c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ac 7a 04 00 00 00 00 00 b4 e3 15 4e bf 65 06 00 04 22 38 02 38 02 00 00 00 11 00 03 02 00 00 00 50 d0 1a 08 30 de be bf e4 f3 19 08
user@host> show security datapath-debug counters Datapath debug counters Packet Filter 1: lt-enter Chassis 0 FPC 0 PIC 1: 0 lt-enter Chassis 0 FPC 0 PIC 0: 1 lt-leave Chassis 0 FPC 0 PIC 1: 0 lt-leave Chassis 0 FPC 0 PIC 0: 1 np-egress Chassis 0 FPC 1 PIC 3: 0 np-egress Chassis 0 FPC 1 PIC 1: 0 np-egress Chassis 0 FPC 1 PIC 2: 0 np-egress Chassis 0 FPC 1 PIC 0: 1 pot Chassis 0 FPC 0 PIC 1: 0 pot Chassis 0 FPC 0 PIC 0: 6 np-ingress Chassis 0 FPC 1 PIC 3: 0 np-ingress Chassis 0 FPC 1 PIC 1: 0 np-ingress Chassis 0 FPC 1 PIC 2: 0 np-ingress Chassis 0 FPC 1 PIC 0: 1 lbt Chassis 0 FPC 0 PIC 1: 0 lbt Chassis 0 FPC 0 PIC 0: 4 jexec Chassis 0 FPC 0 PIC 1: 0 jexec Chassis 0 FPC 0 PIC 0: 4
See Also
Troubleshooting DNS Name Resolution in Logical System Security Policies (Primary Administrators Only)
Problem
Description
The address of a hostname in an address book entry that is used in a security policy might fail to resolve correctly.
Cause
Normally, address book entries that contain dynamic hostnames refresh automatically for SRX Series Firewalls. The TTL field associated with a DNS entry indicates the time after which the entry should be refreshed in the policy cache. Once the TTL value expires, the SRX Series Firewall automatically refreshes the DNS entry for an address book entry.
However, if the SRX Series Firewall is unable to obtain a response from the DNS server (for example, the DNS request or response packet is lost in the network or the DNS server cannot send a response), the address of a hostname in an address book entry might fail to resolve correctly. This can cause traffic to drop as no security policy or session match is found.
Solution
The primary administrator can use the show security dns-cache command to display
DNS cache information on the SRX Series Firewall. If the DNS cache information needs
to be refreshed, the primary administrator can use the clear security
dns-cache command.
These commands are only available to the primary administrator on devices that are configured for logical systems. This command is not available in user logical systems or on devices that are not configured for logical systems.