Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

external-header-nav
keyboard_arrow_up
close
keyboard_arrow_left
Juniper Cloud-Native Router User Guide
Table of Contents Expand all
list Table of Contents
file_download PDF
keyboard_arrow_right

Troubleshoot via the vRouter CLI

date_range 27-Nov-24

Accessing the vRouter CLI

You can access the command-line interface (CLI) of the vRouter by accessing the shell of the running vRouter-agent container.
Note: The commands below are provided as an example. The vRouter pod name must be replaced from your environment. The command outputs may differ based on your environment.

List the K8s Pods running on the cluster

content_copy zoom_out_map
kubectl get pods -A
content_copy zoom_out_map
NAMESPACE         NAME                                       READY   STATUS      RESTARTS      AGE
contrail-deploy   contrail-k8s-deployer-7b5dd699b9-nd7xf     1/1     Running     0             41m
contrail          contrail-vrouter-masters-dfxgm             3/3     Running     0             41m
jcnr              kube-crpd-worker-ds-8tnf7                  1/1     Running     0             41m
jcnr              syslog-ng-54749b7b77-v24hq                 1/1     Running     0             41m
kube-system       calico-kube-controllers-57b9767bdb-5wbj6   1/1     Running     2 (92d ago)   129d
kube-system       calico-node-j4m5b                          1/1     Running     2 (92d ago)   129d
kube-system       coredns-8474476ff8-fpw78                   1/1     Running     2 (92d ago)   129d
kube-system       dns-autoscaler-7f76f4dd6-q5vdp             1/1     Running     2 (92d ago)   129d
kube-system       kube-apiserver-5a5s5-node2                 1/1     Running     3 (92d ago)   129d
kube-system       kube-controller-manager-5a5s5-node2        1/1     Running     4 (92d ago)   129d
kube-system       kube-multus-ds-amd64-4zm5k                 1/1     Running     2 (92d ago)   129d
kube-system       kube-proxy-l6xm8                           1/1     Running     2 (92d ago)   129d
kube-system       kube-scheduler-5a5s5-node2                 1/1     Running     4 (92d ago)   129d
kube-system       nodelocaldns-6kwg5                         1/1     Running     2 (92d ago)   129d

Copy the name of the vRouter pod—contrail-vrouter-masters-dfxgm in this example output . You will use the pod name to connect to the running container's shell.

Connect to the vRouter CLI

Issue the kubectl exec command to access the running container's shell:
content_copy zoom_out_map
kubectl exec -n <namespace> -it <pod name> --container <container name> -- bash

where <namespace> identifies the namespace in which the pod is running, <pod name> specificies the name of the pod and the <container name> specifies the name of the container (to be specified if the pod has more than one container).

The vRouter pod has three containers. When the container name is not specified, the command will default to the vrouter-agent container shell. Here is an example:

content_copy zoom_out_map
[root@jcnr-01]# kubectl exec -n contrail -it contrail-vrouter-masters-dfxgm -- bash
Defaulted container "contrail-vrouter-agent" out of: contrail-vrouter-agent, contrail-vrouter-agent-dpdk, 
contrail-vrouter-telemetry-exporter, contrail-init (init), contrail-vrouter-kernel-init-dpdk (init)
[root@jcnr-01 /]# 

At this point, you have connected to the vRouter's CLI.

Troubleshooting via the vRouter CLI

You can run commands in the CLI to learn about the state of the vRouter.

Verify vRouter Interfaces via the vif Command

The command shown below allows you to see which interfaces are present on the vRouter:

content_copy zoom_out_map
vif --list
Vrouter Operation Mode: PureL2
Vrouter Interface Table

Flags: P=Policy, X=Cross Connect, S=Service Chain, Mr=Receive Mirror
       Mt=Transmit Mirror, Tc=Transmit Checksum Offload, L3=Layer 3, L2=Layer 2
       D=DHCP, Vp=Vhost Physical, Pr=Promiscuous, Vnt=Native Vlan Tagged
       Mnp=No MAC Proxy, Dpdk=DPDK PMD Interface, Rfl=Receive Filtering Offload, Mon=Interface is Monitored
       Uuf=Unknown Unicast Flood, Vof=VLAN insert/strip offload, Df=Drop New Flows, L=MAC Learning Enabled
       Proxy=MAC Requests Proxied Always, Er=Etree Root, Mn=Mirror without Vlan Tag, HbsL=HBS Left Intf
       HbsR=HBS Right Intf, Ig=Igmp Trap Enabled, Ml=MAC-IP Learning Enabled, Me=Multicast Enabled

vif0/0      Socket: unix
            Type:Agent HWaddr:00:00:5e:00:01:00
            Vrf:65535 Flags:L2 QOS:-1 Ref:3
            RX queue errors to lcore 0 0 0 0 0 0 0 0 0 0 0 0
            RX packets:0  bytes:0 errors:0
            TX packets:11  bytes:4169 errors:0
            Drops:0

vif0/1      PCI: 0000:00:00.0 (Speed 25000, Duplex 1)
            Type:Physical HWaddr:46:37:1f:de:df:bc
            Vrf:65535 Flags:L2Vof QOS:-1 Ref:8
            RX queue errors to lcore 0 0 0 0 0 0 0 0 0 0 0 0
            Fabric Interface: eth_bond_bond0  Status: UP  Driver: net_bonding
            Slave Interface(0): 0000:3b:02.0  Status: UP  Driver: net_iavf
            Slave Interface(1): 0000:3b:02.1  Status: UP  Driver: net_iavf
            Vlan Mode: Trunk  Vlan: 100 200 300 700-705
            RX packets:0  bytes:0 errors:0
            TX packets:378  bytes:81438 errors:0
            Drops:0

vif0/2      PCI: 0000:3b:0a.0 (Speed 25000, Duplex 1)
            Type:Workload HWaddr:ba:69:c0:b7:1f:ba
            Vrf:0 Flags:L2Vof QOS:-1 Ref:7
            RX queue errors to lcore 0 0 0 0 0 0 0 0 0 0 0 0
            Fabric Interface: 0000:3b:0a.0  Status: UP  Driver: net_iavf
            Vlan Mode: Access  Vlan Id: 700  OVlan Id: 700
            RX packets:378  bytes:81438 errors:2
            TX packets:0  bytes:0 errors:0
            Drops:391

View the running configuration of the vRouter

To see the status of the vRouter, enter the following command in the vRouter CLI:
content_copy zoom_out_map
[root@jcnr-01 /]# ps -eaf | grep vrouter-dpdk
root         116      90 99 Mar30 ?        118-08:05:37 /contrail-vrouter-dpdk --no-daemon --socket-mem=1024 1024 
--allow=0000:5a:02.0 --vdev=eth_bond_bond0,mode=1,socket_id=0,mac=3a:1a:b7:86:1c:4f,primary=0000:5a:02.0,
slave=0000:5a:02.0 --l2_table_size=10240 --yield_option 0 --ddp --l2_mode
root     1134749 1134365  0 16:41 pts/0    00:00:00 grep --color=auto vrouter-dpdk

The output contains several elements.

Table 1: vRouter Status Attributes
Flag Meaning
--l2_mode

The vRouter is running in L2 mode.

--l2_table_size

The current number of entries in the MAC table. The default size is 10240 entries.

--allow=<PCI Id>

The PCI ID of fabric and fabric workload interfaces. More than one ID can appear in the output. These IDs serve as an allowlist.

--ddp

Enable Intel DDP support.

We enable DDP by default in the values.yaml file in the vRouter.

Note:

The Intel XL710 NIC does not support DDP.

View L2 Configuration and Statistics via the purel2cli Command

The purel2cli command is a useful utility to view the Cloud-Native Router L2 configuration and statistics. Start by using the purel2cli --help command.
content_copy zoom_out_map
[root@jcnr-01 /]# purel2cli --help
Usage: purel2cli [--mac show]
           [--vlan show]
           [--vlan get <VLAN_ID>]
           [--acl show <VLAN_ID>]
           [--acl reset-counters <VLAN_ID>]
           [--l2stats get <VIF_ID> <VLAN_ID>]
           [--clear VLAN_ID]
           [--qos classifier/re-write/scheduler <NAME>]
           [--qos cla/rw/sch <NAME>]
           [--nolocal show]
           [--nolocal get <VLAN_ID>]
           [--sock-dir <sock dir>]
           [--help]

The purel2cli --mac show command shows the MAC addresses that the vRouter has dynamically learned.

content_copy zoom_out_map
purel2cli --mac show
==================================================
||  MAC            vlan      port      hit_count||
==================================================
00:01:01:01:01:03  1221      2          1101892
00:01:01:01:01:02  1221      2          1101819
00:01:01:01:01:04  1221      2          1101863
00:01:01:01:01:01  1221      2          1101879
5a:4c:4c:75:90:fe  1250      5          12
Total Mac entries 5

The purel2cli --vlan show command shows the VLANs and associated ports.

content_copy zoom_out_map
purel2cli --vlan show
VLAN      PORT
===============
1201      1,2,3,4,
1202      1,2,3,4,
1203      1,2,3,4,
1204      1,2,3,4,
1205      1,2,3,4,

You can also issue the purel2cli --vlan get command to get more details about the VLAN.

content_copy zoom_out_map
purel2cli --vlan get <vlan-id>

Issue the purel2cli --l2stats command to view L2 statistics. For example:

content_copy zoom_out_map
purel2cli -- l2stats get <virtual_interface_ID> <VLAN_ID>
content_copy zoom_out_map
purel2cli --l2stats get 2 1221Vlan id count: 1
--------------------------------------------------------------------------------
Statistics for vif 2 vlan 1221
--------------------------------------------------------------------------------
                Rx Pkts           Rx Bytes          Tx Pkts            Tx Bytes
Unicast        245344824       48152682842           835552          1667761792
Broadcast              0                 0                0                   0
Multicast              0                 0                0                   0
Flood                  0                 0                0                   0
--------------------------------------------------------------------------------
content_copy zoom_out_map
purel2cli --clear '*'
content_copy zoom_out_map
purel2cli --clear 100
Table 2: purel2cli Command Options for L2 Statistics
Sample Command Function
purel2cli --l2stats get '*' '*' Get statistics for all virtual interfaces (vif) and all VLAN IDs.
purel2cli --l2stats get '*' 100 Get statistics for all vif that are part of VLAN 100
purel2cli --l2stats get 1 '*' Get statistics for all VLANs for which interface 1 is a member
purel2cli --l2stats get 1 100 Get statistics for interface 1 and VLAN 100

The command shows the VLAN to port mapping in the vRouter.You can use the command to see the bridge domain table entry for a specific VLAN: There are several variations of the command that allow you to display and filter L2 statistics in the vRouter. The base form of the command is: . The table below shows the available command options and what they do. It also provides a sample output using one of the options:The following command is an example of the L2 statistics for interface 2 and VLAN 1221:You can clear the statistics from the vRouter with the purel2cli command in the form: . Clears all statistics from all VLANs in the vRouter. Clears all statistics for VLAN id 100.

The dropstats Command

The vRouter tracks the packets that it drops and includes the reason for dropping them. The table below shows the common reasons for vRouter to drop a packet. When you execute the dropstats command, the vRouter does not show a counter if the count for that counter is 0.

Table 3: Dropstats Counters
Counter Name Meaning
L2 bd table drop No interfaces in bridge domain
L2 untag pkt drop Untagged packet arrives on trunk or sub-interface
L2 Invalid Vlan Packet VLAN does not match interface VLAN
L2 Mac Table Full No more entries available in the MAC table
L2 ACL drop Packet matched firewall filter (ACL) drop rule
L2 Src Mac lookup fail Unable to match (or learn) the source MAC address

Example output from the dropstats command looks like:

content_copy zoom_out_map
dropstats
L2 bd table Drop              43
L2 untag pkt drop             716
L2 Invalid Vlan               7288253
Rate limit exceeded           673179706
L2 Mac Table Full             41398787
L2 ACL drop                   8937037
L2 Src Mac lookup fail        247046

The dpdkinfo Command

The dpdkinfo command provides insight into the status and statistics of DPDK. The dpdkinfo command has many options. The following sections describe the available options and the example output from the dpdkinfo command. You can run the dpdkinfo command only from within the vRouter-agent CLI.

content_copy zoom_out_map
dpdkinfo --help
Usage: dpdkinfo [--help]
                 --version|-v                                                      Show DPDK Version
                 --bond|-b                                                         Show Master/Slave bond information
                 --lacp|-l     <all/conf>                                          Show LACP information from DPDK
                 --mempool|-m  <all/<mempool-name>>                                Show Mempool information
                 --stats|-n    <vif index value>                                   Show Stats information
                 --xstats|-x   <vif index value>                                   Show Extended Stats information
                 --lcore|-c                                                        Show Lcore information
                 --app|-a                                                          Show App information
                 --ddp|-d      <list> <list-flow>						   Show DDP information for X710 NIC
                 --rx_vlan|-z  <value>                                             Show VLan information
       Optional: --buffsz      <value>                                             Send output buffer size (less than 1000Mb)

The command dpdkinfo -c shows the Lcores assigned to DPDK VF fabric interfaces and the queue ID for each interface.

content_copy zoom_out_map
dpdkinfo -c
No. of forwarding lcores: 4

Lcore 10:
	Interface: 0000:18:01.1        Queue ID: 0
	Interface: 0000:18:0d.1        Queue ID: 0
	Interface: 0000:86:00.0        Queue ID: 0

Lcore 11:
	Interface: 0000:18:01.1        Queue ID: 1
	Interface: 0000:18:0d.1        Queue ID: 1
	Interface: 0000:86:00.0        Queue ID: 1

Lcore 12:
	Interface: 0000:18:01.1        Queue ID: 2
	Interface: 0000:18:0d.1        Queue ID: 2
	Interface: 0000:86:00.0        Queue ID: 2

Lcore 13:
	Interface: 0000:18:01.1        Queue ID: 3
	Interface: 0000:18:0d.1        Queue ID: 3
	Interface: 0000:86:00.0        Queue ID: 3

The command dpdkinfo -m all shows all of the memory pool information.

content_copy zoom_out_map
dpdkinfo -m all
---------------------------------------------------
Name			Size	Used	Available
---------------------------------------------------
rss_mempool         	16384	1549	14835
frag_direct_mempool 	4096	0	4096
frag_indirect_mempool	4096	0	4096
packet_mbuf_pool    	8192	2	8190

The command dpdkinfo -n 3 displays statistical information for a specific interface.

content_copy zoom_out_map
dpdkinfo -n 3
Interface Info(0000:18:0d.1):
RX Device Packets:6710, Bytes:1367533, Errors:0, Nombufs:0
Dropped RX Packets:0
TX Device Packets:0, Bytes:0, Errors:0
Queue Rx:
      Tx:
      Rx Bytes:
      Tx Bytes:
      Errors:

The command dpdkinfo -x 3 displays extended statistical information for a specific interface.

content_copy zoom_out_map
dpdkinfo -x 3
Driver Name:net_iavf
Interface Info:0000:18:0d.1
Rx Packets:
	rx_good_packets: 6701
	rx_unicast_packets: 0
	rx_multicast_packets: 2987
	rx_broadcast_packets: 3714
	rx_dropped_packets: 0
Tx Packets:
	tx_good_packets: 0
	tx_unicast_packets: 0
	tx_multicast_packets: 0
	tx_broadcast_packets: 0
	tx_dropped_packets: 0
Rx Bytes:
	rx_good_bytes: 1365696
Tx Bytes:
	tx_good_bytes: 0
Errors:
	rx_missed_errors: 0
	rx_errors: 0
	tx_errors: 0
	rx_mbuf_allocation_errors: 0
	inline_ipsec_crypto_ierrors: 0
	inline_ipsec_crypto_ierrors_sad_lookup: 0
	inline_ipsec_crypto_ierrors_not_processed: 0
	inline_ipsec_crypto_ierrors_icv_fail: 0
	inline_ipsec_crypto_ierrors_length: 0
Others:
	inline_ipsec_crypto_ipackets: 0
---------------------------------------------------------------------

The rt and nh Commands

Use the rt command to display all routes in a VRF. The nh command enables you to inspect the next hops that are known by the vRouter. Next hops tell the vRouter the next location to send a packet in the path to its final destination.

For example, for IPv4 traffic:

content_copy zoom_out_map
rt --get 172.68.20.2/32 --vrf 4
Match 172.68.20.2/32 in vRouter inet4 table 0/4/unicast
Flags: L=Label Valid, P=Proxy ARP, T=Trap ARP, F=Flood ARP, Ml=MAC-IP learnt route
vRouter inet4 routing table 0/4/unicast
Destination           PPL        Flags        Label         Nexthop    Stitched MAC(Index)
172.68.20.2/32          0          LPT         16            193        -
content_copy zoom_out_map
nh --get 193
Id:193        Type:Tunnel         Fmly: AF_INET  Rid:0  Ref_cnt:264        Vrf:0
                Flags:Valid, Policy, MPLSoUDP, Etree Root,
Oif:4 Len:14 Data:88 e6 4b 09 7d 46 40 a6 b7 2c a4 48 08 00 Sip:1.1.1.35 Dip:1.1.24.24

For example, for IPv6 traffic:

content_copy zoom_out_map
rt --get 2001:172:68:20::/64 --vrf 4 --family inet6
Match 2001:172:68:20::/64 in vRouter inet6 table 0/4/unicast
Flags: L=Label Valid, P=Proxy ARP, T=Trap ARP, F=Flood ARP, Ml=MAC-IP learnt route
vRouter inet6 routing table 0/4/unicast
Destination           PPL        Flags        Label         Nexthop    Stitched MAC(Index)
2001:172:68:20::/64     0          LPT         16            193        -
content_copy zoom_out_map
nh --get 193
Id:193        Type:Tunnel         Fmly: AF_INET  Rid:0  Ref_cnt:264        Vrf:0
              Flags:Valid, Policy, MPLSoUDP, Etree Root,
Oif:4 Len:14 Data:88 e6 4b 09 7d 46 40 a6 b7 2c a4 48 08 00 Sip:1.1.1.35 Dip:1.1.24.24

The flow Command

Use the flow command to display all active flows in a system. For example:

content_copy zoom_out_map
flow -l --match 169.83.47.170:9398
Flow table(size 161218560, entries 629760)

Entries: Created 162630 Added 162614 Deleted 35136 Changed 35202Processed 162630 Used Overflow entries 0
(Created Flows/CPU: 0 0 0 0 0 0 0 0 0 0 241 546 15 161828)(oflows 0)

Action:F=Forward, D=Drop N=NAT(S=SNAT, D=DNAT, Ps=SPAT, Pd=DPAT, L=Link Local Port)
 Other:K(nh)=Key_Nexthop, S(nh)=RPF_Nexthop
 Flags:E=Evicted, Ec=Evict Candidate, N=New Flow, M=Modified Dm=Delete Marked
TCP(r=reverse):S=SYN, F=FIN, R=RST, C=HalfClose, E=Established, D=Dead
 Stats:Packets/Bytes

Listing flows matching ([169.83.47.170]:9398)

    Index                Source:Port/Destination:Port                      Proto(V)
-----------------------------------------------------------------------------------
   328196<=>524233       169.83.47.170:9398                                  6 (2)
                         172.68.20.20:2159
(Gen: 3, K(nh):206, Action:F, Flags:, TCP:, E:1, QOS:-1, S(nh):206,  Stats:6/360,
 SPort 63929, TTL 0, Sinfo 38.0.0.0)

   524233<=>328196       172.68.20.20:2159                                   6 (2)
                         169.83.47.170:9398
(Gen: 3, K(nh):206, Action:F, Flags:, TCP:, QOS:-1, S(nh):250,  Stats:0/0,
 SPort 60311, TTL 0, Sinfo 0.0.0.0)
external-footer-nav