Troubleshoot via the vRouter CLI
SUMMARY Read this topic to learn about the various troubleshooting commands available in the vRouter CLI.
The following commands are covered in this topic:
Accessing the vRouter CLI
You can access the command-line interface (CLI) of the vRouter by accessing the shell of the running vRouter-agent container.List the K8s Pods running on the cluster
kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE contrail-deploy contrail-k8s-deployer-7b5dd699b9-nd7xf 1/1 Running 0 41m contrail contrail-vrouter-masters-dfxgm 3/3 Running 0 41m jcnr kube-crpd-worker-ds-8tnf7 1/1 Running 0 41m jcnr syslog-ng-54749b7b77-v24hq 1/1 Running 0 41m kube-system calico-kube-controllers-57b9767bdb-5wbj6 1/1 Running 2 (92d ago) 129d kube-system calico-node-j4m5b 1/1 Running 2 (92d ago) 129d kube-system coredns-8474476ff8-fpw78 1/1 Running 2 (92d ago) 129d kube-system dns-autoscaler-7f76f4dd6-q5vdp 1/1 Running 2 (92d ago) 129d kube-system kube-apiserver-5a5s5-node2 1/1 Running 3 (92d ago) 129d kube-system kube-controller-manager-5a5s5-node2 1/1 Running 4 (92d ago) 129d kube-system kube-multus-ds-amd64-4zm5k 1/1 Running 2 (92d ago) 129d kube-system kube-proxy-l6xm8 1/1 Running 2 (92d ago) 129d kube-system kube-scheduler-5a5s5-node2 1/1 Running 4 (92d ago) 129d kube-system nodelocaldns-6kwg5 1/1 Running 2 (92d ago) 129d
Copy
the name of the vRouter pod—contrail-vrouter-masters-dfxgm
in this example
output . You will use the pod name to connect to the running container's
shell.
Connect to the vRouter CLI
Issue thekubectl exec
command to access the running container's
shell:kubectl exec -n <namespace> -it <pod name> --container <container name> -- bash
where <namespace> identifies the namespace in which the pod is running, <pod name> specificies the name of the pod and the <container name> specifies the name of the container (to be specified if the pod has more than one container).
The vRouter pod has three containers. When the container name is not specified, the command will default to the vrouter-agent container shell. Here is an example:
[root@jcnr-01]# kubectl exec -n contrail -it contrail-vrouter-masters-dfxgm -- bash Defaulted container "contrail-vrouter-agent" out of: contrail-vrouter-agent, contrail-vrouter-agent-dpdk, contrail-vrouter-telemetry-exporter, contrail-init (init), contrail-vrouter-kernel-init-dpdk (init) [root@jcnr-01 /]#
At this point, you have connected to the vRouter's CLI.
Troubleshooting via the vRouter CLI
You can run commands in the CLI to learn about the state of the vRouter.
- Verify vRouter Interfaces via the vif Command
- View the running configuration of the vRouter
- View L2 Configuration and Statistics via the purel2cli Command
- The dropstats Command
- The dpdkinfo Command
- The rt and nh Commands
- The flow Command
Verify vRouter Interfaces via the vif Command
The command shown below allows you to see which interfaces are present on the vRouter:
vif --list Vrouter Operation Mode: PureL2 Vrouter Interface Table Flags: P=Policy, X=Cross Connect, S=Service Chain, Mr=Receive Mirror Mt=Transmit Mirror, Tc=Transmit Checksum Offload, L3=Layer 3, L2=Layer 2 D=DHCP, Vp=Vhost Physical, Pr=Promiscuous, Vnt=Native Vlan Tagged Mnp=No MAC Proxy, Dpdk=DPDK PMD Interface, Rfl=Receive Filtering Offload, Mon=Interface is Monitored Uuf=Unknown Unicast Flood, Vof=VLAN insert/strip offload, Df=Drop New Flows, L=MAC Learning Enabled Proxy=MAC Requests Proxied Always, Er=Etree Root, Mn=Mirror without Vlan Tag, HbsL=HBS Left Intf HbsR=HBS Right Intf, Ig=Igmp Trap Enabled, Ml=MAC-IP Learning Enabled, Me=Multicast Enabled vif0/0 Socket: unix Type:Agent HWaddr:00:00:5e:00:01:00 Vrf:65535 Flags:L2 QOS:-1 Ref:3 RX queue errors to lcore 0 0 0 0 0 0 0 0 0 0 0 0 RX packets:0 bytes:0 errors:0 TX packets:11 bytes:4169 errors:0 Drops:0 vif0/1 PCI: 0000:00:00.0 (Speed 25000, Duplex 1) Type:Physical HWaddr:46:37:1f:de:df:bc Vrf:65535 Flags:L2Vof QOS:-1 Ref:8 RX queue errors to lcore 0 0 0 0 0 0 0 0 0 0 0 0 Fabric Interface: eth_bond_bond0 Status: UP Driver: net_bonding Slave Interface(0): 0000:3b:02.0 Status: UP Driver: net_iavf Slave Interface(1): 0000:3b:02.1 Status: UP Driver: net_iavf Vlan Mode: Trunk Vlan: 100 200 300 700-705 RX packets:0 bytes:0 errors:0 TX packets:378 bytes:81438 errors:0 Drops:0 vif0/2 PCI: 0000:3b:0a.0 (Speed 25000, Duplex 1) Type:Workload HWaddr:ba:69:c0:b7:1f:ba Vrf:0 Flags:L2Vof QOS:-1 Ref:7 RX queue errors to lcore 0 0 0 0 0 0 0 0 0 0 0 0 Fabric Interface: 0000:3b:0a.0 Status: UP Driver: net_iavf Vlan Mode: Access Vlan Id: 700 OVlan Id: 700 RX packets:378 bytes:81438 errors:2 TX packets:0 bytes:0 errors:0 Drops:391
View the running configuration of the vRouter
To see the status of the vRouter, enter the following command in the vRouter CLI:
[root@jcnr-01 /]# ps -eaf | grep vrouter-dpdk root 116 90 99 Mar30 ? 118-08:05:37 /contrail-vrouter-dpdk --no-daemon --socket-mem=1024 1024 --allow=0000:5a:02.0 --vdev=eth_bond_bond0,mode=1,socket_id=0,mac=3a:1a:b7:86:1c:4f,primary=0000:5a:02.0, slave=0000:5a:02.0 --l2_table_size=10240 --yield_option 0 --ddp --l2_mode root 1134749 1134365 0 16:41 pts/0 00:00:00 grep --color=auto vrouter-dpdk
The output contains several elements.
Flag | Meaning |
---|---|
--l2_mode |
The vRouter is running in L2 mode. |
--l2_table_size |
The current number of entries in the MAC table. The default size is 10240 entries. |
--allow=<PCI Id> |
The PCI ID of fabric and fabric workload interfaces. More than one ID can appear in the output. These IDs serve as an allowlist. |
--ddp |
Enable Intel DDP support. We enable DDP by default in the values.yaml file in the vRouter. Note:
The Intel XL710 NIC does not support DDP. |
View L2 Configuration and Statistics via the purel2cli Command
The purel2cli
command is a useful utility to view the JCNR L2
configuration and statistics. Start by using the
command.purel2cli
--help
[root@jcnr-01 /]# purel2cli --help Usage: purel2cli [--mac show] [--vlan show] [--vlan get <VLAN_ID>] [--acl show <VLAN_ID>] [--acl reset-counters <VLAN_ID>] [--l2stats get <VIF_ID> <VLAN_ID>] [--clear VLAN_ID] [--qos classifier/re-write/scheduler <NAME>] [--qos cla/rw/sch <NAME>] [--nolocal show] [--nolocal get <VLAN_ID>] [--sock-dir <sock dir>] [--help]
The
command shows the MAC
addresses that the vRouter has dynamically learned.purel2cli --mac show
purel2cli --mac show ================================================== || MAC vlan port hit_count|| ================================================== 00:01:01:01:01:03 1221 2 1101892 00:01:01:01:01:02 1221 2 1101819 00:01:01:01:01:04 1221 2 1101863 00:01:01:01:01:01 1221 2 1101879 5a:4c:4c:75:90:fe 1250 5 12 Total Mac entries 5
The purel2cli --vlan show command shows the VLANs and associated ports.
purel2cli --vlan show VLAN PORT =============== 1201 1,2,3,4, 1202 1,2,3,4, 1203 1,2,3,4, 1204 1,2,3,4, 1205 1,2,3,4,
You can also issue the purel2cli --vlan get
command to get more
details about the VLAN.
purel2cli --vlan get <vlan-id>
Issue the purel2cli --l2stats
command to view L2 statistics. For
example:
purel2cli -- l2stats get <virtual_interface_ID> <VLAN_ID>
purel2cli --l2stats get 2 1221Vlan id count: 1 -------------------------------------------------------------------------------- Statistics for vif 2 vlan 1221 -------------------------------------------------------------------------------- Rx Pkts Rx Bytes Tx Pkts Tx Bytes Unicast 245344824 48152682842 835552 1667761792 Broadcast 0 0 0 0 Multicast 0 0 0 0 Flood 0 0 0 0 --------------------------------------------------------------------------------
purel2cli --clear '*'
purel2cli --clear 100
Sample Command | Function |
---|---|
purel2cli --l2stats get '*' '*' |
Get statistics for all virtual interfaces (vif) and all VLAN IDs. |
purel2cli --l2stats get '*' 100 |
Get statistics for all vif that are part of VLAN 100 |
purel2cli --l2stats get 1 '*' |
Get statistics for all VLANs for which interface 1 is a member |
purel2cli --l2stats get 1 100 |
Get statistics for interface 1 and VLAN 100 |
The command shows the VLAN to port mapping in the vRouter.You can use the command to see the bridge domain table entry for a specific VLAN: There are several variations of the command that allow you to display and filter L2 statistics in the vRouter. The base form of the command is: . The table below shows the available command options and what they do. It also provides a sample output using one of the options:The following command is an example of the L2 statistics for interface 2 and VLAN 1221:You can clear the statistics from the vRouter with the purel2cli command in the form: . Clears all statistics from all VLANs in the vRouter. Clears all statistics for VLAN id 100.
The dropstats Command
The vRouter tracks the packets that it drops and includes the reason for dropping them. The table below shows the common reasons for vRouter to drop a packet. When you execute the dropstats command, the vRouter does not show a counter if the count for that counter is 0.
Counter Name | Meaning |
---|---|
L2 bd table drop |
No interfaces in bridge domain |
L2 untag pkt drop |
Untagged packet arrives on trunk or sub-interface |
L2 Invalid Vlan |
Packet VLAN does not match interface VLAN |
L2 Mac Table Full |
No more entries available in the MAC table |
L2 ACL drop |
Packet matched firewall filter (ACL) drop rule |
L2 Src Mac lookup fail |
Unable to match (or learn) the source MAC address |
Example output from the dropstats command looks like:
dropstats L2 bd table Drop 43 L2 untag pkt drop 716 L2 Invalid Vlan 7288253 Rate limit exceeded 673179706 L2 Mac Table Full 41398787 L2 ACL drop 8937037 L2 Src Mac lookup fail 247046
The dpdkinfo Command
The dpdkinfo command provides insight into the status and statistics of DPDK. The dpdkinfo command has many options. The following sections describe the available options and the example output from the dpdkinfo command. You can run the dpdkinfo command only from within the vRouter-agent CLI.
dpdkinfo --help Usage: dpdkinfo [--help] --version|-v Show DPDK Version --bond|-b Show Master/Slave bond information --lacp|-l <all/conf> Show LACP information from DPDK --mempool|-m <all/<mempool-name>> Show Mempool information --stats|-n <vif index value> Show Stats information --xstats|-x <vif index value> Show Extended Stats information --lcore|-c Show Lcore information --app|-a Show App information --ddp|-d <list> <list-flow> Show DDP information for X710 NIC --rx_vlan|-z <value> Show VLan information Optional: --buffsz <value> Send output buffer size (less than 1000Mb)
The command dpdkinfo -c
shows the Lcores assigned to DPDK VF
fabric interfaces and the queue ID for each interface.
dpdkinfo -c No. of forwarding lcores: 4 Lcore 10: Interface: 0000:18:01.1 Queue ID: 0 Interface: 0000:18:0d.1 Queue ID: 0 Interface: 0000:86:00.0 Queue ID: 0 Lcore 11: Interface: 0000:18:01.1 Queue ID: 1 Interface: 0000:18:0d.1 Queue ID: 1 Interface: 0000:86:00.0 Queue ID: 1 Lcore 12: Interface: 0000:18:01.1 Queue ID: 2 Interface: 0000:18:0d.1 Queue ID: 2 Interface: 0000:86:00.0 Queue ID: 2 Lcore 13: Interface: 0000:18:01.1 Queue ID: 3 Interface: 0000:18:0d.1 Queue ID: 3 Interface: 0000:86:00.0 Queue ID: 3
The command dpdkinfo -m all
shows all of the memory pool
information.
dpdkinfo -m all --------------------------------------------------- Name Size Used Available --------------------------------------------------- rss_mempool 16384 1549 14835 frag_direct_mempool 4096 0 4096 frag_indirect_mempool 4096 0 4096 packet_mbuf_pool 8192 2 8190
The command dpdkinfo -n 3
displays statistical information for a
specific interface.
dpdkinfo -n 3 Interface Info(0000:18:0d.1): RX Device Packets:6710, Bytes:1367533, Errors:0, Nombufs:0 Dropped RX Packets:0 TX Device Packets:0, Bytes:0, Errors:0 Queue Rx: Tx: Rx Bytes: Tx Bytes: Errors:
The command dpdkinfo -x 3
displays extended statistical
information for a specific interface.
dpdkinfo -x 3 Driver Name:net_iavf Interface Info:0000:18:0d.1 Rx Packets: rx_good_packets: 6701 rx_unicast_packets: 0 rx_multicast_packets: 2987 rx_broadcast_packets: 3714 rx_dropped_packets: 0 Tx Packets: tx_good_packets: 0 tx_unicast_packets: 0 tx_multicast_packets: 0 tx_broadcast_packets: 0 tx_dropped_packets: 0 Rx Bytes: rx_good_bytes: 1365696 Tx Bytes: tx_good_bytes: 0 Errors: rx_missed_errors: 0 rx_errors: 0 tx_errors: 0 rx_mbuf_allocation_errors: 0 inline_ipsec_crypto_ierrors: 0 inline_ipsec_crypto_ierrors_sad_lookup: 0 inline_ipsec_crypto_ierrors_not_processed: 0 inline_ipsec_crypto_ierrors_icv_fail: 0 inline_ipsec_crypto_ierrors_length: 0 Others: inline_ipsec_crypto_ipackets: 0 ---------------------------------------------------------------------
The rt and nh Commands
Use the rt
command to display all routes in a VRF. The
nh
command enables you to inspect the next hops that are known by the
vRouter. Next hops tell the vRouter the next location to send a packet in the path to its
final destination.
For example, for IPv4 traffic:
rt --get 172.68.20.2/32 --vrf 4 Match 172.68.20.2/32 in vRouter inet4 table 0/4/unicast Flags: L=Label Valid, P=Proxy ARP, T=Trap ARP, F=Flood ARP, Ml=MAC-IP learnt route vRouter inet4 routing table 0/4/unicast Destination PPL Flags Label Nexthop Stitched MAC(Index) 172.68.20.2/32 0 LPT 16 193 -
nh --get 193 Id:193 Type:Tunnel Fmly: AF_INET Rid:0 Ref_cnt:264 Vrf:0 Flags:Valid, Policy, MPLSoUDP, Etree Root, Oif:4 Len:14 Data:88 e6 4b 09 7d 46 40 a6 b7 2c a4 48 08 00 Sip:1.1.1.35 Dip:1.1.24.24
For example, for IPv6 traffic:
rt --get 2001:172:68:20::/64 --vrf 4 --family inet6 Match 2001:172:68:20::/64 in vRouter inet6 table 0/4/unicast Flags: L=Label Valid, P=Proxy ARP, T=Trap ARP, F=Flood ARP, Ml=MAC-IP learnt route vRouter inet6 routing table 0/4/unicast Destination PPL Flags Label Nexthop Stitched MAC(Index) 2001:172:68:20::/64 0 LPT 16 193 -
nh --get 193 Id:193 Type:Tunnel Fmly: AF_INET Rid:0 Ref_cnt:264 Vrf:0 Flags:Valid, Policy, MPLSoUDP, Etree Root, Oif:4 Len:14 Data:88 e6 4b 09 7d 46 40 a6 b7 2c a4 48 08 00 Sip:1.1.1.35 Dip:1.1.24.24
The flow Command
Use the flow
command to display all active flows in a system. For
example:
flow -l --match 169.83.47.170:9398 Flow table(size 161218560, entries 629760) Entries: Created 162630 Added 162614 Deleted 35136 Changed 35202Processed 162630 Used Overflow entries 0 (Created Flows/CPU: 0 0 0 0 0 0 0 0 0 0 241 546 15 161828)(oflows 0) Action:F=Forward, D=Drop N=NAT(S=SNAT, D=DNAT, Ps=SPAT, Pd=DPAT, L=Link Local Port) Other:K(nh)=Key_Nexthop, S(nh)=RPF_Nexthop Flags:E=Evicted, Ec=Evict Candidate, N=New Flow, M=Modified Dm=Delete Marked TCP(r=reverse):S=SYN, F=FIN, R=RST, C=HalfClose, E=Established, D=Dead Stats:Packets/Bytes Listing flows matching ([169.83.47.170]:9398) Index Source:Port/Destination:Port Proto(V) ----------------------------------------------------------------------------------- 328196<=>524233 169.83.47.170:9398 6 (2) 172.68.20.20:2159 (Gen: 3, K(nh):206, Action:F, Flags:, TCP:, E:1, QOS:-1, S(nh):206, Stats:6/360, SPort 63929, TTL 0, Sinfo 38.0.0.0) 524233<=>328196 172.68.20.20:2159 6 (2) 169.83.47.170:9398 (Gen: 3, K(nh):206, Action:F, Flags:, TCP:, QOS:-1, S(nh):250, Stats:0/0, SPort 60311, TTL 0, Sinfo 0.0.0.0)