close

keyboard_arrow_left

Table of Contents Expand all

play_arrow Monitoring and Troubleshooting Contrail
play_arrow Contrail Commands and APIs
- play_arrow Contrail Commands
- play_arrow Contrail Application Programming Interfaces (APIs)

list Table of Contents

English

Tracing the vRouter Packet Path

date_range 07-Jun-23

arrow_backward

arrow_forward

Contrail Networking vRouter is the component that takes packets from virtual machines (VM)s and forwards them to their destinations. Tracing is a useful tool for debugging the packet path.

In this topic, we trace the vRouter packet path in the following use cases:

Unicast Packet Path - Intra-VN

This procedure steps through debugging the unicast packet path for intra-virtual network (intra-VN) traffic from VM1 to VM2 (on same compute node) and VM3 (on different compute node). In this example, the VMs listed are in the same subnet 10.1.1.0/24. Intra-VN traffic is within the same virtual network.

VM1	IP address `10.1.1.5/32` (Compute 1)
VM2	IP address `10.1.1.6/32` (Compute 1)
VM3	IP address `10.1.1.7/32` (Compute 2)

Intra-Compute Use Case
Inter-Compute Use Case

Intra-Compute Use Case

Discover the vif interfaces corresponding to the virtual machine interfaces (VMI)s of the VM by using the command:
```
vif --list
```
You can also discover the vif interfaces by entering the introspect URL.

Example:
```
http://10.1.1.5:8085/Snh_ItfReq?name=&type=&uuid=&vn=&mac=&ipv4_address=&ipv6_address=&parent_uuid=&ip_active=&ip6_active=&l2_active=
```
Note:
Replace the IP address with the actual compute IP address in the introspect HTTP URL.

Run the vif --get <index> command to verify the virtual routing and forwarding (VRF) and Policy flags are set in the vRouter interface (VIF).

Example output verifying flags for each vif:

content_copy zoom_out_map
vif0/4      OS: tapdd789d34-27
            Type:Virtual HWaddr:00:00:5e:00:01:00 IPaddr:10.1.1.5
            Vrf:2 Mcast Vrf:2 Flags:PL3L2Er QOS:-1 Ref:6
            RX packets:30  bytes:8676 errors:0
            TX packets:170  bytes:7140 errors:0
            ISID: 0 Bmac: 02:dd:78:9d:34:27
            Drops:81

vif0/6      OS: tapaedbc037-bf
            Type:Virtual HWaddr:00:00:5e:00:01:00 IPaddr:10.1.1.6
            Vrf:2 Mcast Vrf:2 Flags:PL3L2Er QOS:-1 Ref:6
            RX packets:96316  bytes:4858043 errors:0
            TX packets:96562  bytes:4884177 errors:0
            ISID: 0 Bmac: 02:ae:db:c0:37:bf
            Drops:2

Run the following command to display all of the entries from the bridge table:

content_copy zoom_out_map
rt --dump <vrf id> --family bridge

Example:

content_copy zoom_out_map
rt --dump 2 --family bridge
Flags: L=Label Valid, Df=DHCP flood, Mm=Mac Moved, L2c=L2 Evpn Control Word, N=New Entry, Ec=EvpnControlProcessing
vRouter bridge table 0/2
Index       DestMac                  Flags           Label/VNID      Nexthop           Stats
31264       0:0:5e:0:1:0            Df                    -                 3          206834
79784       2:dd:78:9d:34:27        Df                    -                44               4
112924      ff:ff:ff:ff:ff:ff       LDf                   5                48              35
115240      2:0:0:0:0:2             Df                    -                12               0
169408      2:ae:db:c0:37:bf        Df                    -                25               1
183944      2:99:ef:64:96:e1        LDf                  27                20               0
205564      2:0:0:0:0:1             Df                    -                12               0
252380      0:25:90:c5:58:94        Df                    -                 3               0

Highlighted in the example is the destination MAC address of the destination VM in the bridge table and the next-hop identifier associated with it.

Run nh --get <nh id> to display the next-hop details.

Example:
```
nh --get 25
Id:25         Type:Encap          Fmly:AF_BRIDGE  Rid:0  Ref_cnt:3          Vrf:2
              Flags:Valid, Policy, Etree Root,
              EncapFmly:0806 Oif:6 Len:14
              Encap Data: 02 ae db c0 37 bf 00 00 5e 00 01 00 08 00
```
In the example, Oif:6 is the OIF index in the next hop which is the outgoing interface for the packet. The Encap Data corresponds to the L2 encapsulation that is added to the IP packet before the packet is forwarded to the outgoing interface.

Run vif --get <oifindex> to get the outgoing VIF details.

Example:

content_copy zoom_out_map
vif --get 6
vif0/6      OS: tapaedbc037-bf
            Type:Virtual HWaddr:00:00:5e:00:01:00 IPaddr:10.1.1.6
            Vrf:2 Mcast Vrf:2 Flags:PL3L2Er QOS:-1 Ref:6
            RX packets:96935  bytes:4892936 errors:0
            TX packets:97235  bytes:4921004 errors:0
            ISID: 0 Bmac: 02:ae:db:c0:37:bf
            Drops:2

The received packet RX and transmitted packet TX counters for the corresponding VIF interfaces are incremented when traffic is flowing through.

Run the flow -l command to list the flows created. If the Policy flag is enabled on the VIFs, a flow is created as shown in the example.

Example: Ping 10.1.1.6 from 10.1.1.5.

content_copy zoom_out_map
flow -l
Index                Source:Port/Destination:Port                      Proto(V)
-----------------------------------------------------------------------------------------
   876<=>1020         10.1.1.6:1599                                       1 (2)
                      10.1.1.5:0
(Gen: 12, K(nh):21, Action:F, Flags:, QOS:-1, S(nh):21,  Stats:9/882,
 SPort 54920, TTL 0, Sinfo 6.0.0.0)

  1020<=>876          10.1.1.5:1599                                       1 (2)
                      10.1.1.6:0
(Gen: 2, K(nh):29, Action:F, Flags:, QOS:-1, S(nh):29,  Stats:9/882,  SPort 58271,
 TTL 0, Sinfo 4.0.0.0)

The statistics in the forward and reverse flow are incremented when traffic is flowing through. If statistics are not getting incremented for a particular flow, that can indicate a potential problem in that direction. The flow action should be F or N for the packets to be forwarded or NATed out. A flow action of D indicates that packets will be dropped.

Run the vrouter_agent_debug script to collect all of the relevant logs.

Inter-Compute Use Case

In an inter-compute case, the next-hop lookup points to the tunnel that takes the packet to the other compute node. The bridge entry will also indicate the Label/VNID added to the packet during encapsulation. Inter-compute traffic is between VMs on different compute nodes.

For Compute 1:

Discover the vif interfaces corresponding to the virtual machine interfaces (VMI)s of the VM by using the command:
```
vif --list
```
You can also discover the vif interfaces by entering the introspect URL:

Example:
```
http://10.1.1.5:8085/Snh_ItfReq?name=&type=&uuid=&vn=&mac=&ipv4_address=&ipv6_address=&parent_uuid=&ip_active=&ip6_active=&l2_active=
```
Note:
Replace the IP address with the actual compute IP address in the introspect HTTP URL.

Run the vif --get <index> command to verify the virtual routing and forwarding (VRF) and Policy flags are set in the vRouter interface (VIF).

Example output verifying flags for each vif:

content_copy zoom_out_map
vif0/4      OS: tapdd789d34-27
            Type:Virtual HWaddr:00:00:5e:00:01:00 IPaddr:10.1.1.5
            Vrf:2 Mcast Vrf:2 Flags:PL3L2Er QOS:-1 Ref:6
            RX packets:30  bytes:8676 errors:0
            TX packets:170  bytes:7140 errors:0
            ISID: 0 Bmac: 02:dd:78:9d:34:27
            Drops:81

vif0/6      OS: tapaedbc037-bf
            Type:Virtual HWaddr:00:00:5e:00:01:00 IPaddr:10.1.1.6
            Vrf:2 Mcast Vrf:2 Flags:PL3L2Er QOS:-1 Ref:6
            RX packets:96316  bytes:4858043 errors:0
            TX packets:96562  bytes:4884177 errors:0
            ISID: 0 Bmac: 02:ae:db:c0:37:bf

Run the following command to display all of the entries from the bridge table:

content_copy zoom_out_map
rt --dump <vrf id> --family bridge

Example:

content_copy zoom_out_map
rt --dump 2 --family bridge
Flags: L=Label Valid, Df=DHCP flood, Mm=Mac Moved, L2c=L2 Evpn Control Word, N=New Entry, Ec=EvpnControlProcessing
vRouter bridge table 0/2
Index       DestMac                  Flags           Label/VNID      Nexthop           Stats
31264       0:0:5e:0:1:0            Df                    -                 3          206834
79784       2:dd:78:9d:34:27        Df                    -                44               4
112924      ff:ff:ff:ff:ff:ff       LDf                   5                48              35
115240      2:0:0:0:0:2             Df                    -                12               0
169408      2:ae:db:c0:37:bf        Df                    -                25               1
183944      2:99:ef:64:96:e1        LDf                  27                20               0
205564      2:0:0:0:0:1             Df                    -                12               0
252380      0:25:90:c5:58:94        Df                    -                 3               0

In the example, 2:99:ef:64:96:e1 belongs to IP address 10.1.1.7 and label 27 is used to encapsulate the packet.

Run nh --get <nh id> to get the next hop details.

Example:

content_copy zoom_out_map
nh --get 20
Id:20         Type:Tunnel         Fmly: AF_INET  Rid:0  Ref_cnt:12         Vrf:0
              Flags:Valid, MPLSoGRE, Etree Root,
              Oif:0 Len:14 Data:00 25 90 c5 62 1c 00 25 90 c5 58 94 08 00
              Sip:10.204.217.86 Dip:10.204.217.70

In the example, the next-hop output indicates the next-hop type as Tunnel, encapsulation used as MPLSoGRE, the outgoing interface as Oif:0, and the corresponding source and destination IP addresses of the tunnel.

For Compute 2:

Run the mpls --get <label> command to see the next hop mapped to the particular incoming MPLS table.

Example:
```
mpls --get 27
MPLS Input Label Map

   Label    NextHop
-------------------------
      27        28
```

Run nh --get <nh_id> to get the next hop details.

Example:

content_copy zoom_out_map
nh --get 28
Id:28         Type:Encap          Fmly:AF_BRIDGE  Rid:0  Ref_cnt:3          Vrf:2
              Flags:Valid, Policy, Etree Root,
              EncapFmly:0806 Oif:3 Len:14
              Encap Data: 02 99 ef 64 96 e1 00 00 5e 00 01 00 08 00

Run vif --get <oifindex> to get the outgoing VIF details.

Example:
```
vif --get 3

vif0/3      OS: tap99ef6496-e1
            Type:Virtual HWaddr:00:00:5e:00:01:00 IPaddr:10.1.1.7
            Vrf:2 Mcast Vrf:2 Flags:PL3L2Er QOS:-1 Ref:6
            RX packets:34  bytes:10044 errors:0
            TX packets:1699  bytes:78990 errors:0
            Drops:93
```
Note:
If you are using VXLAN encapsulation, do the following on Compute 2:
1. For Step 1, instead of running the mpls --get command, run the vxlan --get <vxlanid> command to get the mapping from VXLAN ID to the next hop.
2. With VXLAN, the next hop points to a VRF translated next hop. Use the bridge lookup in the corresponding VRF, as shown in Step 3 to get the final outgoing next hop, which will point to the VIF interface.

Unicast Packet Path - Inter-VN

The following procedure steps through debugging the packet path from VM1 to VM2 (on the same compute node) and VM3 (on a different compute node). In this example, the virtual machines (VMs) listed are in the same subnet 10.1.1.0/24.

VM1	IP address `10.1.1.5/32` (Compute 1)
VM2	IP address `20.1.1.6/32` (Compute 1)
VM3	IP address `20.1.1.5/32` (Compute 2)

Note:

Replace the IP address with the actual compute IP address in all of the introspect URLs.

Intra-Compute Use Case
Inter-Compute Use Case

Intra-Compute Use Case

Discover the vif interfaces corresponding to the virtual machine interfaces (VMI)s of the VM using the command:
```
vif --list
```
You can also discover the vif interfaces by entering the introspect URL:

Example:
```
http://10.1.1.5:8085/Snh_ItfReq?name=&type=&uuid=&vn=&mac=&ipv4_address=&ipv6_address=&parent_uuid=&ip_active=&ip6_active=&l2_active=
```
Note:
Replace the IP address with the actual compute IP address in the introspect HTTP URLs.

Run the vif --get <index> command to verify the virtual routing and forwarding (VRF) and Policy flags are set in the vRouter interface (VIF).

Example output verifying flags for each vif:

content_copy zoom_out_map
vif0/4      OS: tapdd789d34-27
            Type:Virtual HWaddr:00:00:5e:00:01:00 IPaddr:10.1.1.5
            Vrf:2 Mcast Vrf:2 Flags:PL3L2Er QOS:-1 Ref:6
            RX packets:30  bytes:8676 errors:0
            TX packets:170  bytes:7140 errors:0
            ISID: 0 Bmac: 02:dd:78:9d:34:27
            Drops:81

vif0/6      OS: tapaedbc037-bf
            Type:Virtual HWaddr:00:00:5e:00:01:00 IPaddr:10.1.1.6
            Vrf:2 Mcast Vrf:2 Flags:PL3L2Er QOS:-1 Ref:6
            RX packets:96316  bytes:4858043 errors:0
            TX packets:96562  bytes:4884177 errors:0
            ISID: 0 Bmac: 02:ae:db:c0:37:bf
            Drops:2

Run the following command to display all of the entries from the bridge table:

content_copy zoom_out_map
rt --dump <vrf id> --family bridge

Example:

content_copy zoom_out_map
rt --dump 2 --family bridge
Flags: L=Label Valid, Df=DHCP flood, Mm=Mac Moved, L2c=L2 Evpn Control Word, N=New Entry, Ec=EvpnControlProcessing
vRouter bridge table 0/2
Index       DestMac                  Flags           Label/VNID      Nexthop           Stats
31264       0:0:5e:0:1:0            Df                    -                 3          212744
79784       2:dd:78:9d:34:27        Df                    -                44             408
112924      ff:ff:ff:ff:ff:ff       LDf                   5                51              38
115240      2:0:0:0:0:2             Df                    -                12               0
169408      2:ae:db:c0:37:bf        Df                    -                25             405
183944      2:99:ef:64:96:e1        LDf                   5                37               0
205564      2:0:0:0:0:1             Df                    -                12               0
252380      0:25:90:c5:58:94        Df                    -                 3               0

In the case of inter-virtual network (VN)s, the packets are Layer 3 routed instead of Layer 2 switched. The vRouter does a proxy ARP for the destination network providing it’s virtual MAC address 0:0:5e:0:1:0 back for the ARP request from the source. This can be seen from the rt –dump of the source VN inet table. This results in the packet being received by the vRouter, which does the route lookup to send the packet to the correct destination.

Run nh --get <nh id> to display the next-hop details.

Example:

content_copy zoom_out_map
nh --get 3
Id:3          Type:L2 Receive     Fmly: AF_INET  Rid:0  Ref_cnt:8          Vrf:0
              Flags:Valid, Etree Root,

Run rt --dump 2 --family inet | grep <ip address> to display inet family routes on the specified IP address.

Example:

content_copy zoom_out_map
rt --dump 2 --family inet | grep 20.1.1.6
Destination           PPL        Flags        Label         Nexthop    Stitched MAC(Index)

1.1.6/32            32            P          -             30        -
1.1.60/32            0                        -              0        -
1.1.61/32            0                        -              0        -
1.1.62/32            0                        -              0        -
1.1.63/32            0                        -              0        -
1.1.64/32            0                        -              0        -
1.1.65/32            0                        -              0        -
1.1.66/32            0                        -              0        -
1.1.67/32            0                        -              0        -
1.1.68/32            0                        -              0        -
1.1.69/32            0                        -              0        -
1.106.0/24           0                        -              0        -

Run nh --get <nh id> to get the next hop details.

Example:

content_copy zoom_out_map
nh --get 30
Id:30      Type:Encap          Fmly: AF_INET  Rid:0  Ref_cnt:5          Vrf:3
              Flags:Valid, Policy, Etree Root,
              EncapFmly:0806 Oif:3 Len:14
              Encap Data: 02 60 fc 55 cb bf 00 00 5e 00 01 00 08 00

Run vif --get <oifindex> to get the outgoing VIF details.

Example:

content_copy zoom_out_map
vif --get 3
vif0/3      OS: tap60fc55cb-bf
            Type:Virtual HWaddr:00:00:5e:00:01:00 IPaddr:20.1.1.6
            Vrf:3 Mcast Vrf:3 Flags:PL3L2Er QOS:-1 Ref:6
            RX packets:356  bytes:32586 errors:0
            TX packets:3930  bytes:177290 errors:0
            ISID: 0 Bmac: 02:60:fc:55:cb:bf
            Drops:84

Inter-Compute Use Case

In the case of inter-compute, the next hop looked up to send the packet out, will point to a tunnel next hop. Depending on the encapsulation priority, the appropriate encapsulation is added and the packet is tunneled out. Inter-compute traffic is between VMs on different compute nodes.

For Compute 1:

Run rt --dump 2 --family inet | grep <ip address> to display inet family routes for a specified IP address.

Example:

content_copy zoom_out_map
rt --dump 2 --family inet | grep 20.1.1.5
1.1.5/32            32           LP       32             49        -
1.1.50/32            0                       -              0        -
1.1.51/32            0                       -              0        -
1.1.52/32            0                       -              0        -
1.1.53/32            0                       -              0        -
1.1.54/32            0                       -              0        -
1.1.55/32            0                       -              0        -
1.1.56/32            0                       -              0        -
1.1.57/32            0                       -              0        -
1.1.58/32            0                       -              0        -

Run nh --get <nh id> to display the next-hop details, which points to a tunnel next hop.

Example:

content_copy zoom_out_map
nh --get 49
Id:49      Type:Tunnel         Fmly: AF_INET  Rid:0  Ref_cnt:9          Vrf:0
              Flags:Valid, MPLSoUDP, Etree Root,
              Oif:0 Len:14 Data:00 25 90 c5 62 1c 00 25 90 c5 58 94 08 00
              Sip:10.204.217.86 Dip:10.204.217.70

For Compute 2:

Run the mpls --get <label> command to see the next hop mapped to the particular incoming MPLS table.

Example:
```
mpls --get 32
MPLS Input Label Map

   Label    NextHop
-------------------------
      32        36
```
Run nh --get <nh id> to view the next hop details.

Example:
```
nh --get 36
Id:36      Type:Encap          Fmly: AF_INET  Rid:0  Ref_cnt:5          Vrf:3
              Flags:Valid, Policy, Etree Root,
              EncapFmly:0806 Oif:4 Len:14
              Encap Data: 02 f3 37 7b 53 25 00 00 5e 00 01 00 08 00
```
In the example, Oif:4 is the OIF index in the next hop which is the outgoing interface for the packet. The Encap Data corresponds to the L2 encapsulation that is added to the IP packet before the packet is forwarded to the outgoing interface.

Run vif --get <oifindex> to get the outgoing VIF details.

Example:

content_copy zoom_out_map
vif --get 4
vif0/4      OS: tapf3377b53-25
            Type:Virtual HWaddr:00:00:5e:00:01:00 IPaddr:20.1.1.5
            Vrf:3 Mcast Vrf:3 Flags:PL3L2Er QOS:-1 Ref:6
            RX packets:100  bytes:16915 errors:0
            TX packets:3955  bytes:178363 errors:0
            ISID: 0 Bmac: 02:f3:37:7b:53:25
            Drops:78

For details about EVPN type 5 routing in Contrail Networking, see Support for EVPN Route Type 5.

Broadcast, Unknown Unicast, and Multicast Packet Path

The following procedure steps through debugging the packet path for broadcast, unknown unicast, and multicast (BUM) traffic in Contrail Networking. In this example, the virtual machines (VMs) listed are in the same subnet 70.70.70.0/24.

The ToR Service Node (TSN) actively holds the contrail-tor-agent and is responsible for:

Acting as a receiver of all BUM traffic coming from the ToR switch.
Acting as DNS/DHCP responder for the BMS connected to the ToR switch.

Contrail Networking releases earlier than 5.x, used an Open vSwitch Database (OVSDB)-managed VXLAN environment.

Topology example for an OVSDB-managed VXLAN:

Top-of-Rack Switch 1 (ToR SW1) - 10.204.74.229 (lo0.0 = 1.1.1.229)
Top-of-Rack Switch 2 (ToR SW2) - 10.204.74.230 (lo0.0 = 1.1.1.230)
ToR Services Node 1 (TSN1) = 10.219.94.7
ToR Services Node 2 (TSN2) = 10.219.94.8
Controller1 = 10.219.94.4
Controller2 = 10.219.94.5
Controller3 = 10.219.94.6
Compute1 = 10.219.94.9
Compute2 = 19.219.94.18

Virtual Network (VN) = 70.70.70.0/24
Virtual Machine 1 (VM1) = 70.70.70.3 residing on Compute2
Virtual Machine 2 (VM2) = 70.70.70.5 residing on Compute1
Bare Metal Server 1 (BMS1) = 70.70.70.100
Bare Metal Server 2 (BMS2) = 70.70.70.101

Run the set protocols ovsdb interfaces <interface> command to configure the physical interfaces that you want the OVSDB protocol to manage.

Example:
```
set protocols ovsdb interfaces ge-0/0/46
set protocols ovsdb interfaces ge-0/0/90
```
The ToR interfaces from which the BMS hangs are marked as ovsdb interfaces.

View packets coming into these interfaces by displaying the ovsdb mac table for the ToR switch.

Example:

content_copy zoom_out_map
root@QFX5100-229> show ovsdb mac 
Logical Switch Name: Contrail-9ed1f70a-6eac-4cdb-837a-1579728fd9a1
  Mac                    IP                 Encapsulation      Vtep    
  Address                Address                               Address        
  ff:ff:ff:ff:ff:ff      0.0.0.0            Vxlan over Ipv4    1.1.1.229      
  40:a6:77:d8:37:1d      0.0.0.0            Vxlan over Ipv4    1.1.1.229      
  02:3b:ce:56:61:98      0.0.0.0            Vxlan over Ipv4    10.219.94.18   
  02:53:89:c4:29:1c      0.0.0.0            Vxlan over Ipv4    10.219.94.18   
  02:72:e9:7a:cd:f5      0.0.0.0            Vxlan over Ipv4    10.219.94.9    
  02:75:a1:33:65:3c      0.0.0.0            Vxlan over Ipv4    10.219.94.9    
  ff:ff:ff:ff:ff:ff      0.0.0.0            Vxlan over Ipv4    10.219.94.7    

{master:0}

The entry marked in red (ff:ff:ff:ff:ff:ff:ff - broadcast route) indicates the next hop for a BUM packet coming into the ToR SW’s ovsdb interface. In this case, VTEP address 10.219.94.7 is the next hop, which is TSN1. This changes based on which TSN has the active contrail-tor-agent for the ToR switch in question. With this, the BUM packet is forwarded to the TSN node in a VXLAN tunnel (local VTEP source interface is 1.1.1.229 and RVTEP source interface is 10.219.94.7).

The VXLAN encapsulated packet is sent with a VXLAN Network Identifier (VNI) that is predetermined by Contrail Networking when logical interfaces are created. For example, when ge-0/0/46 was configured as a logical port in Contrail Networking, the following configuration was committed on the ToR.

Example:

content_copy zoom_out_map
set vlans Contrail-9ed1f70a-6eac-4cdb-837a-1579728fd9a1 interface ge-0/0/46.0
set vlans Contrail-9ed1f70a-6eac-4cdb-837a-1579728fd9a1 interface ge-0/0/90.0
set vlans Contrail-9ed1f70a-6eac-4cdb-837a-1579728fd9a1 vxlan vni 4

As the VXLAN encapsulated packet arrives on the TSN node, let’s examine how the vRouter handles this packet.

Run vxlan --dump to dump the VXLAN table. The VXLAN table maps a network ID to a next hop.

Example:
```
root@contrail61:~# vxlan --dump
VXLAN Table

 VNID    NextHop
----------------
      4    13
```
In the example, next hop 13 is programmed for VNI 4.

Run nh --get <nh id> to display the next-hop details and determine the virtual routing and forwarding (VRF) associated.

Example:

content_copy zoom_out_map
root@contrail61:~# nh --get 13
Id:13         Type:Vrf_Translate  Fmly: AF_INET  Rid:0  Ref_cnt:2          Vrf:1
              Flags:Valid, Vxlan, 
              Vrf:1

Run the following command to display all of the entries from the bridge table:

content_copy zoom_out_map
rt --dump <vrf id> --family bridge

Example:

content_copy zoom_out_map
root@contrail61:~# rt --dump 1 --family bridge
Flags: L=Label Valid, Df=DHCP flood
vRouter bridge table 0/1
Index       DestMac                  Flags           Label/VNID      Nexthop
30780       0:1a:a0:e:30:26                               -             1
57812       2:53:89:c4:29:1c          LDf                17            15
70532       2:3b:ce:56:61:98          LDf                20            15
87024       2:72:e9:7a:cd:f5          LDf                17            14
97192       ff:ff:ff:ff:ff:ff         LDf                 4            24
121160      0:1a:a0:a:b4:87                               -             1
225832      40:a6:77:d8:37:1d         LDf                 4            19
237084      0:11:a:6c:50:d4            Df                 -             3
244992      aa:bb:cc:dd:3e:f4                             -             1
252916      0:0:5e:0:1:0               Df                 -             3
256476      2:75:a1:33:65:3c          LDf                20            14

In the example bridge table, since we are tracing the BUM packet path, we need to examine the ff:ff:ff:ff:ff:ff:ff route by selecting the next hop programmed. In the example, it is 24. Note that a series of composite next hops are programmed.

Run nh --get <nh id> to display the next-hop details.

Example:

content_copy zoom_out_map
root@contrail61:~# nh --get 24
Id:24         Type:Composite      Fmly:AF_BRIDGE  Rid:0  Ref_cnt:4          Vrf:1
              Flags:Valid, Multicast, L2, 
              Sub NH(label): 20(0) 22(0) 21(0)

Id:20         Type:Composite      Fmly: AF_INET  Rid:0  Ref_cnt:2          Vrf:1
              Flags:Valid, Tor, 
              Sub NH(label): 19(4)

Id:19         Type:Tunnel         Fmly: AF_INET  Rid:0  Ref_cnt:3          Vrf:0
              Flags:Valid, Vxlan, 
              Oif:0 Len:14 Flags Valid, Vxlan,  Data:00 00 5e 00 01 21 00 11 0a 6c 50 d4 08 00 
              Vrf:0  Sip:10.219.94.7  Dip:1.1.1.229  << Source where the BUM Traffic came from.

Id:22         Type:Composite      Fmly: AF_INET  Rid:0  Ref_cnt:2          Vrf:1
              Flags:Valid, Evpn, 
              Sub NH(label):

Id:21         Type:Composite      Fmly: AF_INET  Rid:0  Ref_cnt:2          Vrf:1
              Flags:Valid, Fabric, 
              Sub NH(label): 15(4099)

Id:15         Type:Tunnel         Fmly: AF_INET  Rid:0  Ref_cnt:6          Vrf:0
              Flags:Valid, MPLSoGRE, 
              Oif:0 Len:14 Flags Valid, MPLSoGRE,  Data:f8 bc 12 33 43 31 00 11 0a 6c 50 d4 08 00 
              Vrf:0  Sip:10.219.94.7  Dip:10.219.94.18  << Compute node which has a VM in this VN.

The multicast tree in the example shows that there are two Dynamic IPs (DIP)s. The DIP where the packet came from is ignored. Therefore, packet gets forwarded to DIP 10.219.94.18 only.

Run vxlan --get <vnid> to examine what DIP 10.219.94.18 does with the incoming VXLAN encapsulated packet.

Example:
```
root@contrail4:~# vxlan --get 4
VXLAN Table

 VNID    NextHop
----------------
      4    20
```

Run nh --get <nh id> to display the next-hop details.

Example:

content_copy zoom_out_map
root@contrail4:~# nh --get 20
Id:20         Type:Vrf_Translate  Fmly: AF_INET  Rid:0  Ref_cnt:2          Vrf:1
              Flags:Valid, Vxlan, 
              Vrf:1

Run the following command to display all of the entries from the bridge table:

content_copy zoom_out_map
rt --dump <vrf id> --family bridge

Example:

content_copy zoom_out_map
root@contrail4:~# rt --dump 1 --family bridge
Flags: L=Label Valid, Df=DHCP flood
vRouter bridge table 0/1
Index       DestMac                  Flags           Label/VNID      Nexthop
57812       2:53:89:c4:29:1c                              -            15
70532       2:3b:ce:56:61:98                              -            22
87024       2:72:e9:7a:cd:f5          LDf                17            25
97192       ff:ff:ff:ff:ff:ff         LDf                 4            50
112856      f8:bc:12:33:43:31          Df                 -             3
225832      40:a6:77:d8:37:1d         LDf                 4            18
252916      0:0:5e:0:1:0               Df                 -             3
256476      2:75:a1:33:65:3c          LDf                20            25

In the example bridge table, since we are tracing the BUM packet path, we need to examine the ff:ff:ff:ff:ff:ff:ff route by selecting the next hop programmed. In the example, it is 50.

Run nh --get <nh id> to display the next-hop details.

Example:

content_copy zoom_out_map
root@contrail4:~# nh --get 50
Id:50         Type:Composite      Fmly:AF_BRIDGE  Rid:0  Ref_cnt:4          Vrf:1
              Flags:Valid, Multicast, L2, 
              Sub NH(label): 43(0) 49(0)

Id:43         Type:Composite      Fmly: AF_INET  Rid:0  Ref_cnt:2          Vrf:1
              Flags:Valid, Fabric, 
              Sub NH(label): 31(4612) 25(4617)

Id:31         Type:Tunnel         Fmly: AF_INET  Rid:0  Ref_cnt:2          Vrf:0
              Flags:Valid, MPLSoGRE, 
              Oif:0 Len:14 Flags Valid, MPLSoGRE,  Data:00 11 0a 6c 50 d4 f8 bc 12 33 43 31 08 00 
              Vrf:0  Sip:10.219.94.18  Dip:10.219.94.7  <<< Source where the BUM traffic came from. 

Id:25         Type:Tunnel         Fmly: AF_INET  Rid:0  Ref_cnt:2562       Vrf:0
              Flags:Valid, MPLSoGRE, 
              Oif:0 Len:14 Flags Valid, MPLSoGRE,  Data:44 a8 42 3a 94 f4 f8 bc 12 33 43 31 08 00 
              Vrf:0  Sip:10.219.94.18  Dip:10.219.94.9  <<< Compute node which has a VM in this VN. 

Id:49         Type:Composite      Fmly: AF_INET  Rid:0  Ref_cnt:2          Vrf:1
              Flags:Valid, Encap, 
              Sub NH(label): 14(0) 21(0)

Id:14         Type:Encap          Fmly:AF_BRIDGE  Rid:0  Ref_cnt:2          Vrf:1
              Flags:Valid, 
              EncapFmly:0806 Oif:4 Len:14
              Encap Data: 02 53 89 c4 29 1c 00 00 5e 00 01 00 08 00 

Id:21         Type:Encap          Fmly:AF_BRIDGE  Rid:0  Ref_cnt:2          Vrf:1
              Flags:Valid, 
              EncapFmly:0806 Oif:3 Len:14
              Encap Data: 02 3b ce 56 61 98 00 00 5e 00 01 00 08 00  <<< Local VM belonging to this VN that is an intended receiver of this multicast traffic.

In the example, you only have to inspect DIP 10.219.94.9. The remaining endpoints are either local or the source where the BUM traffic came from. Now, let us examine, what DIP 10.219.94.9 does with the incoming VXLAN encapsulated packet.

Run vxlan --get <vnid> to examine what DIP 10.219.94.9 does with the incoming VXLAN encapsulated packet.

Example:
```
root@contrail101:~# vxlan --get 4
VXLAN Table

 VNID    NextHop
----------------
      4    20
```

Run nh --get <nh id> to display the next-hop details.

Example:

content_copy zoom_out_map
root@contrail101:~# nh --get 20
Id:20         Type:Vrf_Translate  Fmly: AF_INET  Rid:0  Ref_cnt:2          Vrf:1
              Flags:Valid, Vxlan, 
              Vrf:1

Display the bridge table for the VRF by using the following command:

content_copy zoom_out_map
rt --dump <vrf id> --family bridge

Example:

content_copy zoom_out_map
root@contrail101:~# rt --dump 1 --family bridge
Flags: L=Label Valid, Df=DHCP flood
vRouter bridge table 0/1
Index       DestMac                  Flags           Label/VNID      Nexthop
57812       2:53:89:c4:29:1c          LDf                17            28
70532       2:3b:ce:56:61:98          LDf                20            28
87024       2:72:e9:7a:cd:f5                              -            15
97192       ff:ff:ff:ff:ff:ff         LDf                 4            31
140744      44:a8:42:3a:94:f4          Df                 -             3
225832      40:a6:77:d8:37:1d         LDf                 4            24
252916      0:0:5e:0:1:0               Df                 -             3
256476      2:75:a1:33:65:3c                              -            22
              Encap Data: f8 bc 12 33 43 31 44 a8 42 3a 94 f4 08 00 

Run nh --get <nh id> to display the next-hop details.

Example:

content_copy zoom_out_map
root@contrail101:~# nh --get 31
Id:31         Type:Composite      Fmly:AF_BRIDGE  Rid:0  Ref_cnt:4          Vrf:1
              Flags:Valid, Multicast, L2, 
              Sub NH(label): 30(0) 36(0)

Id:30         Type:Composite      Fmly: AF_INET  Rid:0  Ref_cnt:2          Vrf:1
              Flags:Valid, Fabric, 
              Sub NH(label): 29(4612) 28(4099)

Id:29         Type:Tunnel         Fmly: AF_INET  Rid:0  Ref_cnt:2          Vrf:0
              Flags:Valid, MPLSoGRE, 
              Oif:0 Len:14 Flags Valid, MPLSoGRE,  Data:00 11 0a 6c 50 ac 44 a8 42 3a 94 f4 08 00 
              Vrf:0  Sip:10.219.94.9  Dip:10.219.94.8 << TSN2 in this topology that is managing a ToR with an end-point belonging to this VN.

Id:28         Type:Tunnel         Fmly: AF_INET  Rid:0  Ref_cnt:2566       Vrf:0
              Flags:Valid, MPLSoGRE, 
              Oif:0 Len:14 Flags Valid, MPLSoGRE,  Data:f8 bc 12 33 43 31 44 a8 42 3a 94 f4 08 00 
              Vrf:0  Sip:10.219.94.9  Dip:10.219.94.18 << Source where the BUM traffic came from. 

Id:36         Type:Composite      Fmly: AF_INET  Rid:0  Ref_cnt:2          Vrf:1
              Flags:Valid, Encap, 
              Sub NH(label): 14(0) 21(0)

Id:14         Type:Encap          Fmly:AF_BRIDGE  Rid:0  Ref_cnt:2          Vrf:1
              Flags:Valid, 
              EncapFmly:0806 Oif:3 Len:14
              Encap Data: 02 72 e9 7a cd f5 00 00 5e 00 01 00 08 00 << local VM that is an intended receiver of this traffic as it is tagged to this VN

Id:21         Type:Encap          Fmly:AF_BRIDGE  Rid:0  Ref_cnt:2          Vrf:1
              Flags:Valid, 
              EncapFmly:0806 Oif:4 Len:14
              Encap Data: 02 75 a1 33 65 3c 00 00 5e 00 01 00 08 00  << Local VM that is an intended receiver of this traffic since it is tagged to this VN.

From the above output, the only DIP that you have to further examine is 10.219.94.8. The remaining DIPs are either local or the source where the BUM traffic came from. Now, let’s examine what DIP 10.219.94.8 does with the incoming VXLAN encapsulated packet.

Run vxlan --get <vnid> to examine what DIP 10.219.94.9 does with the incoming VXLAN encapsulated packet.

Example:
```
root@contrail66:~# vxlan --get 4
VXLAN Table

 VNID    NextHop
----------------
      4    14
```

Run nh --get <nh id> to display the next-hop details.

Example:

content_copy zoom_out_map
root@contrail66:~# nh --get 14
Id:14         Type:Vrf_Translate  Fmly: AF_INET  Rid:0  Ref_cnt:2          Vrf:1
              Flags:Valid, Vxlan, 
              Vrf:1

Display the bridge table for the VRF by using the following command:

content_copy zoom_out_map
rt --dump <vrf id> --family bridge

Example:

content_copy zoom_out_map
root@contrail66:~# rt --dump 1 --family bridge
Flags: L=Label Valid, Df=DHCP flood
vRouter bridge table 0/1
Index       DestMac                  Flags           Label/VNID      Nexthop
30780       0:1a:a0:e:30:26                               -             1
57812       2:53:89:c4:29:1c          LDf                17            17
70532       2:3b:ce:56:61:98          LDf                20            17
87024       2:72:e9:7a:cd:f5          LDf                17            16
97192       ff:ff:ff:ff:ff:ff         LDf                 4            24
121160      0:1a:a0:a:b4:87                               -             1
217208      0:11:a:6c:50:ac            Df                 -             3
225832      40:a6:77:d8:37:1d         LDf                 4            20
244992      aa:bb:cc:dd:3e:f4                             -             1
252916      0:0:5e:0:1:0               Df                 -             3
256476      2:75:a1:33:65:3c          LDf                20            16

Run nh --get <nh id> to display the next-hop details.

Example:

content_copy zoom_out_map
root@contrail66:~# nh --get 24
Id:24         Type:Composite      Fmly:AF_BRIDGE  Rid:0  Ref_cnt:4          Vrf:1
              Flags:Valid, Multicast, L2, 
              Sub NH(label): 23(0) 25(0) 21(0)

Id:23         Type:Composite      Fmly: AF_INET  Rid:0  Ref_cnt:2          Vrf:1
              Flags:Valid, Tor, 
              Sub NH(label): 22(4)

Id:22         Type:Tunnel         Fmly: AF_INET  Rid:0  Ref_cnt:2          Vrf:0
              Flags:Valid, Vxlan, 
              Oif:0 Len:14 Flags Valid, Vxlan,  Data:00 00 5e 00 01 21 00 11 0a 6c 50 ac 08 00 
              Vrf:0  Sip:10.219.94.8  Dip:1.1.1.230 <<< Another ToR switch that has an end-point belonging to this VN.

Id:25         Type:Composite      Fmly: AF_INET  Rid:0  Ref_cnt:2          Vrf:1
              Flags:Valid, Evpn, 
              Sub NH(label):

Id:21         Type:Composite      Fmly: AF_INET  Rid:0  Ref_cnt:2          Vrf:1
              Flags:Valid, Fabric, 
              Sub NH(label): 16(4617)

Id:16         Type:Tunnel         Fmly: AF_INET  Rid:0  Ref_cnt:6          Vrf:0
              Flags:Valid, MPLSoGRE, 
              Oif:0 Len:14 Flags Valid, MPLSoGRE,  Data:44 a8 42 3a 94 f4 00 11 0a 6c 50 ac 08 00 
              Vrf:0  Sip:10.219.94.8  Dip:10.219.94.9 <<< Source where the BUM traffic came from. 

Now, you just have one DIP 1.1.1.230 which is the ToR SW2 in the topology. This should also be present in the multicast tree as this ToR SW also has an end-point (which is BMS2) in the same VN (VNI=4) as the one we are tracing.

This completes all levels of forwarding and tracing the BUM packet from one ToR switch and is replicated to other intended receivers in the topology.

These multicast trees are programmed by the controllers that the TSN is connected to. If you want to inspect the controller’s memory and what eventually gets programmed on all TSN computes, enter the following introspect URL using your controller IP address:

content_copy zoom_out_map
http://<controller_ip>:8083/Snh_ShowMulticastManagerDetailReq?x=default-domain:admin:seventy-network:seventy-network.ermvpn.0

arrow_backward PREVIOUS vRouter Command Line Utilities

NEXT arrow_forward Using Contrail Tools

Contrail Networking Monitoring and Troubleshooting Guide

ON THIS PAGE

Tracing the vRouter Packet Path

Unicast Packet Path - Intra-VN

Intra-Compute Use Case

Inter-Compute Use Case

Unicast Packet Path - Inter-VN

Intra-Compute Use Case

Inter-Compute Use Case

Broadcast, Unknown Unicast, and Multicast Packet Path

Stay in touch

Helped me install or use the product
Accelerated my deployment

Discuss the product with a co-worker
Make a purchase inquiry

Contrail Networking Monitoring and Troubleshooting Guide

ON THIS PAGE

Tracing the vRouter Packet Path

Unicast Packet Path - Intra-VN

Intra-Compute Use Case

Inter-Compute Use Case

Unicast Packet Path - Inter-VN

Intra-Compute Use Case

Inter-Compute Use Case

Broadcast, Unknown Unicast, and Multicast Packet Path

Related Documentation

Stay in touch