JCNR-vRouter
Read this chapter to understand more about JCNR-vRouter, which is the JCNR DPDK-based forwarding plane.
Benefits of JCNR vRouter
-
Integration of the DPDK into the JCNR-vRouter:
-
Forwarding plane provides faster forwarding capabilities than kernel-based forwarding
-
Forwarding plane is more scalable than kernel-based forwarding
-
Support for the following NICs:
-
Intel E810 (Columbiaville) with Intel Adapter Virtual Function (IAVF) and Dynamic Device Personalization (DDP)
-
Intel XL710 (Fortville) with Intel Adapter Virtual Function (IAVF)
Note:Dynamic Device Personalization (DDP) is not supported on the Intel XL710 NIC
-
-
Support for the following drivers on Intel XL710 NICs:
-
Intel Adapter Virtual Function (IAVF)
-
Linux base driver i40E
-
-
-
Interface Support:
-
Pod interfaces using virtio
You define these DPDK-enabled vhost-based user socket interfaces as workload interfaces in the
values.yaml
file. The system maintains the socket details for all active interfaces of this type in the /dpdk directory of the workload container. You usually define this type of pod interfaces as trunk interfaces rather than access interfaces. -
Pod interfaces using kernel
veth
pairYou define pod interfaces with kernel veth pairs in the
values.yaml
file as access interfaces so the pod can pass traffic through the kernel rather than using DPDK. -
DPDK Virtual Function (VF) workload interfaces
You define the DPDK VF fabric trunk interfaces in the values.yaml file that is used in the vRouter deployment. This makes JCNR aware of the names of the interfaces, their MAC addresses, and their PCI slot ID.
-
DPDK VF fabric trunk interfaces
You define the DPDK VF fabric trunk interfaces in the values.yaml file that is used in the vRouter deployment. This makes JCNR aware of the names of the interfaces, their MAC addresses, and their PCI slot ID. To keep traffic flows manageable, we apply VLAN filtering to the physical interfaces. When you configure VLANs for use with the cloud-native router, only the configured VLANs can pass through the physical interfaces.
-
Physical Function (PF) workload interfaces
You define PF workload interfaces in the values_l3.yaml file. The system equips each PF workload interfaces with only one receive and one transmit queue. The system assigns one forwarding CPU core to the task of polling the interface for traffic.
-
PF fabric interfaces
You define PF fabric interfaces in the values.yaml file. The system equips each PF fabric interface with as many receive and transmit queues as you assign forwarding CPU cores to handle the polling. For example, if you assign three forwarding CPU cores to the PF fabric interface, the system allocates three receive and three transmit queues to the PF fabric interface.
-
No
vhost0
interface when run in L2 modevRouter-agent detects L2 mode in values.yaml, so does not wait for the
vhost0
interface to come up before completing installation. There is no vhost interface add message sent so the vRouter doesn't create thevhost0
interface.
-
-
Interface Bonding
DPDK vRouter supports interface bonding in active/standby mode on DPDK VF fabric interfaces. The values.yaml file specifies the interface names, mode value, and primary and secondary interface designations. DPDK contains a library with its own bonding driver that it uses for bonding. In operation, the vRouter uses the primary interface to pass traffic. If the primary link goes down, the secondary interface in the pair passes traffic until the primary interface reconnects.
-
Pod DPDK Interfaces
JCNR-vRouter supports virtio communication to the POD application. The JCNR-CNI allocates unique socket directories that it passes to Pod applications and to vRouter. JCNR-CNI ensures that one Pod cannot access the resources of another Pod through isolation of vhost sockets and Pod volume mounts.
-
Pod Kernel Interfaces
JCNR supports the veth interface type to communicate with pod applications that use the Linux Kernel's networking stack.
Access vRouter CLI
kubectl get pods -n contrail
The output of the command looks like:
NAME READY STATUS RESTARTS AGE contrail-vrouter-masters-97v8z 3/3 Running 0 6d1h
To access the vRouter-agent CLI, you use the full pod name from your system in the following command:
kubectl exec -n contrail -it contrail-vrouter-masters-97v8z -- bash
The output of the command looks like: Defaulted container "contrail-vrouter-agent"
out of: contrail-vrouter-agent, contrail-vrouter-agent-dpdk,
contrail-vrouter-telemetry-exporter, contrail-init (init),
contrail-vrouter-kernel-init-dpdk (init)
.
After you access the vRouter CLI, there are a number of commands that you can run to monitor and troubleshoot the system. We illustrate some of the available commands in Monitoring vRouter with CLI Commands.
L2 Mode Packet Flow in vRouter
To understand a switch or router, you must know what happens to packets as they flow through. This section describes the life of a packet in the vRouter when deployed in L2 mode. In this section, we show how the vRouter MAC and bridge domain (BD) tables are populated and introduces some of the CLI commands that you can use to see various parts of the vRouter from inside.
The flowchart Figure 1 illustrates one possible generic packet flow through cloud-native router. It does not cover all possible interactions with the packet.
The vRouter makes a lot of decisions about a received packet to ensure that the packet is handled correctly. Let's look at what the vRouter does with a packet. You can see in Figure 1 that there are several choices made based on the VLAN or BD. One of the tables that vRouter consults for making forwarding decisions is the Table 1.
VLAN ID (KEY) | Interface ID List (Value) |
---|---|
1024 | 2,3,4 |
1042 | 4,5 |
1022 | 1 |
1044 | 6 |
The BD table tells the vRouter which interfaces can carry traffic with a specific VLAN ID. Thus, the VLAN ID serves as the key for the table while the Interface ID List serves as the values for each entry.
The Table 2 is closely related to the BD table. The MAC table uses the MAC address and VLAN ID as a key pair. Then, the Interface ID and hit count serve as the values for each entry in the table as shown in this section.
MAC Address | VLAN ID | Interface ID | Hit Count |
---|---|---|---|
00:11:22:33:44:55 | 1024 | 2 | 123234 |
00:22:33:44:55:66 | 1042 | 4 | 823948 |
00:33:44:55:66:77 | 1022 | 1 | 45980 |
00:44:55:66:77:88 | 1044 | 6 | 86578 |
The primary purpose of the MAC table is to map which MAC addresses can be reached through which interface. The vRouter makes entries in and consults the MAC table while processing packets.
Monitoring vRouter with CLI Commands
In the vRouter, CLI commands are useful for troubleshooting and monitoring purposes. As mentioned in Access vRouter CLI, you can access the CLI of the vRouter. By executing commands in that CLI, you can learn about various aspects of the running vRouter. The following examples assume that you have already connected to the vRouter CLI. The commands that we show in this section do not show a command prompt so that you can copy and paste them into your own vRouter.
We use the purel2cli command in most of the examples in this setion. The command has more options than we show in the examples. In addition, the purel2cli has a help command that you can use to see the available options.
The purel2cli Command
To see all the options of the purel2cli command in the vRouter CLI, execute the command with the --help option.
purel2cli --help
Usage: purel2cli [--mac show] [--vlan show] [--vlan get <VLAN_ID>] [--acl show <VLAN_ID>] [--acl reset-counters <VLAN_ID>] [--l2stats get <VIF_ID> <VLAN_ID>] [--clear VLAN_ID] [--sock-dir <sock dir>] [--help]
See the Current Status of Your Running vRouter
To see the status of the vRouter, enter the following command in the vRouter CLI:
ps -eaf|grep dpdk
The output from the command above looks like: root 127 93 99 Jul29 ? 82-20:31:49
/contrail-vrouter-dpdk --no-daemon --socket-mem=1024 1024 --allow=0000:01:10.1
--allow=0000:01:10.0 --l2_table_size=10240 --yield_option 0 --ddp --l2_mode
The output contains several elements.
Flag | Meaning |
---|---|
--l2_mode |
The vRouter is running in L2 mode. |
--l2_table_size |
The current number of entries in the MAC table. The default size is 10240 entries. |
--allow=<PCI Id> |
The PCI ID of fabric and fabric workload interfaces. More than one ID can appear in the output. These IDs serve as an allowlist. |
--ddp |
Enable Intel DDP support. We enable DDP by default in the values.yaml file in the vRouter. Note:
The Intel XL710 NIC does not support DDP. |
Show MAC Table
The following command shows the MAC addresses that the vRouter has dynamically learned.
purel2cli --mac show
The output from the above command looks like:
================================================== || MAC vlan port hit_count|| ================================================== 00:01:01:01:01:03 1221 2 1101892 00:01:01:01:01:02 1221 2 1101819 00:01:01:01:01:04 1221 2 1101863 00:01:01:01:01:01 1221 2 1101879 5a:4c:4c:75:90:fe 1250 5 12 Total Mac entries 5
Show Bridge Domain Table
The following command shows the VLAN to port mapping in the vRouter.
purel2cli --vlan show
The output from the above command looks like:
VLAN PORT =============== 1201 1,2,3,4, 1202 1,2,3,4, 1203 1,2,3,4, 1204 1,2,3,4, 1205 1,2,3,4,
You can use the following form of the purel2cli command to see the
bridge domain table entry for a specific VLAN: purel2cli --vlan get
<vlan-id>
Show L2 Statistics
There are several variations of the purel2cli command that allow you
to display and filter L2 statistics in the vRouter. The base form of the command is:
purel2cli -- l2stats get <virtual_interface_ID> <VLAN_ID>
.
The table Table 4 shows the available command options and what they do. This section also
provides a sample output using one of the options.
Sample Command | Function |
---|---|
purel2cli --l2stats get '*' '*' |
Get statistics for all virtual interfaces (vif) and all VLAN IDs. |
purel2cli --l2stats get '*' 100 |
Get statistics for all vif that are part of VLAN 100 |
purel2cli --l2stats get 1 '*' |
Get statistics for all VLANs for which interface 1 is a member |
purel2cli --l2stats get 1 100 |
Get statistics for interface 1 and VLAN 100 |
The following command is an example of the L2 statistics for interface 2 and VLAN 1221:
purel2cli --l2stats get 2 1221
Vlan id count: 1 -------------------------------------------------------------------------------- Statistics for vif 2 vlan 1221 -------------------------------------------------------------------------------- Rx Pkts Rx Bytes Tx Pkts Tx Bytes Unicast 245344824 48152682842 835552 1667761792 Broadcast 0 0 0 0 Multicast 0 0 0 0 Flood 0 0 0 0 --------------------------------------------------------------------------------
Clear L2 Statistics
The following example shows commands that allow you to clear L2 statistics information from the vRouter.
You can clear the
statistics from the vRouter with the purel2cli command in the form: purel2cli
--clear <VLAN_ID>
.
purel2cli --clear '*'
purel2cli --clear 100
The dropstats Command
The vRouter tracks the packets that it drops and includes the reason for dropping them. Table 5shows the common reasons for vRouter to drop a packet. When you execute the dropstats command, the vRouter does not show a counter if the count for that counter is 0.
Counter Name | Meaning |
---|---|
L2 bd table drop |
No interfaces in bridge domain |
L2 untag pkt drop |
Untagged packet arrives on trunk or sub-interface |
L2 Invalid Vlan |
Packet VLAN does not match interface VLAN |
L2 Mac Table Full |
No more entries available in the MAC table |
L2 ACL drop |
Packet matched firewall filter (ACL) drop rule |
L2 Src Mac lookup fail |
Unable to match (or learn) the source MAC address |
Example output from the dropstats command looks like:
dropstats
L2 bd table Drop 43 L2 untag pkt drop 716 L2 Invalid Vlan 7288253 Rate limit exceeded 673179706 L2 Mac Table Full 41398787 L2 ACL drop 8937037 L2 Src Mac lookup fail 247046
The dpdkinfo Command
The dpdkinfo command provides insight into the status and statistics of DPDK. The dpdkinfo command has many options. The following sections describe the available options and the example output from the dpdkinfo command. You can run the dpdkinfo command only from within the vRouter-agent CLI.
dpdkinfo Command Usage
dpdkinfo
Usage: dpdkinfo [--help] --version|-v Show DPDK Version --bond|-b Show Master/Slave bond information --lacp|-l <all/conf> Show LACP information from DPDK --mempool|-m <all/<mempool-name>> Show Mempool information --stats|-n <vif index value> Show Stats information --xstats|-x <vif index value> Show Extended Stats information --lcore|-c Show Lcore information --app|-a Show App information --ddp|-d <list> <list-flow> Show DDP information for X710 NIC --rx_vlan|-z <value> Show VLan information Optional: --buffsz <value> Send output buffer size (less than 1000Mb)
dpdkinfo Lcore Information
This command shows the Lcores assigned to DPDK VF fabric interfaces and the queue ID for each interface.
dpdkinfo -c
No. of forwarding lcores: 4 Lcore 10: Interface: 0000:18:01.1 Queue ID: 0 Interface: 0000:18:0d.1 Queue ID: 0 Interface: 0000:86:00.0 Queue ID: 0 Lcore 11: Interface: 0000:18:01.1 Queue ID: 1 Interface: 0000:18:0d.1 Queue ID: 1 Interface: 0000:86:00.0 Queue ID: 1 Lcore 12: Interface: 0000:18:01.1 Queue ID: 2 Interface: 0000:18:0d.1 Queue ID: 2 Interface: 0000:86:00.0 Queue ID: 2 Lcore 13: Interface: 0000:18:01.1 Queue ID: 3 Interface: 0000:18:0d.1 Queue ID: 3 Interface: 0000:86:00.0 Queue ID: 3
dpdkinfo Memory Pool Information
This command shows all of the memory pool information.
dpdkinfo -m all
--------------------------------------------------- Name Size Used Available --------------------------------------------------- rss_mempool 16384 1549 14835 frag_direct_mempool 4096 0 4096 frag_indirect_mempool 4096 0 4096 packet_mbuf_pool 8192 2 8190
dpdkinfo Statistics Information
This command displays statistical information for a specific interface.
dpdkinfo -n 3
Interface Info(0000:18:0d.1): RX Device Packets:6710, Bytes:1367533, Errors:0, Nombufs:0 Dropped RX Packets:0 TX Device Packets:0, Bytes:0, Errors:0 Queue Rx: Tx: Rx Bytes: Tx Bytes: Errors:
dpdkinfo Extended Statistics Information
This command displays extended statistical information for a specific interface.
dpdkinfo -x 3
Driver Name:net_iavf Interface Info:0000:18:0d.1 Rx Packets: rx_good_packets: 6701 rx_unicast_packets: 0 rx_multicast_packets: 2987 rx_broadcast_packets: 3714 rx_dropped_packets: 0 Tx Packets: tx_good_packets: 0 tx_unicast_packets: 0 tx_multicast_packets: 0 tx_broadcast_packets: 0 tx_dropped_packets: 0 Rx Bytes: rx_good_bytes: 1365696 Tx Bytes: tx_good_bytes: 0 Errors: rx_missed_errors: 0 rx_errors: 0 tx_errors: 0 rx_mbuf_allocation_errors: 0 inline_ipsec_crypto_ierrors: 0 inline_ipsec_crypto_ierrors_sad_lookup: 0 inline_ipsec_crypto_ierrors_not_processed: 0 inline_ipsec_crypto_ierrors_icv_fail: 0 inline_ipsec_crypto_ierrors_length: 0 Others: inline_ipsec_crypto_ipackets: 0 ---------------------------------------------------------------------
Troubleshooting vRouter
For vRouter-agent debugging, we use Introspect. You can access the Introspect data at http://<host server IP>:8085. Table 6 shows a sample of the Introspect data..
The table, Table 6 shows grouped output. The cloud-native router does not group or sort the output on live systems.
The http://host server IP address:8085 page displays only a list of HTML links.
Link | and Description |
---|---|
agent.xml |
Shows agent operational data. Using this introspect, you can see the list of interfaces, VMs, VNs, VRFs, security groups, ACLs and mirror configurations. |
agent_ksync.xml |
Shows agent ksync layer for data objects such as interfaces and bridge ports. |
agent_profile.xml |
shows agent operdb, tasks, flows, and statistics summary. |
agent_stats_interval.xml |
View and set collection period for statistics. |
controller.xml |
Shows the connection status of the jcnr-controller (cRPD) |
cpuinfo.xml |
Shows the CPU load and memory usage on the compute node. |
ifmap_agent.xml |
Shows the current configuration data received from ifmap. |
kstate.xml |
Shows data configured in the vRouter data path. |
mac_learning.xml |
Shows entries in vRouter-agent MAC learning table. |
sandesh_trace.xml |
Gives the different agent module traces such as oper, ksync, mac learning, and grpc. |
sandesh_uve.xml |
Lists all the user visible entitities (UVEs) in the vRouter-agent. The UVEs are used for analytics and telemetry. |
stats.xml |
Shows vRouter-agent slow path statistics such as error packets, trapped packets, and debug statistics. |
task.xml |
Shows vRouter-agent worker task details. |