Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

external-header-nav
keyboard_arrow_up
list Table of Contents
file_download PDF
keyboard_arrow_right

Example: Enabling SR-IOV on vMX Instances on KVM

date_range 19-Nov-20

vMX on KVM supports single-root I/O virtualization (SR-IOV) interface types. Single root I/O virtualization (SR-IOV) allows a physical function to appear as multiple, separate vNICs. SR-IOV allows a device, such as a network adapter to have separate access to its resources among various hardware functions. If you have a physical NIC that supports SR-IOV, you can attach SR-IOV-enabled vNICs or virtual functions (VFs) to the vMX instance to improve performance.

System requirements:

  • Junos OS Release 18.4 or later.

  • SR-IOV on the VMX for KVM requires one of the following Intel NIC drivers:

    • Intel X520 or X540 using 10G ports and ixgbe driver

    • Intel X710 or XL710 using 10G ports and i40e driver

    Starting in Junos OS Release 19.1R1-S1 and in Junos OS Release 19.2R1, support for 40G ports with Intel XL710-QDA2 NICs are available for VMX instances. When using 40G ports, the vMX autodetects the port speed and assigns two I/O vCPUs.

To enable SR-IOV on VMX instances, you must complete the following tasks:

  • Prepare a NIC to use SR-IOV in system (/sys/) mode.

  • Install driver from Intel, you must compile the driver, uninstall old driver, and install new compiled driver

    Note:

    The vMX installer provides a modified intel-driver as well. You can either use the native drivers from Intel, or use vMX modified driver.

  • Prepare vmx.conf file

  • Use Junos CLI to configure native driver

  • BIOS requirement to enable SR-IOV- Ensure that Intel VT-d or AMD IOMMU are enabled in the system’s BIOS settings.

Procedure for Identifying PCI-Addresses and Kernel Name for the NIC

  1. To find the PCI address, use the following command:
    content_copy zoom_out_map
    lab@ubuntu2:/etc/modprobe.d$ ethtool -i ens8f1 | grep bus
    
    content_copy zoom_out_map
    bus-info: 0000:85:00.1
  2. To find the kernel name using PCI, use the following command:
    content_copy zoom_out_map
    lab@ubuntu2:~$ cd /sys/bus/pci/devices 
    lab@ubuntu2:/sys/bus/pci/devices$ ls 0000\:85\:00.1/net/ 
    
    content_copy zoom_out_map
    ens8f1
  3. To find out the driver in use for the NIC, use the following command:
    content_copy zoom_out_map
    lab@ubuntu2:~$ ethtool -i ens8f1 | grep ^driver 
    
    content_copy zoom_out_map
    driver: ixgbe

Download and Install the Latest Driver Software from Intel

You can download the latest driver software from Intel and replace existing driver software provided by Ubuntu.

In this example, download the software from Intel® Network Adapter Driver for PCIe* Intel® 10 Gigabit Ethernet Network Connections Under Linux and save it into any directory of your choice and follow the README instructions to proceed next.

To install driver software from Intel:

  1. Install the driver software.
    content_copy zoom_out_map
    cd ~/intel_ixgbe/ixgbe-5.5.3/src     
    sudo make install
    
  2. Uninstall the old driver and load the updated driver by using the rmmod/modprobe command.
    content_copy zoom_out_map
    sudo rmmod ixgbe
    sudo modprobe ixgbe
    
    Warning:

    The command rmmod uninstalls the 10GE driver. If this is the only interface you are connected to, then access to the host will be lost.

  3. Verify if the new driver is installed correctly.
    content_copy zoom_out_map
    lab@ubuntu2:~/intel_ixgbe/ixgbe-5.5.3/src$ modinfo ixgbe | grep -i version
    version:        5.5.3
    

Prepare NIC to Use SR-IOV in System Mode

The host needs to be informed for each dedicated NIC by setting the sriov_numvfs value, how many VFs are going to use SR-IOV for the given NIC. The vmx.sh script have no information of how many VFs will use the shared NIC. Because of this, you must configure the sriov_numvfs accordingly.

This value can be set as a boot-option to be persistent after a reboot and can be changed on-the-fly which would not be persistent after a reboot.

The procedure given in this example is temporary solution for configuring sriov_numvfs using /sys Any setting to /sys/class/net/interface-name/device/sriov_numvfs is non-permanent, hence the configuration does not survive a reboot.

To prepare NIC to use SR-IOV, complete the following steps:

Create a virtual function (VF) using the following command:
content_copy zoom_out_map
# echo num_of_vf > /sys/class/net/interface-name/device/sriov_numvfs

Below command allows 4 VNFs to use shared NIC ens8f1 for SR-IOV. You must either use sudo or need login as root user.

content_copy zoom_out_map
As sudo user:
root@ubuntu2:~# echo 4 | sudo tee -a  /sys/class/net/ens8f1/device/sriov_numvfs
content_copy zoom_out_map
As root user
root@ubuntu2:~# echo 4 > /sys/class/net/ens8f1/device/sriov_numvfs
Note:

The sriov_numvfs option only accepts values 0-n, where n is the maximum number of VFs that are supported by the SR-IOV.

Setting SR-IOV at Boot-Time

The following procedures provide some alternate methods for configuring SR-IOV where the configuration persists a reboot of the host.

Following options are available to set the value during the boot-process of the host:

  • Using rc.local

  • Setting modprobe options

  • Setting kernel-paramater using grub

Below example shows a method to configure the sriov_numvfs value by using grub kernel command

You must set "intel_iommu=on" and ixgbe.max_vfs= value

For more information on hugepages, see Preparing the Ubuntu Host to Install vMX.

Edit the file “/etc/default/grub”:

content_copy zoom_out_map
lab@ubuntu2:~$ cat /etc/default/grub | grep -i cmd
GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on"
GRUB_CMDLINE_LINUX="isolcpus=34-41,48-55 default_hugepagesz=1G
hugepagesz=1G hugepages=120 ixgbe.max_vfs=8"

After editing, update the following:

content_copy zoom_out_map
sudo update-grub

Write new boot-loader to make changes active upon next reboot.

content_copy zoom_out_map
sudo grub-install /dev/sda

Reboot the host to make settings active.

content_copy zoom_out_map
sudo reboot

Verify sriov_numvfs Settings

Purpose

To verify the sriov_numvfs configuration using the CLI. In this example, the required NIC to use with SR-IOV is ens8f1 at PCI-address 85:00.0. Please note the “Virtual Function” in the output.

Action

content_copy zoom_out_map
lab@ubuntu2:~$ lspci | grep 85
content_copy zoom_out_map
85:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network
Connection (rev 01)
85:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network
Connection (rev 01)
85:10.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual
Function (rev 01)
85:10.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual
Function (rev 01)
85:10.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual
Function (rev 01)
85:10.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual
Function (rev 01)
85:10.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual
Function (rev 01)
85:10.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual
Function (rev 01)
85:10.6 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual
Function (rev 01)
85:10.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual
Function (rev 01)
85:11.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual
Function (rev 01)
85:11.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual
Function (rev 01)
85:11.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual
Function (rev 01)
85:11.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual
Function (rev 01)
85:11.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual
Function (rev 01)
85:11.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual
Function (rev 01)
85:11.6 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual
Function (rev 01)
85:11.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual
Function (rev 01)

The /sys/class/net/ens8f1/device/sriov_numvfs also contains the desired value of sriov_numvfs.

content_copy zoom_out_map
lab@ubuntu2:~$ cat /sys/class/net/ens8f1/device/sriov_numvfs 
content_copy zoom_out_map
8

For testing, the sriov_numvfs can be changed quickly by directly writing into /sys/class/net/interface-name/device/sriov_numvfs.

Changing the Number of sriov_numvfs

We recommend creating sriov_numvfs in advance (example: by using grub command at boot-time), because, changing the VF’s number is not allowed after deploying 1st vMX instance which uses given NIC with SR-IOV. If you must change the VF’s number, then you must stop the running vMX. We recommend to set the sriov_numvfs option to a higher value to avoid changing the sriov_numvfs number afterwards.

To change an already configured value of VFs, you must first change it's value as 0, and then change it to required integer value.

If there are no VFs assigned, the number of VFs can be changed to any valid value (0 - n, where n is the maximum number of VFs that are supported by the SR-IOV)

You must perform the following steps to modify the number of VFs:

  1. Stop running VNFs using the shared SR-IOV NIC.

  2. Disable the SR-IOV network adapter by setting the number of Virtual Functors (VFs) to 0.

    content_copy zoom_out_map
    As root:
    root@ubuntu2:~# echo 0 > /sys/class/net/ens8f1/device/sriov_numvfs
    
    content_copy zoom_out_map
    As sudo:
    lab@ubuntu2:~$ echo 0  | sudo tee -a /sys/class/net/ens8f1/device/sriov_numvfs
    
  3. Change the required number of VF (you are using six in this example)

    content_copy zoom_out_map
    root@ubuntu2:~# echo 6 > /sys/class/net/ens8f1/device/sriov_numvfs 
    
    Note:

    If you see the following error message, then first set the value to zero as described in step 2 before performing step 3

    content_copy zoom_out_map
    root@ubuntu2:~# echo 6 > /sys/class/net/ens8f1/device/sriov_numvfs bash: echo: write error: Device or resource busy
  4. Verify your configuration by using the following command:

    content_copy zoom_out_map
    root@ubuntu2:~# cat /sys/class/net/ens8f1/device/sriov_numvfs

Before restarting the vMX, adopt the vmx.conf file for SR-IOV usage.

Updating the VMX Configuration File (vmx.conf) Parameters

The parameters required to configure vMX are defined in the startup configuration file. The configuration file is in YAML format. The default file is config/vmx.conf. You can save your configuration file to a different name for different instances.

To configure interfaces for SR-IOV device types, you must specify the interface, the NIC, and the MAC address. Table 1 provides the details of the configuration parameters that we are using to change vmx.conf file.

Table 1: VMX Configuration File Parameters

Components

VM Parameters

Description

vPFE

device-type

Use sriov for all interfaces using the SR-IOV or use mixed to allow mixing of SR-IOV and non-SR-IOV-based interfaces.

use_native_drivers

Set to true to allow using the host’s Intel ixgbe driver (which was downloaded and complied in above steps)

Interfaces

type

If type is set to sriov, then port-speed-mbps and nic must be set.

port-speed-mbps

Set it to 10000 for 10GE NIC.

nic

The kernel-name for the interface to use.

virtual-function

Set to 0 for first vMX instance using this NIC. Ensure to set to 1 for 2nd vMX using this shared NIC (and so on)

mac-address

Ensure that each VF instance using the shared NIC is using a unique or different MAC address.

A sample vmx.conf file:

content_copy zoom_out_map
lab@ubuntu2:~/vmx/config$ cat vmx.conf.sriov
content_copy zoom_out_map
##############################################################
#
#  vmx.conf
#  Config file for vmx on the hypervisor.
#  Uses YAML syntax.
#  Leave a space after ":" to specify the parameter value.
#
##############################################################

---
#Configuration on the host side - management interface, VM images etc.
HOST:
    identifier                : vmx1   # Maximum 6 characters
    host-management-interface : ens4f0
    routing-engine-image      : "/home/lab/vmx/images/junos-vmx-x86-64-18.1R3-S2.5.qcow2"
    routing-engine-hdd        : "/home/lab/vmx/images/vmxhdd.img"
    forwarding-engine-image   : "/home/lab/vmx/images/vFPC-20181023.img"

---
#External bridge configuration
BRIDGES:
    - type  : external
      name  : br-ext                  # Max 10 characters

---
#vRE VM parameters
CONTROL_PLANE:
    vcpus       : 1
    memory-mb   : 1024
    console_port: 8601

    interfaces  :
      - type      : static
        ipaddr    : 10.102.144.94
        macaddr   : "0A:00:DD:C0:DE:0E"

---
#vPFE VM parameters
FORWARDING_PLANE:
    memory-mb   : 8192
    vcpus       : 4
    console_port: 8602
    device-type : mixed                      <<<< sriov or mixed
    use_native_drivers : true                <<<< use drivers as provided by the host

    interfaces  :
      - type      : static
        ipaddr    : 10.102.144.98
        macaddr   : "0A:00:DD:C0:DE:10"

---
#Interfaces
JUNOS_DEVICES:
   - interface            : ge-0/0/0
     type                 : virtio           <<< required 
     mac-address          : "02:06:0A:0E:FF:F0"
     description          : "ge-0/0/0 interface"

   - interface            : ge-0/0/1
     type                 : sriov           <<< required
     mtu                  : 9192            <<< required 
     port-speed-mbps      : 10000           <<< required
     nic                  : ens8f1          <<< required
     virtual-function     : 0               <<< use consecutive / unique numbers for each vMX instance
     mac-address          : "02:06:0A:0E:FF:F1"  <<< make sure that each vNF is using a DIFFERENT MAC-address
     description          : "ge-0/0/1 interface"

Start vmx-install

content_copy zoom_out_map
lab@ubuntu2:~/vmx$ sudo ./vmx.sh --install --cfg ./config/vmx.conf.sriov

Changes Required for Using Intel ixgbe Driver

When you try to move an existing deployment from modified IXGBE driver to unmodified IXGBE driver, enter edit mode in Junos CLI and use the following command when using "native" drivers.

content_copy zoom_out_map
user@host# set interfaces vlan-offload

For more information, see Modified and Unmodified IXGBE Driver.

external-footer-nav