System Requirements for Azure Deployment

Rocky Linux: 4.18.X

uio_hv_generic

To enable, add the following modules to be loaded at boot:

cat /etc/modules-load.d/k8s.conf 
uio
uio_hv_genericib_uverbs
mlx4_ib

The above libraries are provided by ibverbs package.

GRUB_CMDLINE_LINUX_DEFAULT="console=tty1 console=ttyS0 default_hugepagesz=1G hugepagesz=1G hugepages=6 intel_iommu=on iommu=pt"

Update grub and reboot the host. For example:

grub2-mkconfig -o /boot/grub2/grub.cfg

reboot

Verify the hugepage is set by executing the following commands:

cat /proc/cmdline
grep -i hugepages /proc/meminfo

Set IOMMU and IOMMU-PT in GRUB.

GRUB_CMDLINE_LINUX_DEFAULT="console=tty1 console=ttyS0 default_hugepagesz=1G hugepagesz=1G hugepages=64 intel_iommu=on iommu=pt"

Update grub and reboot.

grub2-mkconfig -o /boot/grub2/grub.cfg 

reboot

Additional kernel modules need to be loaded on the host before deploying JCNR in L3 mode. These modules are usually available in linux-modules-extra or kernel-modules-extra packages.

Create a /etc/modules-load.d/crpd.conf file and add the following kernel modules to it:

tun
fou
fou6
ipip
ip_tunnel
ip6_tunnel
mpls_gso
mpls_router
mpls_iptunnel
vrf
vxlan

Enable kernel-based forwarding on the Linux host.

ip fou add port 6635 ipproto 137

Add firewall rules for loopback address for VPC.

Configure the VPC firewall rule to allow ingress traffic with source filters set to the subnet range to which JCNR is attached, along with the IP ranges or addresses for the loopback addresses.

For example:

Navigate to Firewall policies on the Azure console and create a firewall rule with the following attributes:

1. Name: Name of the firewall rule

2. Network: Choose the VPC network

3. Priority: 1000

4. Direction: Ingress

5. Action on Match: Allow

6. Source filters: 10.2.0.0/24, 10.51.2.0/24, 10.51.1.0/24, 10.12.2.2/32, 10.13.3.3/32

7. Protocols: all

8. Enforcement: Enabled

where 10.2.0.0/24 is the subnet to which JCNR is attached and 10.51.2.0/24, 10.51.1.0/24, 10.12.2.2/32, and 10.13.3.3/32 are loopback IP ranges.

Set the MTU on all fabric interfaces to 1500 bytes.

After JCNR comes up, use the cRPD CLI to set the MTU size on all fabric interfaces to 1500 bytes. Microsoft Azure Cloud Platform recommends an MTU size less than or equal to 1500 bytes on all interfaces that connect directly to the Azure infrastructure. These interfaces are the JCNR fabric interfaces. Failure to follow this rule might lead to packet drops.

For information on how to access the cRPD CLI, see Access cRPD CLI.

Ensure accelerated networking is enabled for the fabric interface.

If accelerated networking is enabled properly, two interfaces become available for the fabric interface. For example:

3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:22:48:23:3b:9e brd ff:ff:ff:ff:ff:ff
    inet 10.225.0.6/24 brd 10.225.0.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::222:48ff:fe23:3b9e/64 scope link 
       valid_lft forever preferred_lft forever
4: enP22960s2: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master eth1 state UP group default qlen 1000
    link/ether 00:22:48:23:3b:9e brd ff:ff:ff:ff:ff:ff
    altname enP22960p0s2

When configuring the fabric interface in the Helm chart, you must provide the interface with hv_netvsc bound to it. Issue the ethtool -i interface_name command to verify it. For example:

user@jcnr01:~# ethtool -i eth1
driver: hv_netvsc
version: 5.15.0-1049-azure
firmware-version: N/A
expansion-rom-version: 
bus-info: 
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: no

Exclude JCNR interfaces from NetworkManager control.

NetworkManager is a tool in some operating systems to make the management of network interfaces easier. NetworkManager may make the operation and configuration of the default interfaces easier. However, it can interfere with Kubernetes management and create problems.

To avoid NetworkManager from interfering with JCNR interface configuration, exclude JCNR interfaces from NetworkManager control. Here's an example on how to do this in some Linux distributions:

1. Create the /etc/NetworkManager/conf.d/crpd.conf file and list the interfaces that you don't want NetworkManager to manage.

   For example:

   [keyfile]
    unmanaged-devices+=interface-name:enp*;interface-name:ens*

   where enp* and ens* refer to your JCNR interfaces.
   Note: enp* indicates all interfaces starting with enp. For specific interface names, provided a comma-separated list.
2. Restart the NetworkManager service:

   sudo systemctl restart NetworkManager

3. Edit the /etc/sysctl.conf file on the host and paste the following content in it:

   net.ipv6.conf.default.addr_gen_mode=0
   net.ipv6.conf.all.addr_gen_mode=0
   net.ipv6.conf.default.autoconf=0
   net.ipv6.conf.all.autoconf=0

4. Run the command sysctl -p /etc/sysctl.conf to load the new sysctl.conf values on the host.

Verify the core_pattern value is set on the host before deploying JCNR.

sysctl kernel.core_pattern
kernel.core_pattern = |/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e

You can update the core_pattern in /etc/sysctl.conf. For example:

kernel.core_pattern=/var/crash/core_%e_%p_%i_%s_%h_%t.gz