System Requirements for GCP Deployment

Rocky Linux: 4.18.X

The K8s version for Google Cloud Marketplace JCNR subscription is v1.27.5.

cat /etc/modules-load.d/vfio.conf
vfio
vfio-pci

echo Y > /sys/module/vfio_iommu_type1/parameters/allow_unsafe_interrupts
echo Y > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode

GRUB_CMDLINE_LINUX_DEFAULT="console=tty1 console=ttyS0 default_hugepagesz=1G hugepagesz=1G hugepages=64 intel_iommu=on iommu=pt"

Update grub and reboot the host. For example:

grub2-mkconfig -o /boot/grub2/grub.cfg

reboot

Verify the hugepage is set by executing the following commands:

cat /proc/cmdline
grep -i hugepages /proc/meminfo

Set IOMMU and IOMMU-PT in GRUB.

GRUB_CMDLINE_LINUX_DEFAULT="console=tty1 console=ttyS0 default_hugepagesz=1G hugepagesz=1G hugepages=64 intel_iommu=on iommu=pt"

Update grub and reboot.

grub2-mkconfig -o /boot/grub2/grub.cfg 

reboot

Additional kernel modules need to be loaded on the host before deploying JCNR in L3 mode. These modules are usually available in linux-modules-extra or kernel-modules-extra packages.

Create a /etc/modules-load.d/crpd.conf file and add the following kernel modules to it:

tun
fou
fou6
ipip
ip_tunnel
ip6_tunnel
mpls_gso
mpls_router
mpls_iptunnel
vrf
vxlan

Enable kernel-based forwarding on the Linux host.

ip fou add port 6635 ipproto 137

Enable IP Forwarding for VMs in GCP.

1. Specify it as an option while creating the VM. For example:

   gcloud compute instances create instance-name --can-ip-forward

2. For an exisiting VM, enable IP forwarding by updating the compute instance via a file. For example:

   gcloud compute instances export transit-jcnr01 --project jcnr-ci-admin --zone us-west1-a --destination=instance_file_1

   Edit the instance file to set the value canIpForward=true.

   Update the compute instance from the file:

   gcloud compute instances update-from-file transit-jcnr01 --project jcnr-ci-admin --zone us-west1-a --source=instance_file_1 --most-disruptive-allowed-action ALLOWED_ACTION

Enable Multi-IP subnet on Guest OS.

gcloud compute images create debian-9-multi-ip-subnet \
     --source-disk debian-9-disk \
     --source-disk-zone us-west1-a \
     --guest-os-features MULTI_IP_SUBNET 

Add firewall rules for loopback address for VPC.

Configure the VPC firewall rule to allow ingress traffic with source filters set to the subnet range to which JCNR is attached, along with the IP ranges or addresses for the loopback addresses.

For example:

Navigate to Firewall policies on the GCP console and create a firewall rule with the following attributes:

1. Name: Name of the firewall rule

2. Network: Choose the VPC network

3. Priority: 1000

4. Direction: Ingress

5. Action on Match: Allow

6. Source filters: 10.2.0.0/24, 10.51.2.0/24, 10.51.1.0/24, 10.12.2.2/32, 10.13.3.3/32

7. Protocols: all

8. Enforcement: Enabled

where 10.2.0.0/24 is the subnet to which JCNR is attached and 10.51.2.0/24, 10.51.1.0/24, 10.12.2.2/32, and 10.13.3.3/32 are loopback IP ranges.

Exclude JCNR interfaces from NetworkManager control.

NetworkManager is a tool in some operating systems to make the management of network interfaces easier. NetworkManager may make the operation and configuration of the default interfaces easier. However, it can interfere with Kubernetes management and create problems.

To avoid NetworkManager from interfering with JCNR interface configuration, exclude JCNR interfaces from NetworkManager control. Here's an example on how to do this in some Linux distributions:

1. Create the /etc/NetworkManager/conf.d/crpd.conf file and list the interfaces that you don't want NetworkManager to manage.

   For example:

   [keyfile]
    unmanaged-devices+=interface-name:enp*;interface-name:ens*

   where enp* and ens* refer to your JCNR interfaces.
   Note: enp* indicates all interfaces starting with enp. For specific interface names, provided a comma-separated list.
2. Restart the NetworkManager service:

   sudo systemctl restart NetworkManager

3. Edit the /etc/sysctl.conf file on the host and paste the following content in it:

   net.ipv6.conf.default.addr_gen_mode=0
   net.ipv6.conf.all.addr_gen_mode=0
   net.ipv6.conf.default.autoconf=0
   net.ipv6.conf.all.autoconf=0

4. Run the command sysctl -p /etc/sysctl.conf to load the new sysctl.conf values on the host.

Verify the core_pattern value is set on the host before deploying JCNR.

sysctl kernel.core_pattern
kernel.core_pattern = |/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e

You can update the core_pattern in /etc/sysctl.conf. For example:

kernel.core_pattern=/var/crash/core_%e_%p_%i_%s_%h_%t.gz