Manage Multi-Cluster Contrail
SUMMARY Learn how to perform life cycle management tasks specific to a multi-cluster installation.
This section covers tasks that are specific to a multi-cluster installation. If you want to perform management tasks in a specific cluster (such as adding and removing nodes within a cluster and upgrading a cluster) within the multi-cluster installation, then see Manage Single Cluster Contrail.
Attach a Workload Cluster
- Attach a Workload Cluster Running Kernel Mode Data Plane in Release 22.1
- Attach a Workload Cluster Running Kernel Mode Data Plane in Release 22.2 or Higher
Attach a Workload Cluster Running Kernel Mode Data Plane in Release 22.1
Use this procedure to create and attach a distributed workload cluster (running a kernel mode data plane) to the central cluster in release 22.1. If you want to run a DPDK data plane, then extrapolate from the DPDK examples in this document.
The manifests that you will use in this example procedure are multi-cluster/dist-kernel/distributed_cluster_deployer.yaml and multi-cluster/dist-kernel/deployer_ha_kernel.yaml. The procedure assumes that you've placed these manifests into a manifests directory.
- Prepare the distributed workload cluster as described in Before You Install.
-
Create the distributed workload cluster.
Create a fresh cluster. You can follow the procedure in Create a Kubernetes Cluster or use your own methods to create a cluster. The cluster should have the following characteristics:
- Cluster has no CNI plug-in.
- Disable Node Local DNS.
- In a multi-cluster setup, you must configure different pod and service subnets on each cluster. These subnets must be unique within the entire multi-cluster.
- In a multi-cluster setup, you must configure different node names for nodes in each cluster. Node names must be unique across the entire multi-cluster.
-
Install Contrail components on the distributed workload cluster.
-
On the workload cluster, create the contrail-deploy
namespace.
user@workload:~/contrail$ kubectl create ns contrail-deploy
-
On the workload cluster, copy the kubeconfig from the central
cluster. We'll call this
central-cluster-kubeconfig here.
user@workload:~/contrail$ scp user@central:~/.kube/config central-cluster-kubeconfig
-
On the workload cluster, create a Kubernetes secret from the
central cluster kubeconfig and name the secret
central-kubeconfig.
Note:
You must name the secret central-kubeconfig.
user@workload:~/contrail$ kubectl create secret generic central-kubeconfig -n contrail-deploy --from-file=kubeconfig=/root/contrail/central-cluster-kubeconfig
Note:You must specify the absolute path to the central-cluster-kubeconfig file.
-
On the workload cluster, apply the deployer manifest. The
deployer provides life cycle management for the Contrail
components.
This manifest includes a reference to the central-kubeconfig secret you created in the previous substep.
user@workload:~/contrail$ kubectl apply -f manifests/distributed_cluster_deployer.yaml
-
Check that the contrail-deployer has come up. This may take a
few minutes.
user@workload:~/contrail# kubectl get pods -n contrail-deploy NAME READY STATUS RESTARTS AGE contrail-k8s-deployer-6458859585-xhwx6 1/1 Running 0 6m
-
On the workload cluster, create the contrail-deploy
namespace.
-
On the central cluster, attach the new workload cluster by creating a
kubemanager for the new workload cluster.
-
On the central cluster, copy the kubeconfig from the
distributed workload cluster. We'll call this
workload-cluster-kubeconfig here.
user@central:~/contrail$ scp user@workload:~/.kube/config workload-cluster-kubeconfig
-
On the central cluster, create a Kubernetes secret from the
distributed workload cluster kubeconfig and choose a meaningful
name for the secret (for example,
workload-kubeconfig).
user@central:~/contrail$ kubectl create secret generic workload-kubeconfig -n contrail --from-file=kubeconfig=/root/contrail/workload-cluster-kubeconfig
-
Create the kubemanager manifest with the following content.
Choose a meaningful name for the manifest (for example,
kubemanager-cluster1.yaml).
apiVersion: configplane.juniper.net/v1alpha1 kind: Kubemanager metadata: name: <CR name> namespace: contrail spec: common: containers: - image: <contrail-image-repository> name: contrail-k8s-kubemanager nodeSelector: node-role.kubernetes.io/master: "" podV4Subnet: <pod-v4-subnet-of-remote-cluster> serviceV4Subnet: <service-v4-subnet-of-remote-cluster> podV6Subnet: <pod-v6-subnet-of-remote-cluster> serviceV6Subnet: <service-v6-subnet-of-remote-cluster> clusterName: <worker-cluster-name> kubeconfigSecretName: <secret-name> enableNad: <true/false> listenerPort: <listener-port> constantRouteTargetNumber: <rt-number>Table 1: Kubemanager CRD Parameter Meaning Example name The name of the custom resource. kubemanager-cluster1 image The repository where you pull images hub.juniper.net/cn2/contrail-k8s-kubemanager:22.1.0.93 podV4Subnet The IPv4 pod subnet that you configured earlier for the distributed workload cluster. This subnet must be unique within the entire multi-cluster.
10.234.64.0 serviceV4Subnet The IPv4 service subnet that you configured earlier for the distributed workload cluster. This subnet must be unique within the entire multi-cluster.
10.234.0.0/18 podV6Subnet The IPv6 pod subnet that you configured earlier for the distributed workload cluster. This subnet must be unique within the entire multi-cluster.
fd85:ee78:d8a6:8608::1:0000/112 serviceV6Subnet The IPv6 service subnet that you configured earlier for the distributed workload cluster. This subnet must be unique within the entire multi-cluster.
fd85:ee78:d8a6:8608::1000/116 clusterName The name of the workload cluster. workload-cluster kubeconfigSecretName The name of the secret containing the workload cluster kubeconfig token. workload-kubeconfig enableNad True or false (to enable network address definition or not) true listenerPort The port that the Contrail controller listens on for communications with this workload cluster. Set the port for the first workload cluster to 19446 and increment by 1 for each subsequent workload cluster.
19446 constantRouteTargetNumber The route target for this workload cluster. Set the route target for the first workload cluster to 7699 and increment by 100 for each subsequent workload cluster.
7699 -
On the central cluster, apply the kubemanager manifest you just
created.
user@central:~/contrail$ kubectl apply -f manifests/kubemanager-cluster1.yaml
-
On the central cluster, verify that you can see the workload
cluster's namespaces.
kubectl get ns
The namespaces are in the following format:
<kubemanager-name>-<workload-cluster-name>-<namespace>. For example:kubemanager-workload1-workload-cluster-contrail
-
On the central cluster, copy the kubeconfig from the
distributed workload cluster. We'll call this
workload-cluster-kubeconfig here.
-
On the workload cluster, modify the
deployer_ha_kernel.yaml manifest as necessary.
The deployer_ha_kernel.yaml manifest that we
provide is a sample that you may need to tailor to your setup.
The provided deployer_ha_kernel.yaml file specifies different networks for the different cluster nodes, which is different from our example cluster. In our cluster, the cluster nodes all connect to the same data network. We'll need to modify deployer_ha_kernel.yaml to remove the extra networks and extra node configuration.
-
Remove the extra networks.
Edit deployer_ha_kernel.yaml to remove the following lines that define the
contrail-network-configConfigMap:--- apiVersion: v1 kind: ConfigMap metadata: name: contrail-network-config namespace: contrail data: networkConfig: | controlDataNetworks: - subnet: 5.5.6.0/24 gateway: 5.5.6.254 - subnet: 5.5.7.0/24 gateway: 5.5.7.254 - subnet: 5.5.8.0/24 gateway: 5.5.8.254 - subnet: 5.5.14.0/24 gateway: 5.5.14.254 -
Remove the following lines that define the
contrail-vrouter-nodes-7vRouter:--- apiVersion: dataplane.juniper.net/v1alpha1 kind: Vrouter metadata: name: contrail-vrouter-nodes-7 namespace: contrail spec: clusterName: distributed-cluster common: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: chassis operator: In values: - "5d10s3-node3" imagePullSecrets: - name: registrypullsecret containers: - image: hub.juniper.net/cn2/contrail-vrouter-agent:22.1.0.93 name: contrail-vrouter-agent - image: hub.juniper.net/cn2/contrail-init:22.1.0.93 name: contrail-watcher - image: hub.juniper.net/cn2/contrail-telemetry-exporter:22.1.0.93 name: contrail-vrouter-telemetry-exporter initContainers: - image: hub.juniper.net/cn2/contrail-init:22.1.0.93 name: contrail-init - image: hub.juniper.net/cn2/contrail-cni-init:22.1.0.93 name: contrail-cni-init maxUnavailablePercentage: 100% -
Similarly, remove the
contrail-vrouter-nodes-8section completely -
Similarly, remove the
contrail-vrouter-nodes-14section completely -
Add the following
contrail-vrouter-nodessection to the end:--- apiVersion: dataplane.juniper.net/v1alpha1 kind: Vrouter metadata: name: contrail-vrouter-nodes namespace: contrail spec: clusterName: distributed-cluster common: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: node-role.kubernetes.io/master operator: NotIn values: - "" imagePullSecrets: - name: registrypullsecret containers: - image: hub.juniper.net/cn2/contrail-vrouter-agent:22.1.0.93 name: contrail-vrouter-agent - image: hub.juniper.net/cn2/contrail-init:22.1.0.93 name: contrail-watcher - image: hub.juniper.net/cn2/contrail-telemetry-exporter:22.1.0.93 name: contrail-vrouter-telemetry-exporter initContainers: - image: hub.juniper.net/cn2/contrail-init:22.1.0.93 name: contrail-init - image: hub.juniper.net/cn2/contrail-cni-init:22.1.0.93 name: contrail-cni-init maxUnavailablePercentage: 100%
-
Remove the extra networks.
-
Finally, on the workload cluster, install the vRouter.
user@workload:~/contrail$ kubectl apply -f manifests/deployer_ha_kernel.yaml
After a few minutes, verify that all pods are up:
user@workload:~# kubectl get pods -A NAMESPACE NAME READY STATUS RESTARTS contrail-deploy contrail-k8s-deployer-6c88f6798f-jjx96 1/1 Running 3 contrail contrail-vrouter-masters-phttx 3/3 Running 0 contrail contrail-vrouter-nodes-xkkk8 3/3 Running 0 kube-system coredns-657959df74-26jrk 1/1 Running 0 kube-system dns-autoscaler-b5c786945-m4vtz 1/1 Running 0 kube-system kube-apiserver-k8s-cp0 1/1 Running 0 kube-system kube-controller-manager-k8s-cp0 1/1 Running 0 kube-system kube-proxy-5rbp4 1/1 Running 0 kube-system kube-proxy-kr95n 1/1 Running 0 kube-system kube-scheduler-k8s-cp0 1/1 Running 0 kube-system nginx-proxy-k8s-worker0 1/1 Running 0
Attach a Workload Cluster Running Kernel Mode Data Plane in Release 22.2 or Higher
Use this procedure to create and attach a distributed workload cluster (running a kernel mode data plane) to the central cluster in release 22.2 or higher. If you want to run a DPDK data plane, then extrapolate from the DPDK examples in this document.
The manifests that you will use in this example procedure are multi-cluster/distributed_cluster_deployer_example.yaml and multi-cluster/distributed_cluster_vrouter_example.yaml. The procedure assumes that you've placed these manifests into a manifests directory.
- Prepare the distributed workload cluster as described in Before You Install.
-
Create the distributed workload cluster.
Create a fresh cluster. You can follow the procedure in Create a Kubernetes Cluster or use your own methods to create a cluster. The cluster should have the following characteristics:
- Cluster has no CNI plug-in.
- Disable Node Local DNS.
- In a multi-cluster setup, you must configure different pod and service subnets on each cluster. These subnets must be unique within the entire multi-cluster.
- In a multi-cluster setup, you must configure different node names for nodes in each cluster. Node names must be unique across the entire multi-cluster.
-
Install Contrail components on the distributed workload cluster.
-
On the workload cluster, copy the kubeconfig from the central
cluster. We'll call this
central-cluster-kubeconfig here.
user@workload:~/contrail$ scp user@central:~/.kube/config central-cluster-kubeconfig
-
On the workload cluster, create a Kubernetes secret from the
central cluster kubeconfig and name the secret
central-kubeconfig.
Note:
You must name the secret central-kubeconfig.
user@workload:~/contrail$ kubectl create secret generic central-kubeconfig -n contrail-deploy --from-file=kubeconfig=/root/contrail/central-cluster-kubeconfig
Note:You must specify the absolute path to the central-cluster-kubeconfig file.
-
On the workload cluster, apply the deployer manifest. The
deployer provides life cycle management for the Contrail
components.
This manifest includes a reference to the central-kubeconfig secret you created in the previous substep.
user@workload:~/contrail$ kubectl apply -f manifests/distributed_cluster_deployer_example.yaml
-
Check that the contrail-deployer has come up. This may take a
few minutes.
user@workload:~/contrail# kubectl get pods -n contrail-deploy NAME READY STATUS RESTARTS AGE contrail-k8s-deployer-6458859585-xhwx6 1/1 Running 0 6m
-
On the workload cluster, copy the kubeconfig from the central
cluster. We'll call this
central-cluster-kubeconfig here.
-
On the central cluster, attach the new workload cluster by creating a
kubemanager for the new workload cluster.
-
On the central cluster, copy the kubeconfig from the
distributed workload cluster. We'll call this
workload-cluster-kubeconfig here.
user@central:~/contrail$ scp user@workload:~/.kube/config workload-cluster-kubeconfig
-
On the central cluster, create a Kubernetes secret from the
distributed workload cluster kubeconfig and choose a meaningful
name for the secret (for example,
workload-kubeconfig).
user@central:~/contrail$ kubectl create secret generic workload-kubeconfig -n contrail --from-file=kubeconfig=/root/contrail/workload-cluster-kubeconfig
-
Create the kubemanager manifest with the following content.
Choose a meaningful name for the manifest (for example,
kubemanager-cluster1.yaml).
apiVersion: configplane.juniper.net/v1alpha1 kind: Kubemanager metadata: name: <CR name> namespace: contrail spec: common: containers: - image: <contrail-image-repository> name: contrail-k8s-kubemanager nodeSelector: node-role.kubernetes.io/master: "" podV4Subnet: <pod-v4-subnet-of-remote-cluster> serviceV4Subnet: <service-v4-subnet-of-remote-cluster> podV6Subnet: <pod-v6-subnet-of-remote-cluster> serviceV6Subnet: <service-v6-subnet-of-remote-cluster> clusterName: <worker-cluster-name> kubeconfigSecretName: <secret-name> enableNad: <true/false> listenerPort: <listener-port> constantRouteTargetNumber: <rt-number>Table 2: Kubemanager CRD Parameter Meaning Example name The name of the custom resource. kubemanager-cluster1 image The repository where you pull images hub.juniper.net/cn2/contrail-k8s-kubemanager:22.1.0.93
enterprise-hub.juniper.net/contrail-container-prod/contrail-k8s-kubemanager:22.3.0.71
podV4Subnet The IPv4 pod subnet that you configured earlier for the distributed workload cluster. This subnet must be unique within the entire multi-cluster.
10.234.64.0 serviceV4Subnet The IPv4 service subnet that you configured earlier for the distributed workload cluster. This subnet must be unique within the entire multi-cluster.
10.234.0.0/18 podV6Subnet The IPv6 pod subnet that you configured earlier for the distributed workload cluster. This subnet must be unique within the entire multi-cluster.
fd85:ee78:d8a6:8608::1:0000/112 serviceV6Subnet The IPv6 service subnet that you configured earlier for the distributed workload cluster. This subnet must be unique within the entire multi-cluster.
fd85:ee78:d8a6:8608::1000/116 clusterName The name of the workload cluster. workload-cluster kubeconfigSecretName The name of the secret containing the workload cluster kubeconfig token. workload-kubeconfig enableNad True or false (to enable network address definition or not) true listenerPort The port that the Contrail controller listens on for communications with this workload cluster. Set the port for the first workload cluster to 19446 and increment by 1 for each subsequent workload cluster.
19446 constantRouteTargetNumber The route target for this workload cluster. Set the route target for the first workload cluster to 7699 and increment by 100 for each subsequent workload cluster.
7699 -
On the central cluster, apply the kubemanager manifest you just
created.
user@central:~/contrail$ kubectl apply -f manifests/kubemanager-cluster1.yaml
-
On the central cluster, verify that you can see the workload
cluster's namespaces.
kubectl get ns
The namespaces are in the following format:
<kubemanager-name>-<workload-cluster-name>-<namespace>. For example:kubemanager-workload1-workload-cluster-contrail
-
On the central cluster, copy the kubeconfig from the
distributed workload cluster. We'll call this
workload-cluster-kubeconfig here.
-
Finally, on the workload cluster, install the vRouter.
user@workload:~/contrail$ kubectl apply -f manifests/distributed_cluster_vrouter_example.yaml
After a few minutes, verify that all pods are up:
user@workload:~# kubectl get pods -A NAMESPACE NAME READY STATUS RESTARTS contrail-deploy contrail-k8s-deployer-6c88f6798f-jjx96 1/1 Running 3 contrail contrail-vrouter-masters-phttx 3/3 Running 0 contrail contrail-vrouter-nodes-xkkk8 3/3 Running 0 kube-system coredns-657959df74-26jrk 1/1 Running 0 kube-system dns-autoscaler-b5c786945-m4vtz 1/1 Running 0 kube-system kube-apiserver-k8s-cp0 1/1 Running 0 kube-system kube-controller-manager-k8s-cp0 1/1 Running 0 kube-system kube-proxy-5rbp4 1/1 Running 0 kube-system kube-proxy-kr95n 1/1 Running 0 kube-system kube-scheduler-k8s-cp0 1/1 Running 0 kube-system nginx-proxy-k8s-worker0 1/1 Running 0
Detach a Workload Cluster
-
On the central cluster, delete the associated kubemanager.
user@central:~/contrail$ kubectl delete kubemanager -n contrail <kubemanager-name>
-
On the distributed workload cluster that you're deleting, delete the
vRouters.
user@workload:~/contrail$kubectl delete vrouter -n contrail contrail-vrouter-masters
user@workload:~/contrail$ kubectl delete vrouter -n contrail contrail-vrouter-nodes
-
On the central cluster, delete the namespaces of the distributed workload
cluster.
List the namespaces and delete all namespaces associated with the distributed workload cluster.
user@central:~/contrail$ kubectl get ns
user@central:~/contrail$ kubectl delete ns <kubemanager-name>-<workload-cluster-name>-<namespace>
-
On the central cluster, delete the secret associated with the workload
cluster that you're detaching.
kubectl delete secret <secret-name> -n contrail
Uninstall Contrail
- Follow the steps in Detach a Workload Cluster to detach the workload cluster that you want to delete.
-
Uninstall Contrail in the workload cluster.
-
Follow the steps in Uninstall Contrail to uninstall Contrail in the workload cluster.
Ignore any NotFound errors because those resources have already been deleted.
-
On the workload cluster, verify that all resources related to
default pod network namespace of the distributed workload cluster
are gone, and verify that all resources related to the contrail
namespace are gone.
user@workload:~/contrail$ kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get --show-kind --ignore-not-found -n <default pod network namespace>
user@workload:~/contrail$ kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get --show-kind --ignore-not-found -n contrail
If all you want to do is to uninstall Contrail from the workload cluster, then you're done. If you want to also uninstall Contrail from the central cluster, then proceed to the next step.
-
Follow the steps in Uninstall Contrail to uninstall Contrail in the workload cluster.
-
Uninstall Contrail in the central cluster.
-
Follow the steps in Uninstall Contrail to uninstall Contrail in the central cluster.
Ignore any NotFound errors because those resources have already been deleted.
-
On the central cluster, verify that all resources related to the
contrail, contrail-system, and contrail-deploy namespaces are
deleted.
user@central:~/contrail$ kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get --show-kind --ignore-not-found -n contrail
user@central:~/contrail$ kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get --show-kind --ignore-not-found -n contrail-system
user@central:~/contrail$ kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get --show-kind --ignore-not-found -n contrail-deploy
-
Follow the steps in Uninstall Contrail to uninstall Contrail in the central cluster.