Manage Multi-Cluster CN2
SUMMARY Learn how to perform life cycle management tasks specific to a multi-cluster installation.
This section covers tasks that are specific to a multi-cluster installation. If you want to perform management tasks in a specific cluster (such as adding and removing nodes within a cluster and upgrading a cluster) within the multi-cluster installation, then see Manage Single Cluster CN2.
Attach a Workload Cluster
Use this procedure to create and attach a distributed workload cluster (running a kernel mode data plane) to the central cluster. If you want to run a DPDK data plane, then extrapolate from the DPDK examples in this document.
The manifests that you will use in this example procedure are multi-cluster/distributed_cluster_deployer_example.yaml and multi-cluster/distributed_cluster_vrouter_example.yaml. The procedure assumes that you've placed these manifests into a manifests directory.
- Prepare the distributed workload cluster as described in Before You Install.
-
Create the distributed workload cluster.
Create a fresh cluster. You can follow the procedure in Create a Kubernetes Cluster or use your own methods to create a cluster. The cluster should have the following characteristics:
- Cluster has no CNI plug-in.
- Disable Node Local DNS.
- In a multi-cluster setup, you must configure different pod and service subnets on each cluster. These subnets must be unique within the entire multi-cluster.
- In a multi-cluster setup, you must configure different node names for nodes in each cluster. Node names must be unique across the entire multi-cluster.
-
Install CN2 components on the distributed workload cluster.
-
On the workload cluster, copy the kubeconfig from the central
cluster. We'll call this
central-cluster-kubeconfig here.
scp user@central:~/.kube/config central-cluster-kubeconfig
-
On the workload cluster, create the contrail-deploy
namespace.
kubectl create ns contrail-deploy
-
On the workload cluster, create a Kubernetes secret from the
central cluster kubeconfig and name the secret
central-kubeconfig.
Note:
You must name the secret central-kubeconfig.
kubectl create secret generic central-kubeconfig -n contrail-deploy --from-file=kubeconfig=/root/contrail/central-cluster-kubeconfig
Note:You must specify the absolute path to the central-cluster-kubeconfig file.
-
On the workload cluster, apply the deployer manifest. The deployer
provides life cycle management for the CN2 components.
This manifest includes a reference to the central-kubeconfig secret you created in the previous substep.
kubectl apply -f manifests/distributed_cluster_deployer_example.yaml
-
Check that the contrail-deployer has come up. This may take a few
minutes.
kubectl get pods -n contrail-deploy
NAME READY STATUS RESTARTS AGE contrail-k8s-deployer-6458859585-xhwx6 1/1 Running 0 6m
-
On the workload cluster, apply the cert-manager manifest. The
cert-manager provides encryption for all management and control
plane connections.
kubectl apply -f manifests/distributed_cluster_certmanager_example.yaml
-
On the workload cluster, copy the kubeconfig from the central
cluster. We'll call this
central-cluster-kubeconfig here.
-
On the central cluster, attach the new workload cluster by creating a
kubemanager for the new workload cluster.
-
On the central cluster, copy the kubeconfig from the distributed
workload cluster. We'll call this
workload-cluster-kubeconfig here.
scp user@workload:~/.kube/config workload-cluster-kubeconfig
-
On the central cluster, create a Kubernetes secret from the
distributed workload cluster kubeconfig and choose a meaningful name
for the secret (for example,
workload-kubeconfig).
kubectl create secret generic workload-kubeconfig -n contrail --from-file=kubeconfig=/root/contrail/workload-cluster-kubeconfig
-
Create the kubemanager manifest with the following content. Choose
a meaningful name for the manifest (for example,
kubemanager-cluster1.yaml).
apiVersion: configplane.juniper.net/v1alpha1 kind: Kubemanager metadata: name: <CR name> namespace: contrail spec: common: containers: - image: <contrail-image-repository> name: contrail-k8s-kubemanager podV4Subnet: <pod-v4-subnet-of-remote-cluster> serviceV4Subnet: <service-v4-subnet-of-remote-cluster> podV6Subnet: <pod-v6-subnet-of-remote-cluster> serviceV6Subnet: <service-v6-subnet-of-remote-cluster> clusterName: <worker-cluster-name> kubeconfigSecretName: <secret-name> enableNad: <true/false> listenerPort: <listener-port> constantRouteTargetNumber: <rt-number>Table 1: Kubemanager CRD Parameter Meaning Example name The name of the custom resource. kubemanager-cluster1 image The repository where you pull images enterprise-hub.juniper.net/contrail-container-prod/contrail-k8s-kubemanager:23.3.0.180 podV4Subnet The IPv4 pod subnet that you configured earlier for the distributed workload cluster. This subnet must be unique within the entire multi-cluster.
10.234.64.0 serviceV4Subnet The IPv4 service subnet that you configured earlier for the distributed workload cluster. This subnet must be unique within the entire multi-cluster.
10.234.0.0/18 podV6Subnet The IPv6 pod subnet that you configured earlier for the distributed workload cluster. This subnet must be unique within the entire multi-cluster.
fd85:ee78:d8a6:8608::1:0000/112 serviceV6Subnet The IPv6 service subnet that you configured earlier for the distributed workload cluster. This subnet must be unique within the entire multi-cluster.
fd85:ee78:d8a6:8608::1000/116 clusterName The name of the workload cluster. workload-cluster kubeconfigSecretName The name of the secret containing the workload cluster kubeconfig token. workload-kubeconfig enableNad True or false (to enable network address definition or not) true listenerPort The port that the Contrail controller listens on for communications with this workload cluster. Set the port for the first workload cluster to 19446 and increment by 1 for each subsequent workload cluster.
19446 constantRouteTargetNumber The route target for this workload cluster. Set the route target for the first workload cluster to 7699 and increment by 100 for each subsequent workload cluster.
7699 -
On the central cluster, apply the kubemanager manifest you just
created.
kubectl apply -f manifests/kubemanager-cluster1.yaml
-
On the central cluster, verify that you can see the workload
cluster's namespaces.
kubectl get ns
The namespaces are in the following format:
<kubemanager-name>-<workload-cluster-name>-<namespace>. For example:kubemanager-workload1-workload-cluster-contrail
-
On the central cluster, copy the kubeconfig from the distributed
workload cluster. We'll call this
workload-cluster-kubeconfig here.
-
Finally, on the workload cluster, install the vRouter.
kubectl apply -f manifests/distributed_cluster_vrouter_example.yaml
After a few minutes, verify that all pods are up:
kubectl get pods -A NAMESPACE NAME READY STATUS RESTARTS cert-manager cert-manager-7d6d9465db-mgbk4 1/1 Running 0 cert-manager cert-manager-cainjector-6f567667c6-vm4 1/1 Running 0 cert-manager cert-manager-webhook-f947d7c68-tlfx8 1/1 Running 0 contrail-deploy contrail-k8s-deployer-6c88f6798f-jjx96 1/1 Running 3 contrail contrail-vrouter-masters-phttx 3/3 Running 0 contrail contrail-vrouter-nodes-xkkk8 3/3 Running 0 kube-system coredns-657959df74-26jrk 1/1 Running 0 kube-system dns-autoscaler-b5c786945-m4vtz 1/1 Running 0 kube-system kube-apiserver-k8s-cp0 1/1 Running 0 kube-system kube-controller-manager-k8s-cp0 1/1 Running 0 kube-system kube-proxy-5rbp4 1/1 Running 0 kube-system kube-proxy-kr95n 1/1 Running 0 kube-system kube-scheduler-k8s-cp0 1/1 Running 0 kube-system nginx-proxy-k8s-worker0 1/1 Running 0
Detach a Workload Cluster
-
On the central cluster, delete the associated kubemanager.
kubectl delete kubemanager -n contrail <kubemanager-name>
-
On the distributed workload cluster that you're deleting, delete the
vRouters.
kubectl delete vrouter -n contrail contrail-vrouter-masters
kubectl delete vrouter -n contrail contrail-vrouter-nodes
-
On the central cluster, delete the namespaces of the distributed workload
cluster.
List the namespaces and delete all namespaces associated with the distributed workload cluster.
kubectl get ns
kubectl delete ns <kubemanager-name>-<workload-cluster-name>-<namespace>
-
On the central cluster, delete the secret associated with the workload
cluster that you're detaching.
kubectl delete secret <secret-name> -n contrail
Uninstall CN2
- Follow the steps in Detach a Workload Cluster to detach the workload cluster that you want to delete.
-
Uninstall CN2 in the workload cluster.
-
Follow the steps in Uninstall CN2 to uninstall CN2 in the workload cluster.
Ignore any NotFound errors because those resources have already been deleted.
-
On the workload cluster, verify that all resources related to
default pod network namespace of the distributed workload cluster
are gone, and verify that all resources related to the contrail
namespace are gone.
kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get --show-kind --ignore-not-found -n <default pod network namespace>
kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get --show-kind --ignore-not-found -n contrail
If all you want to do is to uninstall CN2 from the workload cluster, then you're done. If you want to also uninstall CN2 from the central cluster, then proceed to the next step.
-
Follow the steps in Uninstall CN2 to uninstall CN2 in the workload cluster.
-
Uninstall CN2 in the central cluster.
-
Follow the steps in Uninstall CN2 to uninstall CN2 in the central cluster.
Ignore any NotFound errors because those resources have already been deleted.
-
On the central cluster, verify that all resources related to the
contrail, contrail-system, and contrail-deploy namespaces are
deleted.
kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get --show-kind --ignore-not-found -n contrail
kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get --show-kind --ignore-not-found -n contrail-system
kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get --show-kind --ignore-not-found -n contrail-deploy
-
Follow the steps in Uninstall CN2 to uninstall CN2 in the central cluster.