Contrail Ingress

Contrail Ingress Load Balancer

Like Contrail’s service implementation, Contrail Ingress is also implemented through load balancer, but with a different loadbalancer_provider attribute. Accordingly, the contrail-svc-monitor component takes different actions to implement Ingress in Contrail environments.

Remember that way back in the Contrail/Kubernetes architecture section we gave the object mapping between Kubernetes and Contrail, and that Kubernetes service maps to ECMP load balancer (native) and Ingress maps to Haproxy load balancer.

In the service section when we were exploring the load balancer and the relevant objects (listener, pool, and member), we noticed the load balancer’s loadbalancer_provider type is native.

In this section we’ll see the loadbalancer_provider type is opencontrail for ingress’s load balancer. We’ll also look into the similarities and differences between service load balancer and Ingress load balancer.

Contrail Ingress Workflow

• The contrail-kube-manager keeps listening to the events of the kube-apiserver.

• User creates an ingress object (rules).

• The contrail-kube-manager gets the event from kube-apiserver.

• The contrail-kube-manager creates a load balancer object in contrail DB, and sets the load balancer_ provider type as open contrail for ingress (where as it is native for service).

• If the load balancer_provider type is native, it will invoke the ECMP loadbalancer driver for ECMP load balancing, which we reviewed in the previous section.

• If the load balancer_provider type is open contrail, it will invoke the haproxy load balancer driver, which triggers the haproxy processes to be launched in Kubernetes nodes.

• For each Ingress object, one load balancer will be created;

• Two haproxy processes will be created for Ingress and they are working in active-standby mode:

  • one compute node runs the active haproxy process

  • the other compute node runs the standby haproxy process

• Both haproxy processes are programmed with appropriate configuration, based on the rules defined in the Ingress object.

Contrail Ingress Traffic Flow

Client requests, such as a type of overlay traffic, may come from two different sources, depending on who initiates the request:

• An internal request: requests coming from another pod inside of the cluster.

• An external request: requests coming from an Internet host outside of the cluster.

The only difference between the two is how the traffic hits the active haproxy. An ingress will be allocated two IPs: a cluster-internal virtual IP and an external virtual IP.

1. For internal requests, it hits Ingress’s internal VIP directly.
2. For external requests, it first hits Ingress’s external VIP – the floating IP, which is the one exposed to external, and that’s when NAT starts to play as we’ve explained in the FIP section. After NAT processing, traffic is forwarded to the internal Ingress VIP.
3. From this moment on, both types of requests are processed exactly the same way.
4. The requests will be proxied to the corresponding service IP.
5. Based on the availability of the backend pods, it will be sent to the node where one of the backend pods are located and eventually reaches the target pods.
6. In case the backend pods are running in a different compute node than the one running active haproxy, a MPLS over UDP tunnel is created between the two compute nodes.

Figure 1 and Figure 2 illustrate the end-to-end service request flow when accessing from a pod in the cluster and when accessing from an Internet host.

Contrail supports all three types of ingress:

• HTTP-based single-service ingress

• simple-fanout ingress

• name-based virtual hosting

ingress Next we’ll look into each type of ingress

Figure 1: Ingress Traffic Flow: Access from Internal

Figure 2: Ingress Traffic Flow: Access from External

Ingress Setup

This book’s lab uses the same testbed as used for the service test, shown in Figure 3.

Figure 3: Ingress Testbed

Single Service Ingress

Single service ingress is the most basic form of ingress. It does not define any rules and its main use is to expose service to the outside world. It proxies all incoming service requests to the same single backend service:

To demonstrate single service type of ingress, the objects that we need to create are:

• an Ingress object that defines the backend service

• a backend service object

• at least one backend pod for the service

Ingress Definition

In the single service ingress test lab, we want to request that any URLs are directed to service-web-clusterip with servicePort 8888. Here is the corresponding YAML definition file:

This does not look fancy. Basically there is nothing else but a reference to a single service webserver-1 as its backend. All HTTP requests will be dispatched to this service, and from there the request will reach a backend pod. Simple enough. Let’s look at the backend service.

Backend Service Definition

You can use the exact same service as introduced in the service example:

Backend Pod Definition

Just as in the service example, you can use exactly the same webserver deployment to launch backend pods:

All in One YAML File

As usual, you can create an individual YAML file for each of the objects, but considering that these objects always need to be created and removed together in Ingress, it’s better to merge definitions of all these objects into one YAML file. YAML syntax supports this by using document delimiters (a --- line between each object definition):

• You can create/update all objects in the YAML file in one go, using just one kubectl apply command.

• Similarly, if anything goes wrong and you need to clean up, you can delete all objects created with the YAML file in one kubectl delete command.

• Whenever needed, you can still delete each individual object independently, by providing the object name.

Deploy the Single Service Ingress

Before applying the YAML file to get all the objects created, let’s take a quick look at our two nodes. You want to see if there is any haproxy process running without ingress, so later, after you deploy ingress, you can compare:

So the answer is no, there is no haproxy process in either of the nodes. Haproxy will be created only after you create Ingress and the corresponding load balancer object is seen by the contrail-service-monitor. We’ll check this again after an Ingress is created:

The ingress, one service, and one deployment object have now been created.

Ingress Object

Let’s examine the ingress object:

As expected, the backend service is properly applied to the ingress. In this single-service ingress there are no explicit rules defined to map a certain URL to a different service – all HTTP requests will be dispatched to the same backend service.

• 10.47.255.x is a cluster-internal podIP allocated from the pod’s default subnet, and

• 101.101.101.x is the public FIP associated with an internal IP. The question is: Why does ingress even require a podIP and FIP?

Let’s hold off on the answer for now and continue to check the service and pod object created from the all-in-one YAML file. We’ll come back to this question shortly.

Service Objects

Let’s check on

services:

The service is created and allocated a clusterIP. We’ve seen this before and it looks like nothing special. Now, let’s look at the backend and client pods:

Everything looks fine, here. There is a backend pod running for the service. You have already learned how selector and label works in service-pod associations. Nothing new here. So let’s examine the haproxy and try to make some sense out of the two IPs allocated to the ingress object.

Haproxy Processes

Earlier, before the ingress was created, we were looking for the haproxy process in nodes but could not see anything. Let’s check it again and see if any magic happens:

On node cent222:

On node cent333:

And right after ingress is created, you can see a haproxy process created in each of our two nodes! Previously we stated that Contrail Ingress is also implemented through load balancer (just like service). Since ingress’s loadbalancer_provider type is opencontrail, ‘contrail-svc-monitor invokes the haproxy load balancer driver. The haproxy driver generates the required haproxy configuration for the ingress rules and triggers haproxy processes to be launched (in active-standby mode) with the generated configuration in Kubernetes nodes.

Ingress Load Balancer Objects

We’ve mentioned the ingress load balancer a few times but haven’t looked at it yet. In the service section, we’ve looked into the service load balancer object in the UI, and we inspected some details about the object data structure. Now, after creating the ingress object, let’s check the list of load balancer objects again and see what ingress brings in here starting with the UI at Configure>Load Balancers.

Figure 4: UI: Load Balancers Configuration

• Load balancer ns-user-1 ingress-ss for ingress ingress-ss

• Load balancer ns-user-1 webservice-clusterip for service webserver-clusterip

We’ve been through the service load balancer object previously, and if you expand the service you will see lots of detail but nothing should surprise you.

Figure 5: Service Load Balancer Object (click the triangle in the left of the load balancer)

As you can see, the service load balancer has a clusterIP and a listener object that is listening on port 8888. One thing to highlight is the loadbalancer_provider. This type is native, so the action contrail-svc-monitor takes is the Layer 4 (Application Layer) ECMP process, which is explored extensively in the service section. Let’s expand the ingress load balancer and glance at the details.

Figure 6: Ingress Load Balancer Object

• The loadbalancer_provider is opencontrail

• The ingress load balancer has a reference to a virtual-machine-interface (VMI) object

• And the VMI object is referred to by an instance-ip object with a (fixed) IP 10.47.255.238 and a floating-ip object with a (floating) IP 101.101.101.1.

• It is a cluster-internal IP address allocated from the default pod network as a load balancer VIP

• It is the frontend IP that the ingress load balancer will listen to for HTTP requests

• And it is also what the public floating IP 101.101.101.1 maps to with NAT.

• Ingress pod IP is the VIP facing other pods inside of a cluster. To reach ingress from inside of the cluster, requests coming from other pods will have their destination IP set to Ingress podIP.

• Ingress floating IP is the VIP facing the Internet host outside world. To reach ingress from outside of the cluster, requests coming from Internet hosts need to have their destinations IP set to Ingress FIP. When the node receives traffic destined to the ingress floating IP from outside of the cluster, the vRouter will translate it into the Ingress podIP.

The detailed ingress load balancer object implementation refers to a service instance, and the service instance includes other data structures or references to other objects (VMs, VMIs, etc.). Overall it is more complicated and involves more details than what’s been covered in this book. We’ve tailored some of the details into a high-level overview so that important concepts like haproxy and the two ingress IPs can at least be understood.

Once an HTTP/HTTPS request arrives at the ingress podIP, internally or externally, the ingress load balancer will do HTTP/HTTPS proxy operations through the haproxy process, and dispatch the requests towards the service and eventually to the backend pod.

We’ve seen that the haproxy process is running, to examine more details of this proxy operation, let’s check its configuration file for details on the running parameters.

Haproxy.conf File

• The haproxy frontend represents the frontend of an ingress, facing clients.

• The haproxy backend represents the backend of an ingress, facing services.

• The haproxy frontend defines a bind to the ingress podIP and mode http. These knobs indicate what the frontend is listening to.

• The haproxy backend section defines the server, which is a backend service in our case. It has a format of serviceIP:servicePort, which is the exact service object we’ve created using the all-in-one YAML file.

• The default_backend in the frontend section defines which backend is the default: it will be used when a haproxy receives a URL request that has no explicit match anywhere else in the frontend section. In this case the default_backend refers to only the backend service 10.97.226.91:8888 This is due to the fact that there are no rules defined in single service Ingress, so all HTTP requests will go to the same default_backend service, regardless of what URL the client sent.

Figure 7: Single Service Ingress

Throughout this configuration, the haproxy implemented our single service ingress.

Gateway Router VRF Table

• The vRouter advertises the floating IP prefix to Contrail Controller through XMPP. At least two pieces of information from the output indicate who represents the floating IP in this example - node cent222:

  • The Protocol next hop being 10.169.25.20

  • The Route Distinguisher being 10.169.25.20:5

• And through MP-BGP, Contrail Controller reflects the floating IP prefix to the gateway router. Source: 10.169.25.19 indicates this fact.

So, it looks like cent222 is selected to be the active haproxy node and the other node, cent333, is the standby one. Therefore you should expect a client request coming from the Internet host to go to node cent222 first. Of course, the overlay traffic will be carried in MPLS over the GRE tunnel, same as what you’ve seen from the service example.

The floating IP advertisement towards the gateway router is exactly the same in all types of ingresses.

Another fact that we’ve somewhat skipped on purpose is the different local preference value used by the active and standby node when advertising the floating IP prefix. A complete examination involves other complex topics, like the active node selection algorithm, and so on, but it is worth it to understand this from a high level.

Both nodes have load balancer and haproxy running, so both will advertise the floating IP prefix 101.101.101.1 to the gateway router. However, they are advertised with different local preference values. The active node will advertise with a value of 200 and the standby node with 100. Contrail Controller both have routes from the two nodes, but only the winning one will be advertised to the gateway router. That is why the other BGP route is dropped and only one is displayed. Localpref being 200 proves it is coming from the active compute node. This applies to both the ingress public floating IP route and the internal VIP route advertisement.

Ingress Verification: Internal

You still use the curl command to trigger HTTP requests towards the ingress’s private IP. The return proves our Ingress works: requests towards different URLs are all proxied to the same backend pods, through the default backend service, service-web-clusterip.

In the fourth request we didn’t give a URL via -H, so curl will fill host with the request IP address, 10.47.255.238 in this test, and again it goes to the same backend pod and gets the same returned response.

Ingress Verification: External (Internet host)

Now, from the Internet host’s desktop, launch your browser, and input one of the three URLs. By refreshing the pages you can confirm all HTTP requests are returned by the same backend pod, as shown in Figure 8.

Figure 8: Accessing www.juniper.net from the Internet Host

Simple Fanout Ingress

Both the simple fanout Ingress and name-based virtual host Ingress support URL routing, the only difference is that the former is based on path and the latter is based on host.

• An Ingress object: it defines the rules, mapping two paths to two backend services

• Two backend services objects

• Each service requires at least one pod as a backend

• The same client pod as the cluster-internal client used in previous examples.

Ingress Objects Definition

• Requests toward path /dev will be directed to a service webservice-1 with servicePort 8888.

• Requests toward path /qa will be directed to a service webservice-2 with servicePort 8888. Here is the corresponding YAML file to implement these goals:

In contrast to single service Ingress, in simple fanout Ingress object (and name-based virtual host Ingress) you can see rules defined – here it is the mappings from multiple paths to different backend services.

Backend Service Definition

Since we defined two rules each for a path, you need two services, accordingly. You can clone the previous service in the single service Ingress example and just change that service’s name and selector to generate the second service. For example, this is definition of webservice-1 and webservice-2 service:

Backend Pod Definition

Because there are two backend services now, you also need at least two backend pods each with a label matching to a service. You can clone the previous Deployment into two and just change the name and label of the second Deployment.

Deploy Simple Fanout Ingress

Now apply the all-in-one YAML file to create all objects:

The ingress, two service, and two Deployment objects are now created.

Ingress Post Examination

The rules are defined properly, and within each rule there is a mapping from a path to the corresponding service. You can see the same ingress internal podIP and external floating IP as seen in the previous single service Ingress example:

That is why, from the gateway router’s perspective, there are no differences between all the types of ingress. In all cases, a public floating IP will be allocated to the ingress and it is advertised to the gateway router:

Now, check the backend services and pods. First the service objects:

Then the backend and client pods:

Two services are created, each with a different allocated clusterIP. For each service there is a backend pod. Later, when we verify ingress from the client, we’ll see these podIPs in the returned web pages.

Contrail Ingress Load Balancer Object

Compared with single service Ingress, the only difference is one more service load balancer as shown in the next screen capture.

Figure 9: Simple Fanout Ingress Load Balancers (UI: configuration > Networking > Floating IPs)

• Load balancer ns-user-1 ingress-sf for ingress ingress-sf

• Load balancer ns-user-1 webservice-1 for service webserver-1

• Load balancer ns-user-1 webservice-2 for service webserver-2

We won’t explore the details of the objects again since we’ve investigated the key parameters of service and Ingress load balancers in single service Ingress and there is really nothing new here

Haproxy Process and Haproxy.cfg File

In the single service Ingress example, we demonstrated two haproxy processes invoked by contrail-svc-monitor when it sees loadbalancer appearing with loadbalancer_provider and set to opencontrail. At the end of that example, after we removed the single service Ingress, since there is no more Ingress left in the cluster, the two haproxy processes will be ended. Now, with a new ingress creation, the two new haproxy processes are invoked again:

Node cent222:

Node cent333:

This time what interests us is how the simple fanout Ingress rules are programmed in the haproxy.conf file. Let’s look at the haproxy configuration file:

• The haproxy frontend section: It now defines URLs. Each URL is represented by a pair of acl statements, one for the host, and the other for the path. In a nutshell, host is the domain name and path is what follows the host in the URL string. Here, for simple fanout Ingress there is the host www.juniper. net with two different paths: \dev and \qa.

• The haproxy backend section: Now there are two of them. For each path there is a dedicated service.

• The use_backend…if… command in the frontend section: This statement declares the ingress rules – if the URL request includes a specified path that matches to what is programmed in one of the two ACL pairs, use the corresponding backend (that is a service) to forward the traffic.

For example, acl 020e371c-e222-400f-b71f-5909c93132de_path path /qa defines path /qa. If the URL request contains such a path, haproxy will use_backend 020e371c-e222-400f-b71f-5909c93132de, which you can find in the backend section. The backend is a UUID referring to server c13b0d0d-6e4a-4830-bb46-2377ba4caf23 10.100.156.38:8888 weight 1, which is essentially a service. You can identify this by look- ing at the serviceIP:port: 10.100.156.38:8888.

The configuration file is illustrated in Figure 10.

Figure 10: Simple Fanout Service

• If the full URL is composed of host www.juniper.net and path /dev, the request will be dispatched to webservice-1 (10.96.51.227:8888).

• If the full URL is composed of host www.juniper.net and path /qa, the request will be dispatched to webservice-2 (10.100.156.38:8888).

• For any other URLs the request will be dropped because there is no corresponding backend service defined for it.

Ingress Verification: from Internal

Let’s test the URL with different paths:

The returned output shows the Ingress works: the two requests towards the /qa and /dev paths are proxied to two different backend pods through two backend services: webservice-1 and webservice-2, respectively.

The third request with a path abc composes an unknown URL which does not have a matching service in Ingress configuration, so it won’t be served. It’s the same for the last two requests. Without a path, or with a different host, the URLs become unknown to the ingress so they won’t be served.

You may think that you should be adding more rules to include these scenarios. Doing that works fine, but it’s not scalable – you can never cover all the possible paths and URLs that could come into your server. As we mentioned earlier, one solution is to use the default_backend service to process all other HTTP requests, which happens to be covered in the next example.

Ingress Verification: From External (Internet Host)

When you test simple fanout Ingress from outside of the cluster, the command is the same as what you’ve done for initiating the HTTP request from inside of a pod, except this time you are initiating from an Internet host. Let’s send the HTTP requests to the ingress’s public floating IP, instead of its internal podIP. So, from an Internet host machine:

Virtual Hosting Ingress

Virtual hosting ingress supports routing HTTP traffic to multiple host names at the same IP address. Based on the URL and the rules, an ingress load balancer directs traffic to different backend services, and each service directs traffic to its backend pods, like in this diagram:

• An Ingress object: the rules that map two URLs to two backend services

• Two backend services objects

• Each service requires at least one pod as a backend

Ingress Objects Definition

• A request toward URL www.juniper.net will be directed to a service webservice-1 with servicePort 8888.

• A request toward URL www.cisco.com will be directed to a service webservice-2 with servicePort 8888.

• A request toward any URLs other than these two, will be directed to webservice-1 with servicePort 8888. Effectively we want webservice-1 to become the default backend service.

And here is the corresponding YAML definition file:

Backend Service and Pod Definition.

The same exact service and Deployment definition that were used in simple fanout Ingress can be used here. And to be even briefer, here’s the all-in-one YAML file:

Now let’s apply the all-in-one YAML file to create ingress and the other necessary objects:

You can see that the Ingress, two services, and two Deployment objects have now been created.

Ingress Post Examination Let’s examine the Ingress object:

Compared to simple fanout Ingress, this time you can see two hosts instead of one. Each host represents a domain name:

The rules are defined properly, and within each rule there is a mapping from a host to the corresponding service. Note that the services, pods, and floating IP prefix advertisement to gateway router behavior are all exactly the same as those in simple fanout Ingress.

Exploring Ingress Load Balancer Objects

Three load balancers were generated after we applied the all-in-one YAML file, one for ingress, and two for services.

Load balancers created in this test are almost the same as the ones created in simple fanout Ingress test, as shown in the next screen capture, Figure 11.

Figure 11: Load Balancers

Okay so let’s check the haproxy configuration file for name-based

virtual host Ingress. Here’s an examination of the haproxy.conf file:

• The haproxy frontend section defines each URL, or host, and its path. Here the two hosts are www.juniper.net and www.cisco.com and for both path is /.

• The haproxy backend section defines the servers, which is all service in our case. It has a format of serviceIP:servicePort, which is what the service created.

• The use_backend…if… command in the frontend section declares the ingress rules: if the request includes a specified URL and path, use the corresponding backend to forward the traffic.

• The default_backend defines the service that will act as the default: it will be used when a haproxy receives a URL request that has no explicit match in the defined rules.

The workflow of the configuration file is illustrated in Figure 12.

Figure 12: The Haproxy Configuration File’s Workflow

• If www.juniper.net and / compose the full URL, request will be dispatched to webservice-1 (10.99.225.17:8888).

• If www.cisco.net and / compose the full URL, request will be dispatched to webservice-2 (10.105.134.79:8888).

• Other URLs go to the default backend, which is service

webservice-1. Let’s go ahead and verify these behaviors.

Ingress Verification: From Internal

From inside of cluster:

The Ingress works. The two requests towards Juniper and Cisco are proxied to two different backend pods, through two backend services, webservice-1 and webservice-2, respectively. The third request towards Google is an unknown URL, which does not have a matching service in Ingress configuration, so it goes to the default backend service, webservice-1, and reaches the same backend pod.

The same rule applies to the fourth request. When not given a URL using -H, curl will fill the host with the request IP address, in this case 10.47.255.238. Since that URL doesn’t have a defined backend service, the default backend service will be used. In our lab, we use backend pods for each service spawned by the same Deployment, so the podIP in a returned webpage tells us who is who. Except in the second test the returned podIP was 10.47.255.235, representing webservice-2, while the other three tests returned the podIP for webservice-1, as expected.

Ingress Verification: from External (Internet host)

From an Internet host’s desktop, we launched two Chrome pages side-by-side and input www.juniper.net in one and www.cisco.com in the other, and kept refreshing the pages. We can confirm that the Juniper page is always returned by the Deployment webserver-1 pod 10.47.255.236, and the Cisco page is always returned by the Deployment webserver-2 pod 10.47.255.235. Then we launched a third Chrome page towards www.google.com, and it was returned by the same pod serving the Juniper instance, as shown by the screen shots in Figure 13.

Figure 13: Internet Host

The same result can be seen from curl, too. Here it’s shown from the Internet host machine:

Service Versus Ingress Traffic Flow

Even though both service and ingress are implemented via load balancers (but with different loadbalancer_ provider types), the forwarding modes for service and ingress are quite different. With service forwarding it’s a one-hop process: the client sends the request to the clusterIP or the floating IP. With NAT, the request reaches the destination backend pod; while with Ingress forwarding, the traffic takes a two-hop process to arrive at the destination pod. The request first goes to the active haproxy, which then starts a HTTP/HTTPS level proxy procedure and does the service forwarding to reach the final pod. NAT processing happens in both forwarding processes, since both ingress and service public floating IP implementation relies on it.

Chapter 7 provides a detailed view of this Contrail packet flow.