Observability

Visualization

NorthStar is an MPLS underlay and selective overlay controller. Its genesis is rooted in the rise of centralization of traffic policy and control. Rather than aiming to be an illusory single pane of glass, it stays grounded by implementing well-defined traffic engineering functionality.

This focus of this chapter will not be administering NorthStar, but enabling an existing deployment to add value to an SR network.

In turn, we will also stay grounded. First, we’ll visualize the network built so far. A new BGP address family – link-state – will be enabled between the area border routers (ABR) and NorthStar’s Junos VM. This exports the topology for each area – nodes, links, link attributes, and prefixes. Both Junos and NorthStar have been enhanced to support export of SR-specific capabilities, such as the SRGB range, node, and adjacency SIDs.

Apply the configuration below to the ABRs in both New York and Washington – p1.nyc, p2.nyc, p1.iad, and p2.iad, substituting router-specific values as appropriate. This provides redundancy by configuring all the ABRs in a region to export area-specific topology; it also ensures coverage of the entire multi-level network:

Let’s take a peek at what p1.nyc is exporting from both its areas. We’ll review one example from each of the nodes, links, and prefixes and how that information has been enriched with SR attributes.

Control plane: p1.nyc exporting pe2.nyc as a node, as well as its SR attributes

Control Plane: p1.nyc exporting the link between itself and pe1.nyc, and its local adjacency SID

Control plane: p1.nyc exporting pe2.iad’s prefix and associated SID index

The sum total of the BGP-LS exchange allows NorthStar to construct a graph of the network as shown in Figure 1 and Figure 2.

Figure 1: Topology View With ASNs Discovered After Device Configuration Parsing

Figure 2: NorthStar Device Collection

Besides unmarshalling BGP-LS ‘routes,’ NorthStar can collect, via NETCONF, and parse device configuration. You can do this by traversing it’s menu at the top right (Administration > Task Scheduler > Add > Device Collection). Be sure to specify a non-root user’s credentials.

Collecting device configuration is optional but improves the user experience. For instance, the ASNs to which our network is attached become visible in the UI. You can drill into Layer 3 VPN VRFs to discover their PEs, attachment circuits, and route targets. These aren’t particulars exported by BGP-LS. Configuration parsing presents the operator with multiple aspects of their network.

But this is a book about segment routing. NorthStar has several valuable SR-specific visualizations. Each link can display not just an IP address, but also its locally-allocated adjacency SID. You can select this by clicking the gear icon in the workspace (below the +/- zoom buttons in Figure 3). From there, choose Links > Show Label > IP, SID A::Z.

Figure 3: Northstar Topology View With Adjacency SIDs Selected

Another outstandingly useful feature is to right-click a router and select Node SIDs from selected node. The labels of the other router change to display the node SID alongside their hostname. This occurs by adding each node’s (IPv4-only at the moment) index to the advertised SRGB of the router’s neighbors.

As a best practice, the SRGB base and size have been kept consistent in our network. In networks where this isn’t or can’t be followed, this feature alone can be a lifesaver.

We’ll pause our exploration of NorthStar as a visualization tool. We’ll return to it in its day job as a traffic engineering controller. At that point, we’ll build upon this view-only exercise by calculating and provisioning SR-TE policies (the proper name for SR-TE LSPs).

Figure 4: Northstar Shows Node SIDs Based On a Single Router’s Point of View

Telemetry

Befitting its radical nature, Junos’ SR implementation forgoes exporting any statistics via SNMP. Long known to be antiquated, SNMP is being superseded by vastly more capable protocols and frameworks that organize information.

On the protocol front, gRPC obsoletes SNMP. It inherits the benefits of HTTP/2, such as:

• Binary: Unlike text, serializing and un-serializing binary protocols is more efficient, compact, and less prone to errors over aspects such as white space handling. gRPC uses Protocol Buffers to encode and decode data. SNMP uses ASN.1.

• Multiplexed: This allows multiple outstanding requests over a single TCP connection, while still allow- ing for parallelism. In practice, SNMP is UDP-based and inherits its non-guaranteed nature.

• Bidirectional: Unlike its predecessor, HTTP/2 allows the server to push potentially useful data to the client, without being explicitly requested.

gRPC supports mutual certificate-based authentication; the rarely used SNMPv3 only supports shared key authentication. gRPC also offers both event-driven and periodic publish/subscribe approaches. This is similar to SNMP’s polling of statistics and relying on unsolicited traps to convey state changes. What makes gRPC more powerful is that it allows a client to express interest in subsets of the state information tree; as that state changes, the server publishes the update to clients – this could include a router’s ARP or ND table, LSP state, et al.

Building upon this improved transport is OpenConfig, an initiative to model networks in a vendor-neutral manner. This is an attempt to improve upon SNMP’s method of organizing information – the Management Information Base (MIB) – which resulted in sparse vendor-neutral support. Both configuration and operational data models have been defined by OpenConfig for the most widely used features – BGP, LLDP, and of course, SR. Others remain in the works.

The robust transport protocol and intuitive data model representation makes modern telemetry easier for the operator.

For instance, this is how an SNMP poll for an interface’s operational status would traditionally be done. First, the interface’s ifIndex would have to be discovered. This is an arbitrary integer value assigned to an interface by the device. All interface-specific state and counters are keyed using that index. In long-standing environments the ifIndex would be cached, possibly reassigned, and consequently relearned on device reboot. Then the state and counters associated with that interface would have to be polled:

IF-MIB::ifDescr is a human-friendly abbreviation for the underlying OID – .1.3.6.1.2.1.2. Not every branch of the MIB has a human-friendly representation. In contrast, the equivalent subtree path with OpenConfig is /interfaces/interface[name=’ge-0/0/0’]/state/oper-status/. It doesn’t take a leap in logic to understand the intent even if the operator is initially unfamiliar with this X-Path syntax.

Enabling OpenConfig With gRPC Transport

Enabling gRPC transport is trivial for testing purposes. Enabling it for production does overwhelmingly imply certificate-based authentication. As a result, it requires a functional public key infrastructure (PKI), specifically a certificate authority (CA), that can issue digital certificates.

There are six steps needed to enable and secure gRPC:

1. Instantiate a root CA. This will need to be done on an x86 host. The example below uses OpenSSL to create a private key (.key extension) and self-signed certificate (.crt extension).

2. You will next generate a private key and certificate signing request (CSR, .csr extension) for the gRPC client. In our example, jtimon is the gRPC client. The x86 host it runs on needs to present its identity to the gRPC servers (routers) for mutual authentication. The previously created CA can sign off on the CSR, generating a certificate for jtimon to use.

3. Each router needs to generate a private key and CSR. The CA will then fulfil each CSR, as in the previous step, generating a certificate for the respective router. Both the certificate and private key are bundled together (.pem extension). The example below is for pe1.nyc:

4. All three identities – the root CA’s and the jtimon host’s certificates, as well as the router’s own bundle – should be copied to the /var/tmp location on each router. As an example, this is what results on pe1.nyc:

5. The certificates then need to be activated via configuration. The example below is for pe1.nyc:

6. Finally, gRPC can be enabled to listen on a secure port and require mutual authentication. A username and password will need to be provided by jtimon – the user needs to be present at the[system login] configuration hierarchy on the router:

Strictly for testing purposes, you may wish to use gRPC insecurely. Do so outside a lab environment at your own peril. Skip steps 1-5 above and replace the stanza in step 6 with:

SR Telemetry

With this backdrop, let’s explore which SR statistics can be streamed. The most fundamental is the volume of segment routed traffic transiting an interface. During a migration from other MPLS transport protocols, this sensor can differentiate how much of the forwarding plane has shifted away to SR.

You can configure this as follows:

This installs sensors for each interface that has an SR next-hop.

Management plane: Verify the OpenConfig sensors have been installed

You can now subscribe to the OpenConfig path that exports these statistics (/mpls/signaling-protocols/segment-routing/interfaces/). Configure jtimon using a file similar to the one below, replacing the authentication credentials and paths to the private key and certificates as needed:

The configuration is self-explanatory. The router is requested to publish the SR interface statistics every 10 seconds. Launch jtimon and observe what pe1.nyc reports:

As an ingress PE, you can see that inbound IP traffic from CEs has been sent out with an SR label via the interface connected to p1.nyc. It doesn’t explain the traffic's destination as the imposed label isn’t communicated. Note that the counter name (oc-84) uses an arbitrary number, unrelated to labels in use.

Pointing jtimon at p1.nyc shows bidirectional traffic flow. An equal number of packets ingress and egress ge-0/0/2 and ge-0/0/3 – these interfaces connect to pe1.nyc and pe2.nyc, respectively:

As a final step, instrumenting pe2.nyc shows SR traffic egressing ge-0/0/1, which is pe2.nyc’s local interface to p1.nyc:

The eagle-eyed will have observed that p1.nyc reports an average packet size 4 bytes larger than the PE routers – confirming the imposition of a single node SID derived MPLS label.

This is a good start, and we can only get more granular. Per-SID statistics, especially with a uniform domain-wide SRGB, can be enormously useful in calculating traffic matrices. The configuration to enable that is:

Management plane: Verify additional OpenConfig sensors have been installed per-SID

An array of additional sensors has been created. The ingress per-SID sensor names correspond to the IPv4 and IPv6 node as well as adjacency SIDs. In the egress direction, they are named based on the underlying protocol and advertised prefix SID.

Now change the path that jtimon subscribes to: /mpls/signaling-protocols/segment-routing/aggregate-sid-counters/. The detail wells up from pe1.nyc, p1.nyc, and pe2.nyc to include per-SID traffic volume.

Management plane: Per-SID statistics at pe1.nyc

Management plane: Per-SID statistics at p1.nyc

Management plane: Per-SID statistics at pe2.nyc

What is fantastic about this is that it shows unerringly how much traffic is destined from pe1.nyc to pe2. nyc, and vice versa. Observing p1.nyc’s counters shows how much traffic is destined to pe1.nyc or pe2.nyc from other PEs. With a universal SRGB, each router shares a common understanding of the label associated with a node SID. By summing counters associated with the same MPLS label, the operator can see how much traffic is being sent to the node SID originator through each transit router. Accumulating counters associated with a prefix SID shows the amount of traffic sent to a remote router. This can greatly simplify the construction of traffic matrices.

SR-TE Telemetry

Nowhere is telemetry more important than Segment Routing Traffic Engineering (SR-TE). Gleaning the volume of traffic directed into an SR-TE policy – loosely speaking, an SR-TE LSP – starts the feedback loop that can be used by a centralized control mechanism that can reroute the LSP based on bandwidth needs and availability.

To focus on its telemetry aspect, an SR-TE LSP has been created between pe2.nyc and pe1.iad. It is a colored LSP, which means it has been assigned an arbitrary, locally-significant numeric value. When a service route is learned via BGP, and carries an extended color community with a matching value, that route’s protocol next hop is automatically pointed at this SR-TE LSP.

Traffic destined to service routes that resolve over the colored LSP is accounted for as SR-TE statistics. As earlier, SR-TE telemetry is specifically enabled via:

A simple, colored SR-TE LSP configuration is in effect:

Management plane: confirm sensor is attached to SR-TE LSP

Control plane: confirm service route points at SR-TE LSP

By default, there are two sensors attached to an SR-TE LSP. The ingress sensor measures the amount of IP traffic directed into the policy, and the transit sensor measures the amount of labeled traffic similary directed into the policy. The transit sensor is necessary if the binding SID (bSID), another locally-significant and separate from the color numeric identifier, associated with the SR-TE policy is used to steer MPLS-labeled traffic into the LSP.

As with all segment routing sensors, nothing is exported until non-zero statistics accumulate. As soon as IP traffic matching the colored service route arrives, the counters increase and are subsequently published at the frequency requested by the gRPC client.

Management plane: SR-TE IP statistics

The IP policy name, intuitively enough, contains the endpoint of the LSP as well as its color. If MPLS labeled traffic had been forwarded into this LSP, using the binding SID, the policy name would additionally contain the bSID identifier. Also inarguable is SR’s treatment of IPv6 as a first-class citizen throughout the entire stack – from creating a colored SR-TE policy towards an IPv6 endpoint, to directing IPv6 traffic into that LSP, ending with receiving telemetry for the effort.

SR EPE Telemetry

The last application of SR telemetry is with egress peer engineering (EPE). Broadly speaking, an autonomous system boundary router (ASBR) can associate a label with a BGP peer, a specific link to a given BGP peer, or a set of BGP peers. This label is only installed in that ASBR’s forwarding plane, and is exported to a controller via BGP-LS as an EPE SID. The controller can craft SR-TE policies that lead beyond a particular ASBR to a distinct peer, a unique link to one peer, or even a collection of peers.

As used earlier, an SR-TE LSP has been installed on pe2.nyc to a particular BGP peer of pe1.iad. This requires the EPE label configured at pe1.iad to be present at the bottom of the segment list in the SR-TE policy. SR-EPE telemetry needs to be explicitly enabled, as you would have come to expect:

Control plane: confirm sensor is installed for the EPE SID

Traffic matching the colored service route is transmitted, resulting in pe1.iad emitting telemetry for the EPE SID:

OAM (Operations, Administration, Maintenance)

Admittedly, OAM is often an afterthought for network engineers. It’s only when things tilt sideways in production that many reach for tools that assert not just control plane path liveness, but also verify data plane hygiene.

This is even more important for SR-TE as it eliminates path signaling. Instead, it delegates this to sBFD (Seamless Bidirectional Forwarding Detection). BFD has been a workhorse for detecting data path failures. Each node in a point-to-point relationship starts by learning the other’s discriminator, the node and application identifier in BFD-speak. Following this discovery, a BFD session establishes for connectivity checking.

sBFD simplifies BFD by eliminating the handshake needed to learn discriminators. Instead, the operator assigns at least one local discriminator to each node in the network. In order to establish sBFD for an SR- TE path, the head-end is configured with the remote node’s discriminator.

The head-end acts as the initiator and pushes the SR-TE path’s label stack onto the liveness checking packets and ensuring the packets traverse the same path that user traffic will; the SR-TE policy destination statelessly responds to the sBFD packets. The SR-TE path is viable as long as the sBFD session is up at the ingress router.

You can configure sBFD by adding a discriminator on each node in the network. Let’s add sBFD liveness checking to the IPv4 SR-TE path we have defined from pe2.nyc to pe1.iad:

After adding this configuration the SR-TE path will remain Up as long as sBFD forwarding exists between the two routers, using the same label stack.

Control plane: Verify pe2.nyc’s sBFD session to pe1.iad is Up

Control plane: Verify pe2.nyc’s SR-TE policy has a valid path

The first hop uses the adjacency SID to p1.nyc (21). If the adjacency between pe2.nyc and p1.nyc falters, this SR-TE policy will be brought Down.

Control plane: Verify the SR-TE policy goes Down if the label stack is rendered unusable

There are no alternate SID lists, otherwise known as paths, for the policy to utilize. User traffic will resolve over a non-SR-TE path, if one is available. In contrast, statically configured SR-TE paths that don’t take advantage of sBFD remain misleadingly Up.

Debuggability

Telemetry is but one part of observability. Troubleshooting network performance and availability depends on both the clarity of the underlying protocols and the diagnostics and logging of the available implementations.

SR’s minimization of state on transit routers, and its use of an uniform SRGB, make it easier for a human to parse node SID label forwarding actions – there is less to grok. Junos builds on this simplicity with a rich logging functionality to detect human error.

The SRMS section gave an example of how misconfigured node SIDs were flagged. There are two different types of conflicts that SR identifies:

1. SID conflict: this is caught if different SIDs are assigned to the same prefix.

2. Prefix conflict: this occurs if different prefixes are assigned the same SID.

The earlier example demonstrated a SID conflict. Let’s temporarily force a prefix conflict and use gRPC to stream that as an event. Traditional syslog remains available of course, but just like SNMP being succeeded by gRPC for telemetry, events can now be transmitted over gRPC as well.

Let’s configure jtimon to subscribe to /junos/events/. To cause a prefix conflict, pe1.iad is temporarily configured with an additional loopback address that actually belongs to p1.iad (128.49.106.9/32). This would be fine if these addresses were used to generate a common anycast SIDs. Instead, pe1.iad is misconfigured to advertise a different prefix index (99) than p1.iad (9) using an IS-IS export policy.

Control plane: pe1.iad misconfiguration to cause a prefix conflict

Management plane: streamed system event about the prefix conflict

Attention is beckoned and the misconfiguration recitifed. SR specifies a number of tie-breaking rules to address the conflict until human intervention takes place. The rules are evolving. At this point they are:

1. Prefer IPv6 entries over IPv4

2. Prefer longest prefix lengths

3. Prefer the numerically lowest prefix address

4. Prefer the numerically lowest segment index

Prefix conflicts are resolved first; SID conflict resolution follows on the outstanding entries. With this, let’s surface from our dive into observability and move on to optimizability.