Multinode High Availability

Services Redundancy Groups

A services redundancy group (SRG) is a failover unit in a Multinode High Availability setup. There are two types of SRGs:

• SRG0—Manages security service from Layer 4-Layer 7 except IPsec VPN services. The SRG0 operates in active mode on both nodes at any point in time. On SRG0, each security session must traverse the node in a symmetric flow, Backup of these flows are fully state-synchronized to the other node,
• SRG1+—Manages IPsec services and virtual IPs for hybrid and default gateway mode and are backed up to the other node. The SRG1 operates in active mode on one node and in backup node on another node.

Figure 10 shows SRG0 and SRG1 in a Multinode High Availability setup.

Figure 10: Single SRG Support in Multinode High Availability (Active-Backup Mode)

Figure 11 shows SRG0 and SRG1+ in a Multinode High Availability setup.

Figure 11: Multi SRG Support in Multinode High Availability (Active-Active Mode)

Starting in Junos OS Release 22.4R1, you can configure Multinode High Availability to operate in active-active mode with support of multi SRG1s (SRG1+). In this mode, some SRGs remain active on one node and some SRGs remain active on another node. A particular SRG always operates in active-backup mode; it operates in active mode on one node and backup mode on another node. In this case, both the nodes can have the active SRG1 forwarding stateful services. Each node has a different set of floating IP addresses assigned to SRG1+.

Table 3 explains the behavior of SRGs in a Multinode High Availability setup.

Starting in Junos OS Release 23.4R1, the Multinode High Availability setup operates in a combined mode. You don’t have to reboot the system when you add or delete any SRG (SRG0 or SRG1+) configurations.

Activeness Determination and Enforcement

In a Multinode High Availability setup, activeness is determined at the service level, not at the node level. The active/backup state is at the SRG level and the traffic is steered toward the active SRG. SRG0 remains active on both the nodes, whereas SRG1 can remain in active or in backup state in each node

If you prefer a certain node to take over as the active node on boot, you can do one of the followings:

• Configure the upstream routers to include preferences for the path where the node is located.
• Configure activeness priority.
• Allow the node with higher node ID (in case the above two options not configured) to take the active role.

In a Multinode High Availability setup, both the SRX Series Firewalls initially advertise the route for the floating IP address to the upstream routers. There isn’t a specific preference between the two paths advertised by SRX Series Firewalls. However, the router can have its own preferences on one of the paths depending on the configured metrics.

Figure 12 represents the sequence of events for activeness determination and activeness enforcement.

Figure 12: Activeness Determination and Enforcement

1. On boot, devices enter the hold state and start probing continuously. The devices use the floating IP address (activeness-probing source IP address) as the source IP address and IP addresses of the upstream routers as the destination IP address for the activeness determination probe.

2. The router hosting the probe destination IP address replies to the SRX Series Firewall that is available on its preferred routing path. In the following example, SRX-1 gets the response from the upstream router.

   Figure 13: Activeness Determination and Enforcement

3. SRX-1 promotes itself to the active role since it got the probe reply. SRX-1 communicates its role change to the other device and takes up the active role.

4. After the activeness is determined, the active node (SRX-1):

   • Hosts the floating IP address assigned to it.
   • Advertises the high-preference path to adjacent BGP neighbors.
   • Continues to advertise the active (higher) preference path for all remote and local routes to draw the traffic.
   • Notifies the active node status to the other node through the ICL.

5. The other device (SRX-2) stops probing and takes over the backup role. The backup node advertises the default (lower) priority, ensuring that the upstream routers do not forward any packets to the backup node.

The Multinode High Availability module adds active and backup signal routes for the SRG to the routing table when the node moves to the active role. In case of node failures, the ICL goes down and the current active node releases its active role and removes the active signal route. Now the backup node detects the condition through its probes and transitions to the active role. The route preference is swapped to drive all the traffic towards the new active node.

The switch in the route preference advertisement is part of routing policies configured on SRX Series Firewalls. You must configure the routing policy to include the active signal route with the if-route-exists condition.

• For Default Gateway Deployments
• For Hybrid Deployments
• Activeness Priority and Preemption
• Configuring Activeness Probe Settings

Resiliency and Failover

The Multinode High Availability solution supports redundancy at the service level. Service-level redundancy minimizes the effort needed to synchronize the control plane across the nodes.

After the Multinode High Availability setup determines activeness, it negotiates subsequent high availability (HA) state through the ICL.​ The backup node sends ICMP probes using the floating IP address. If the ICL is up, the node gets the response to its probe and remains as the backup node. If the ICL is down, and there are no probe response, the backup node transitions into the active node.

The SRG1 of the previous backup node now transitions to the active state and continues to operate seamlessly. When the transition happens, the floating IP address is assigned to the active SRG1. In this way, the IP address floats between the active and backup nodes and remains reachable to all the connected hosts. Thus, traffic continues to flow without any disruption.

Services, such as IPsec VPN, that require both control plane and data plane states are synchronized across the nodes. Whenever an active node fails for this service function, both control plane and data plane fail over to the backup node at the same time.

The nodes use the following messages to synchronize data:

• Routing Engine to Routing Engine control application messages
• Routing Engine configuration-related messages
• Data plane RTO messages

Interchassis Link (ICL) Encryption

In Multinode High Availability, the active and backup nodes communicate with each other using an interchassis link (ICL) connected over a routed network or connected directly. The ICL is a logical IP link and it is established using IP addresses that are routable in the network.

Nodes use the ICL to synchronize control plane and data plane states between them. ICL communication could go over a shared or untrusted network and packets sent over the ICL may traverse a path that is not always trusted. Therefore, you must secure the packets traversing the ICL by encrypting the traffic using IPsec standards.

IPsec protects traffic by establishing an encryption tunnel for the ICL. When you apply HA link encryption, the HA traffic flows between the nodes only through the secure, encrypted tunnel. Without HA link encryption, communication between the nodes may not be secure.

To encrypt the HA link for the ICL:

• Install the Junos IKE package on your SRX Series Firewall by using the following command:

  request system software add optional://junos-ike.tgz .

• Configure a VPN profile for the HA traffic and apply the profile for both the nodes. The IPsec tunnel negotiated between the SRX Series Firewalls uses the IKEv2 protocol.

• Ensure you have included the statement ha-link-encryption in your IPsec VPN configuration. Example: user@host# set security ipsec vpn vpn-name ha-link-encryption.

We recommend following for setting up an ICL:

• Use ports and network which is less likely to be saturated
• Not to use the dedicated HA ports (control and fabric ports, if available on your SRX Series Firewall)

• Bind the ICL to the loopback interface (lo0) or an aggregated Ethernet interface (ae0) and have more than one physical link (LAG/LACP) that ensure path diversity for highest resiliency.

• You can use a revenue Ethernet port on the SRX Series Firewalls to setup an ICL connection. Ensure that you separate the transit traffic in revenue interfaces from the high availability (HA) traffic.

See Configuring Multinode High Availability for more details.

PKI-Based Link Encryption for ICL

Starting in Junos OS Release 22.3R1, we support PKI-based link encryption for interchassis link (ICL) in Multinode High Availability. As a part of this support, you can now generate and store node-specific PKI objects such as local keypairs, local certificates, and certificate-signing requests on both nodes. The objects are specific to local nodes and are stored in the specific locations on both nodes.

The node local objects enable you to distinguish between PKI objects that are used for ICL encryption and PKI objects used for IPsec VPN tunnel created between two endpoints.

You can use the following commands run on local node to work with node-specific PKI objects.

On your security device in Multinode High Availability, if you've configured the automatic re-enrollment option and if the ICL goes down at the time of re-enrollment trigger, both the devices start enrolling the same certificate separately with the CA server and download the same CRL file. Once Multinode High Availability re-establishes the ICL, the setup uses only one local certificate. You must synchronize the certificates from the active node to backup node using the user@host> request security pki sync-from-peer command on the backup node.

If you don't synchronize the certificates, the certificate mismatch issue between peer nodes persists till the next re-enrollment.

Optionally you can enable TPM (Trusted Platform module) on both nodes before generating any keypairs on the nodes. See Using Trusted Platform Module to Bind Secrets on SRX Series devices.

ICMP-Based Split-Brain Probing

Consider a scenario where two SRX Series devices are part of Multinode High Availability setup. Lets consider SRX-1 as local node and SRX-2 remote node. The local node is currently in active role and hosting floating IP address to steer traffic towards it. The upstream router has higher priority path for the local node.

When the ICL between the nodes goes down, both nodes initiate a activeness determination probe (ICMP probe). The nodes use the floating IP address (activeness determination IP address) as source IP address and IP addresses of the upstream routers as destination IP address for the probes.

Case 1: If Active Node is Up

Figure 14: Active Node is Up and ICL is Down

• The upstream router, that hosts the probe destination IP address, receives the ICMP probes from both nodes.
• Upstream router replies to only to the active node; because it's configuration has higher preference path for the active node
• The active node retains the active role.

If Active Node is Down:

Figure 15: Active Node is Down and ICL is Down

• The remote node restarts the activeness determination probes.
• The router hosting the probe destination IP address has lost its higher preference path (of former active node) and replies to the remote node.
• The probe result is a success for the remote node and the remote node transitions to the active state.
• As demonstrated in the above cases, activeness determination probes and the configuration of higher path preference in the upstream router ensures one node always stays in the active role and prevents split-brain taking place.

BFD-Based Split-Brain Probing

In Junos OS Release 23.4R1, we support BFD-based split-brain probing for default gateway and hybrid mode of deployments for Multinode High Availability.

Interchassis Link (ICL) failure can often be attributed to two key factors: network disruptions or inconsistent configurations. You can use activeness probe to determine the node that can take active role for each SRG1+. Based on the probe result, one of the node transitions to the active state and this action prevents spilt-brain scenario.

With BFD-based split-brain probing, you can now have more granular control on the probes as you can define the interface, minimal-interval, and multipliers. In the BFD-based split-brain probing, the probing starts immediately after an SRG is configured and starts functioning. In the default ICMP-based split-brain probing, the probing starts only after ICL link goes down.

In comparison, the BFD-based probing is much more proactive in the following ways to ensures a quicker response to prevent split-brain scenarios:

• The probing initiates directly post an SRG configuration.

• If both ICL BFD and split-brain probe break at the same time, the backup node immediately assumes the active role and takes over the VIP.

This ensures a quicker response to prevent split-brain scenarios.

The following table shows differences in ICMP-based probing and BFD-based probing for split-brain detection.

show chassis 
high-availability services-redundancy-group 1 
activeness-probe
dest-ip {
    192.168.21.1; 
    src-ip 192.168.21.2;
    }

show chassis 
high-availability services-redundancy-group 1 
activeness-probe
bfd-liveliness {
    source-ip 192.168.21.1; (inet address of the local SRX sub interface) 
    destination-ip 192.168.21.2; (inet address of the peers SRX sub interface) 
    interface xe-0/0/1.0;     
}

The following figure shows configuration options for ICMP-based probing and BFD-based probing for split-brain detection.

Figure 16: Activeness-Probes Configuration for ICMP-Based and BFD-Based Probing

In hybrid mode and default gateway deployments, you can configure the activeness-probe interval and threshold at following two levels:

• Global-level which is applicable to ICMP-based split-brain probing

• BFD-Liveliness level which is specific to BFD split-brain probe. When you configure BFD-based probing, do not configure global minimum-interval and multiplier options under activeness-probe statement.

To configure activeness probe for default gateway deployments, use the primary virtual IP (VIP1) address interface on both nodes (local and peer) to set up your activeness probe. The destination IP is from the peer node, and the source IP is from your local node. Both VIPs must have the same index value. The IP addresses must be the inet addresses assigned to the LAN interface of the SRX Series Firewall.

You can configure split-brain probing on a Multinode Node High Availability setup in the following ways:

• Routing and Hybrid mode —If you've configured the probe destination IP details for activeness determination (by using the activeness-probe dest-ip statement), then do not configure the multiplier and minimal-interval values. Configure these parameters when you are using VIP-based activeness probing.

• Hybrid and switching mode—Layer 2 split-brain probing using ICMP. Use the probe type ICMP and set interval and timeout threshold using the following statement:

• Hybrid and switching mode—Layer 2 split-brain probing using BFD. Use the probe type BFD and set the timeout threshold that can be sub-second based on the BFD minimum-interval configured.

Figure 17 shows the sample topology. Two SRX Series Firewalls are connected to adjacent routers on trust and untrust side forming a BGP neighborship. An encrypted logical interchassis link (ICL) connects the nodes over a routed network. The nodes communicate with each other using a routable IP address (floating IP address) over the network.

Figure 17: Multinode High Availability Configuration for Split-Brain Probing

Lets consider SRX-1 as a local node and SRX-2 a remote node. The local node is currently in active role and the upstream router has higher priority path for the local node.

For activeness-probe, you must configure the following options:

• Source IP address: Use the virtual IP address 1 (VIP1) of SRG1 of the local node.

• Destination IP address: Use the VIP1 of SRG1 of the peer node.

• Interface: Interface associated with VIP1

In this example, assign a virtual IP (VIP) address (192.168.21.1) and an interface xe-0/0/1.0 for BFD liveliness. Here, you configure BFD-based split-brain probing by specifying source and destination IP addresses and the interface.

The nodes use the family inet address of the interface associated to virtual IP address (VIP1) of SRG1.

Both nodes initiate an activeness determination probe (BFD-based probe) as soon as the SRGs start operating.

The following table shows how Multinode High Availability setup resolves split-brain situation with BFD-based probing when the ICL is down. Depending on node states and probe results, Multinode High Availability system selects the node to take up the active role.

Table 6 shows how Multinode High Availability setup resolves split-brain situation with BFD-based probing when the ICL is down. Depending on node states and probe results, Multinode High Availability system selects the node to take up the active role.

In this example, we assume that SRG1 of node 1 has the higher activeness-priority.

• Sample Configuration
• Verification

• Use the show chassis high-availability services-redundancy-group 1 command to see the type of split-brain probe configured on the device.

  (BFD-based Probing)(ICMP-based Probing)

• Use the show bfd session command to see if BFD-based probe status.

  In the sample, you can notice that BFD-based split-brain probing is running for interface xe-0/0/1.0.

• Use the show chassis high-availability services-redundancy-group 1 command to get the details pf BFD-based probes.