Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

 
ON THIS PAGE
 

Multinode High Availability

SUMMARY Learn about the Multinode High Availability solution and how you can use it in simple and reliable deployment models. Currently, we support two nodes in any Multinode High Availability deployment.

Overview

Business continuity is an important requirement of the modern network. Downtime of even a few seconds might cause disruption and inconvenience apart from affecting the OpEx and CapEx. Modern networks also have data centers spread across multiple geographical areas. In such scenarios, achieving high availability can be very challenging.

Juniper Networks® SRX Series Firewalls support a new solution, Multinode High Availability, to address high availability requirements for modern data centers. In this solution, both the control plane and the data plane of the participating devices (nodes) are active at the same time. Thus, the solution provides interchassis resiliency.

The participating devices could be co-located or physically separated across geographical areas or other locations such as different rooms or buildings. Having nodes with high availability across geographical locations ensures resilient service. If a disaster affects one physical location, Multinode High Availability can fail over to a node in another physical location, thereby ensuring continuity.

Benefits of Multinode High Availability

  • Reduced CapEx and OpEx—Eliminates the need for a switched network surrounding the firewall complex and the need for a direct Layer 2 (L2) connectivity between nodes

  • Network flexibility—Provides greater network flexibility by supporting high availability across Layer 3 (L3) and switched network segments.

  • Stateful resilient solution—Supports active control plane and data plane at the same time on both nodes.

  • Business continuity and disaster recovery—Maximizes availability, increasing redundancy within and across data centers and geographies.

  • Smooth upgrades—Supports different versions of Junos OS on two nodes to ensure smooth upgrades between the Junos OS releases, also allows to run two different version of Junos.

We support two nodes in Multinode High Availability solution.

Active/Backup Multinode High Availability

We support active/backup Multinode High Availability on:

  • SRX5800, SRX5600, SRX5400 with SPC3, IOC3, IOC4, SCB3, SCB4, and RE3 in Junos OS Release 20.4R1

  • SRX4600, SRX4200, SRX4100, and SRX1500 in Junos OS Release 22.3R1

  • SRX2300 and SRX1600 in Junos OS Release 23.4R1.

  • SRX4300 and SRX1600 in Junos OS Release 24.2R1.

  • vSRX3.0 virtual firewalls for the following private and public cloud platforms:

    • KVM (kernel-based virtual machine), VMWare ESXi in Junos OS Release 22.3R1
    • Amazon Web Services (AWS) in Junos OS Release 22.3R1
    • Microsoft Azure Cloud in Junos OS Release 23.4R1

Active/Active Multinode High Availability

Starting in , you can operate Multinode High Availability in active-active mode with support of multiple services redundancy groups (SRGs). Multiple SRG support is available on the following security devices:

  • SRX5400, SRX5600, and SRX5800 with SPC3, IOC3, IOC4, SCB3, SCB4, and RE3 in Junos OS Release 22.4R1.
  • SRX4600, SRX4200, SRX4100, SRX2300, SRX1600, SRX1500, and vSRX in Junos OS Release 23.4R1.
  • SRX4300 in Junos OS Release 24.2R1.

Supported Features

SRX Series Firewalls with Multinode High Availability support the firewall and advanced security services—such as application security, Content Security , intrusion prevention system (IPS), firewall user authentication, NAT, ALG.

For the complete list of features supported with Multinode High Availability, see Feature Explorer.

Multinode High Availability does not support transparent mode high availability (HA)

Deployment Scenarios

Multinode High Availability supports two SRX Series Firewalls presenting themselves as independent nodes to the rest of the network. The nodes are connected to adjacent infrastructure belonging to the same or different networks, all depending on the deployment mode. These nodes can either be collocated or separated across geographies. Participating nodes back up each other to ensure a fast synchronized failover in case of system or hardware failure.

We support the following types of network deployment models for Multinode High Availability:

  • Route mode (all interfaces connected using a Layer 3 topology)
    Figure 1: Layer 3 ModeLayer 3 Mode
  • Default gateway mode (all interfaces connected using an Layer 2 topology) used in more traditional environments. Common deployment of DMZ networks where the firewall devices act as the default gateway for the hosts and applications on the same segment.
    Figure 2: Default Gateway ModeDefault Gateway Mode
  • Hybrid mode (one or more interfaces are connected using a Layer 3 topology and one or more interfaces are connected using a Layer 2 topology)
    Figure 3: Hybrid ModeHybrid Mode
  • Public Cloud Deployment
    Figure 4: Public Cloud Deployment (Example: AWS) Public Cloud Deployment (Example: AWS)

How Is Multinode High Availability Different from Chassis Cluster?

A chassis cluster operates in Layer 2 network environment and requires two links between the nodes (control link and fabric link). These links connect both nodes over dedicated VLANs using back-to-back cabling or over dark fiber. Control links and fabric links use dedicated physical ports on the SRX Series Firewall.

Multinode High Availability uses an encrypted logical interchassis link (ICL). The ICL connects the nodes over a routed path instead of a dedicated Layer 2 network. This routed path can use one or more revenue ports for best resiliency, it’s even possible to dedicate its own routing instance to these ports and paths to ensure total isolation which maximizes the resiliency of the solution.

Figure 5 and Figure 6 show two architectures.

Figure 5: Chassis Cluster Topology in a Layer 2 NetworkChassis Cluster Topology in a Layer 2 Network
Figure 6: Multinode High Availability in a Layer 3 NetworkMultinode High Availability in a Layer 3 Network

Table 1 lists the differences between the two architectures

Table 1: Comparing Chassis Cluster and Multinode High Availability
Parameters Chassis Cluster Multinode High Availability
Network topology Nodes connect to a broadcast domain Nodes connect to a router, a broadcast domain, or a combination of both.
  • Nodes connect to a router
  • Broadcast domain
  • Combination of both above
Network environment Layer 2
  • Layer 3 (Route mode)
  • Layer 2 (defaut gateway mode
  • Combination of Layer 3 and Layer 2 (hybrid mode)
  • Public cloud (AWS) deployments
Traffic switchover approach SRX Series Firewall sends GARP to the switch

Switchover using IP path selection by a peer Layer 3 router or Layer 2 GARP from an SRX Series Firewall to a peer Layer 2 switch

  • Route mode: Switchover using IP path selection (route updates)
  • Hybrid mode: Switchover using IP path selection (route updates), and sending GARP to the switch
  • Default gateway mode: SRX Series Firewall sends GARP to the switch
Public cloud Not supported Supported
Dynamic routing function Routing process active on the SRX Series where the control plane (RG0) is active Routing process active on each SRX Series Firewall participating in Multinode High Availability
Connection between SRX Series Firewalls

  • Control link (Layer 2 path)

  • Fabric link (Layer 2 path)
Interchassis link (Layer 3 path)
Connectivity / Geo-redundance Requires a dedicated Layer 2 stretch between the SRX Series nodes for the control link and fabric link. Uses any routed path between the nodes for the Interchassis link.
IP monitoring to detect network failure
  • Interfaces
  • IP monitoring using IPv4 addresses
  • Interfaces
  • IP monitoring using IPv4 and IPv6 addresses
  • Bidirectional Forwarding Detection (BFD) using IPv4 addresses and IPv6 addresses

Multinode High Availability Glossary

Let's begin by getting familiar with Multinode High Availability terms used in this documentation.

Table 2: Multinode High Availability Glossary
Term Description
active/active state (SRG0) All security services/flows are inspected at each node and backed up on the other node. Security flows must be symmetric.
active/backup state (SRG1+) SRG1+ remains active on one node at any given time and remains in backed up state on the other node. SRG1+ in the backup state is ready to take over traffic from the active SRG1 in case on a failure.
device priority Priority value determines whether a node can act as the active node in a Multinode High Availability setup. The node with a lower numerical value has a higher priority and, therefore, acts as the active node while the other node acts as the backup node.
device preemption Preemptive behavior allows the device with the higher priority (lower numerical value) to resume as active node after it recovers from a failure. If you need to use a specific device in Multinode High Availability as active node, then you must enable the preemptive behavior on both the devices and assign a device priority value for each device.
failover A failover happens when one node detects a failure (hardware/software and so on) and traffic transitions to the other node in a stateful manner. As result, the backup node in a high availability system takes over the task of the active node when the active node fails.
floating IP address or activeness probing IP address An IP address that moves from an active node to the backup node during failover in a Multinode High Availability setup. This mechanism enables clients to communicate with the nodes using a single IP address.

high availability/resiliency

 Ability of a system to eliminate single points of failure to ensure continuous operations over an extended period of time.
interchassis link IP-based link (logical link) that connects nodes over a routed network in a Multinode High Availability deployment. The ICL link is normally bound to the loopback interfaces for most flexible deployments. Connectivity can be any routed or switched path as long as the connectivity is reachable between the two IP addresses.

The security device uses the ICL to synchronize and maintain state information and to handle device failover scenarios.
Interchassis link encryption Link encryption provides data privacy for messages traversing over the network. As the ICL link transmits private data, it is important to encrypt the link. You must encrypt the ICL using IPsec VPN.
monitoring (BFD) Monitoring of one or more links using Bidirectional Forwarding Detection (BFD). BFD monitoring triggers a routing path change or a system failover, depending on system configuration.
monitoring (IP) Monitoring of a reliable IP address and system state in case of loss of communication with the peer node.
monitoring (path) Method that uses ICMP to verify the reachability of the IP address. The default interval for ICMP ping probes is 1 second.
monitoring (system) Monitoring of key hardware and software resources and infrastructures by triggering failover when a failure is detected on a node.
probing Mechanism used to exchange messages between active and backup nodes in the high availability setup. The messages determine the status and health of the application on each individual node.
real-time object (RTO) Special payload packet that contains the necessary information to synchronize the data from one node to the other node.
split-brain detection (also known as control plane detection or activeness conflict detection) Event where the ICL between two Multinode High Availability nodes is down, and both nodes initiate an activeness determination probe (split-brain probe). Based on the response to the probe, subsequent failover to a new role is triggered
services redundancy group (SRG) Failover unit that includes and manages a collection of objects on the participating nodes. The SRG on one node switches over to the other node when a failover is detected.
SRG0 Manages all control plane stateless services such as firewall, NAT, and ALG. SRG0 is active on all participating nodes and handles symmetric security flows.
SRG1+ Manages control plane stateful service (IPsec VPN or virtual IPs in hybrid or default gateway mode.).
synchronization Process where controls and data plane states are synchronized across the nodes.
virtual IP (VIP) address Virtual IP addresses in hybrid or default gateway mode are used for activeness determination and enforcement on the switching side in a Multinode High Availability setup. The virtual IP is controlled by the SRG1+.
virtual MAC (VMAC) address (For hybrid and default gateway deployments). Virtual MAC address dynamically assigned to the interface on active node that faces the switching side.

Now we are that familiar with Multinode High Availability features and terminology, let's proceed to understand how Multinode High Availability works.

How Multinode High Availability Works

Note:

We support a two-node configuration for the Multinode High Availability solution.

In a Multinode High Availability setup, you connect two SRX Series Firewalls to adjacent upstream and downstream routers (for Layer 3 deployments), routers and switches (hybrid deployment), or switches (default gateway deployment) using the revenue interfaces.

The nodes communicate with each other using an interchassis link (ICL). The ICL link uses Layer 3 connectivity to communicate with each other, This communication can take place over a routed network (Layer 3), or a directly connected Layer 2 path. It is recommended to bind the ICL to the loopback interface and have more than one physical link (LAG/LACP) to ensure path diversity for the highest resiliency.

Multinode High Availability operates in active/active mode for data plane and active/backup mode for control plane services. The active SRX Series Firewall hosts the floating IP address and steers traffic towards it using the floating IP address

Multinode High Availability operates in:

  • Active/active mode (SRG0) for the security services
  • Active/backup mode (SRG1 and above) for security and system services

Floating IP addresses controlled by SRG1 or above moves between the nodes . Active SRG1+ hosts and controls the floating IP address. In failover scenarios, this IP address 'floats' to another active SRG1 based on configuration, system health, or path monitoring decisions. The the newly active SRG1+ can take on the function of a now-standby SRG1 and starts responding to incoming requests.

Figure 7, Figure 8, and Figure 9 show deployments in Layer 3, hybrid, and default gateway modes.

Figure 7: Layer 3 DeploymentLayer 3 Deployment

In this topology, two SRX Series Firewalls are part of a Multinode High Availability setup. The setup has Layer 3 connectivity between SRX Series Firewalls and neighboring routers. The devices are running on separate physical Layer 3 networks and are operating as two independent nodes. The nodes shown in the illustration are co-located in the topology. The nodes can also be geographically separated.

Figure 8: Default Gateway DeploymentDefault Gateway Deployment

In a typical default gateway deployment, hosts and servers in a LAN are configured with a default gateway of the security device. So the security device must host a virtual IP (VIP) address that moves between nodes based on the activeness. The configuration on hosts remains static, and security device failover is seamless from the hosts' perspective.

You must create static routes or dynamic routing on SRX Series Firewalls to reach other networks not directly connected.

Figure 9: Hybrid DeploymentHybrid Deployment

In hybrid mode, an SRX Series Firewall uses a VIP address on the Layer 2 side to draw traffic toward it. You can optionally configure the static ARP for the VIP using the VMAC address to ensure no change in the IP address during the failover

Let's now understand the components and functionality of Multinode High Availability in detail.

Services Redundancy Groups

A services redundancy group (SRG) is a failover unit in a Multinode High Availability setup. There are two types of SRGs:

  • SRG0—Manages security service from Layer 4-Layer 7 except IPsec VPN services. The SRG0 operates in active mode on both nodes at any point in time. On SRG0, each security session must traverse the node in a symmetric flow, Backup of these flows are fully state-synchronized to the other node,
  • SRG1+—Manages IPsec services and virtual IPs for hybrid and default gateway mode and are backed up to the other node. The SRG1 operates in active mode on one node and in backup node on another node.

Figure 10 shows SRG0 and SRG1 in a Multinode High Availability setup.

Figure 10: Single SRG Support in Multinode High Availability (Active-Backup Mode)Single SRG Support in Multinode High Availability (Active-Backup Mode)

Figure 11 shows SRG0 and SRG1+ in a Multinode High Availability setup.

Figure 11: Multi SRG Support in Multinode High Availability (Active-Active Mode)Multi SRG Support in Multinode High Availability (Active-Active Mode)

Starting in Junos OS Release 22.4R1, you can configure Multinode High Availability to operate in active-active mode with support of multi SRG1s (SRG1+). In this mode, some SRGs remain active on one node and some SRGs remain active on another node. A particular SRG always operates in active-backup mode; it operates in active mode on one node and backup mode on another node. In this case, both the nodes can have the active SRG1 forwarding stateful services. Each node has a different set of floating IP addresses assigned to SRG1+.

Note:

Starting in Junos OS Release 22.4R1, you can configure upto 20 SRGs in a Multinode Highavailability setup.

Table 3 explains the behavior of SRGs in a Multinode High Availability setup.

Table 3: Services Redundancy Group Details in Multinode High Availability
Related Services Redundancy Group (SRG) Managed Services Operates in Synchronization Type When Active Node Fails Configuration Options
SRG0 Manages security service L4-L7 except IPsec VPN. Active/active mode Stateful synchronization of security services Traffic processed on the failed node will transition to the healthy node in a stateful manner.
  • Shutdown on failure option
  • Install on failure route options
SRG1+ Manages IPsec and virtual-IP addresses with associated security services Active/backup mode Stateful synchronization of security services Traffic processed on the failed node will transition to the healthy node in a stateful manner.
  • Active/backup signal route
  • Deployment type
  • Activeness priority and preemption
  • Virtual IP address (for default gateway deployments)
  • Activeness probing
  • Process packet on backup option

  • BFD monitoring

  • IP monitoring
  • Interface monitoring
Note:

When you configure monitoring (BFD or IP or Interface) options on SRG1+, we recommend not to configure the shutdown-on-failure option on SRG0.

Starting in Junos OS Release 23.4R1, the Multinode High Availability setup operates in a combined mode. You don’t have to reboot the system when you add or delete any SRG (SRG0 or SRG1+) configurations.

Activeness Determination and Enforcement

In a Multinode High Availability setup, activeness is determined at the service level, not at the node level. The active/backup state is at the SRG level and the traffic is steered toward the active SRG. SRG0 remains active on both the nodes, whereas SRG1 can remain in active or in backup state in each node

If you prefer a certain node to take over as the active node on boot, you can do one of the followings:

  • Configure the upstream routers to include preferences for the path where the node is located.
  • Configure activeness priority.
  • Allow the node with higher node ID (in case the above two options not configured) to take the active role.

In a Multinode High Availability setup, both the SRX Series Firewalls initially advertise the route for the floating IP address to the upstream routers. There isn’t a specific preference between the two paths advertised by SRX Series Firewalls. However, the router can have its own preferences on one of the paths depending on the configured metrics.

Figure 12 represents the sequence of events for activeness determination and activeness enforcement.

Figure 12: Activeness Determination and EnforcementActiveness Determination and Enforcement
  1. On boot, devices enter the hold state and start probing continuously. The devices use the floating IP address (activeness-probing source IP address) as the source IP address and IP addresses of the upstream routers as the destination IP address for the activeness determination probe.

  2. The router hosting the probe destination IP address replies to the SRX Series Firewall that is available on its preferred routing path. In the following example, SRX-1 gets the response from the upstream router.

    Figure 13: Activeness Determination and EnforcementActiveness Determination and Enforcement

  3. SRX-1 promotes itself to the active role since it got the probe reply. SRX-1 communicates its role change to the other device and takes up the active role.

  4. After the activeness is determined, the active node (SRX-1):

    • Hosts the floating IP address assigned to it.
    • Advertises the high-preference path to adjacent BGP neighbors.
    • Continues to advertise the active (higher) preference path for all remote and local routes to draw the traffic.
    • Notifies the active node status to the other node through the ICL.

  5. The other device (SRX-2) stops probing and takes over the backup role. The backup node advertises the default (lower) priority, ensuring that the upstream routers do not forward any packets to the backup node.

The Multinode High Availability module adds active and backup signal routes for the SRG to the routing table when the node moves to the active role. In case of node failures, the ICL goes down and the current active node releases its active role and removes the active signal route. Now the backup node detects the condition through its probes and transitions to the active role. The route preference is swapped to drive all the traffic towards the new active node.

The switch in the route preference advertisement is part of routing policies configured on SRX Series Firewalls. You must configure the routing policy to include the active signal route with the if-route-exists condition.

For Default Gateway Deployments

If both the nodes are booting up at the same time, then the Multinode High Availability system uses the configured priority value of an SRG to determine activeness. Activeness enforcement takes place when the node with an active SRG1+ owns the virtual IP (VIP) address and the virtual MAC (VMAC) address. This action triggers Gratuitous ARP (GARP) toward the switches on both sides and results in updating the MAC tables on the switches.

For Hybrid Deployments

Activeness enforcement takes place on the Layer 3 side, when the configured signal route enforce activeness with the corresponding route advertisements. On the Layer 2 side, the SRX Series Firewall triggers a Gratuitous ARP (GARP) to the switch layer and owns the VIP and VMAC addresses

When the failover happens and the old backup node transitions to the active role, the route preference is swapped to drive all the traffic to the new active node.

Activeness Priority and Preemption

Configure the preemption priority (1-254) for SRG1+. You must configure the preemption value on both nodes. The preempt option ensures that the traffic always falls back to the specified node, when the node recovers from a failover.

You can configure activeness priority and preemption for an SRG1+ as in the following sample:

See Configuring Multinode High Availability In a Layer 3 Network for the complete configuration example.

As long as the nodes can communicate with each other through the ICL, the activeness priority is honored.

Configuring Activeness Probe Settings

Starting in Junos OS 22.4R1, default gateway (switching) and in hybrid deployments of Multinode High Availability, you can optionally configure activeness probe parameters using the following statements:

The probe interval sets the time period between the probes sent to the destination IP addresses. You can set the probe interval as 1000 milliseconds.

The multiplier value determines the time period, after which the backup node transitions to active state, if the backup node fails to receive response to the activeness-probes from the peer node.

The default is 2, and the minimum value is 2, and the maximum is 15.

Example: If you configure the multiplier value to two, the backup node will transition to the active state if it does not receive a response to activeness probing request from the peer node after two seconds.

You can configure multiplier and minimal-interval in switching and hybrid deployments.

In hybrid mode deployments, if you've configured the probe destination IP details for activeness determination (by using the activeness-probe dest-ip statement), then do not configure the multiplier and minimal-interval values. Configure these parameters when you are using VIP-based activeness probing.

Resiliency and Failover

The Multinode High Availability solution supports redundancy at the service level. Service-level redundancy minimizes the effort needed to synchronize the control plane across the nodes.

After the Multinode High Availability setup determines activeness, it negotiates subsequent high availability (HA) state through the ICL.​ The backup node sends ICMP probes using the floating IP address. If the ICL is up, the node gets the response to its probe and remains as the backup node. If the ICL is down, and there are no probe response, the backup node transitions into the active node.

The SRG1 of the previous backup node now transitions to the active state and continues to operate seamlessly. When the transition happens, the floating IP address is assigned to the active SRG1. In this way, the IP address floats between the active and backup nodes and remains reachable to all the connected hosts. Thus, traffic continues to flow without any disruption.

Services, such as IPsec VPN, that require both control plane and data plane states are synchronized across the nodes. Whenever an active node fails for this service function, both control plane and data plane fail over to the backup node at the same time.

The nodes use the following messages to synchronize data:

  • Routing Engine to Routing Engine control application messages
  • Routing Engine configuration-related messages
  • Data plane RTO messages

Interchassis Link (ICL) Encryption

In Multinode High Availability, the active and backup nodes communicate with each other using an interchassis link (ICL) connected over a routed network or connected directly. The ICL is a logical IP link and it is established using IP addresses that are routable in the network.

Nodes use the ICL to synchronize control plane and data plane states between them. ICL communication could go over a shared or untrusted network and packets sent over the ICL may traverse a path that is not always trusted. Therefore, you must secure the packets traversing the ICL by encrypting the traffic using IPsec standards.

IPsec protects traffic by establishing an encryption tunnel for the ICL. When you apply HA link encryption, the HA traffic flows between the nodes only through the secure, encrypted tunnel. Without HA link encryption, communication between the nodes may not be secure.

To encrypt the HA link for the ICL:

  • Install the Junos IKE package on your SRX Series Firewall by using the following command:

    request system software add optional://junos-ike.tgz .

  • Configure a VPN profile for the HA traffic and apply the profile for both the nodes. The IPsec tunnel negotiated between the SRX Series Firewalls uses the IKEv2 protocol.

  • Ensure you have included the statement ha-link-encryption in your IPsec VPN configuration. Example: user@host# set security ipsec vpn vpn-name ha-link-encryption.

We recommend following for setting up an ICL:

  • Use ports and network which is less likely to be saturated
  • Not to use the dedicated HA ports (control and fabric ports, if available on your SRX Series Firewall)

  • Bind the ICL to the loopback interface (lo0) or an aggregated Ethernet interface (ae0) and have more than one physical link (LAG/LACP) that ensure path diversity for highest resiliency.

  • You can use a revenue Ethernet port on the SRX Series Firewalls to setup an ICL connection. Ensure that you separate the transit traffic in revenue interfaces from the high availability (HA) traffic.

See Configuring Multinode High Availability for more details.

PKI-Based Link Encryption for ICL

Starting in Junos OS Release 22.3R1, we support PKI-based link encryption for interchassis link (ICL) in Multinode High Availability. As a part of this support, you can now generate and store node-specific PKI objects such as local keypairs, local certificates, and certificate-signing requests on both nodes. The objects are specific to local nodes and are stored in the specific locations on both nodes.

The node local objects enable you to distinguish between PKI objects that are used for ICL encryption and PKI objects used for IPsec VPN tunnel created between two endpoints.

You can use the following commands run on local node to work with node-specific PKI objects.

Table 4: Commands for Node-Specific PKI Objects
Generating a private/public key pair for a local node

request security pki node-local generate-key-pair

Generating and enrolling a local digital certificate in a local node
Clear node-specific certificates
Display node-specific local certificates and certificate requests.

On your security device in Multinode High Availability, if you've configured the automatic re-enrollment option and if the ICL goes down at the time of re-enrollment trigger, both the devices start enrolling the same certificate separately with the CA server and download the same CRL file. Once Multinode High Availability re-establishes the ICL, the setup uses only one local certificate. You must synchronize the certificates from the active node to backup node using the user@host> request security pki sync-from-peer command on the backup node.

If you don't synchronize the certificates, the certificate mismatch issue between peer nodes persists till the next re-enrollment.

Optionally you can enable TPM (Trusted Platform module) on both nodes before generating any keypairs on the nodes. See Using Trusted Platform Module to Bind Secrets on SRX Series devices.

Split-Brain Detection and Prevention

Split-brain detection or activeness conflict happens when the ICL between two Multinode High Availability nodes is down and both node cannot reach each other to gather the status of peer node anymore.

ICMP-Based Split-Brain Probing

Consider a scenario where two SRX Series devices are part of Multinode High Availability setup. Lets consider SRX-1 as local node and SRX-2 remote node. The local node is currently in active role and hosting floating IP address to steer traffic towards it. The upstream router has higher priority path for the local node.

When the ICL between the nodes goes down, both nodes initiate a activeness determination probe (ICMP probe). The nodes use the floating IP address (activeness determination IP address) as source IP address and IP addresses of the upstream routers as destination IP address for the probes.

Case 1: If Active Node is Up

Figure 14: Active Node is Up and ICL is Down Active Node is Up and ICL is Down
  • The upstream router, that hosts the probe destination IP address, receives the ICMP probes from both nodes.
  • Upstream router replies to only to the active node; because it's configuration has higher preference path for the active node
  • The active node retains the active role.

If Active Node is Down:

Figure 15: Active Node is Down and ICL is Down Active Node is Down and ICL is Down
  • The remote node restarts the activeness determination probes.
  • The router hosting the probe destination IP address has lost its higher preference path (of former active node) and replies to the remote node.
  • The probe result is a success for the remote node and the remote node transitions to the active state.
  • As demonstrated in the above cases, activeness determination probes and the configuration of higher path preference in the upstream router ensures one node always stays in the active role and prevents split-brain taking place.

BFD-Based Split-Brain Probing

In Junos OS Release 23.4R1, we support BFD-based split-brain probing for default gateway and hybrid mode of deployments for Multinode High Availability.

Interchassis Link (ICL) failure can often be attributed to two key factors: network disruptions or inconsistent configurations. You can use activeness probe to determine the node that can take active role for each SRG1+. Based on the probe result, one of the node transitions to the active state and this action prevents spilt-brain scenario.

With BFD-based split-brain probing, you can now have more granular control on the probes as you can define the interface, minimal-interval, and multipliers. In the BFD-based split-brain probing, the probing starts immediately after an SRG is configured and starts functioning. In the default ICMP-based split-brain probing, the probing starts only after ICL link goes down.

In comparison, the BFD-based probing is much more proactive in the following ways to ensures a quicker response to prevent split-brain scenarios:

  • The probing initiates directly post an SRG configuration.

  • If both ICL BFD and split-brain probe break at the same time, the backup node immediately assumes the active role and takes over the VIP.

The following table shows differences in ICMP-based probing and BFD-based probing for split-brain detection.

Table 5: Difference in ICMP-Based and BFD-Based Probing

Parameters

ICMP-Based Probing

BFD-Based Probing
Probe type ICMP packet

BFD packet, single-hop BFD
Minimal interval 1000 ms SRX Series Firewall's minimal BFD interval depends on platform.

For example:

SRX5000-line Firewalls with SPC3, interval is 100 ms .

SRX4200, the interval is 300ms.
SRG backup node probes Yes Yes
SRG active node probes No Yes
SRG split-brain resolving when ICL down Only when ICL goes down. After SRG is configured.
  Not possible Possible

Configuration options

 
show chassis 
high-availability services-redundancy-group 1 
activeness-probe
dest-ip {
    192.168.21.1; 
    src-ip 192.168.21.2;
    }
 
show chassis 
high-availability services-redundancy-group 1 
activeness-probe
bfd-liveliness {
    source-ip 192.168.21.1; (inet address of the local SRX sub interface) 
    destination-ip 192.168.21.2; (inet address of the peers SRX sub interface) 
    interface xe-0/0/1.0;     
}

The following figure shows configuration options for ICMP-based probing and BFD-based probing for split-brain detection.

Figure 16: Activeness-Probes Configuration for ICMP-Based and BFD-Based Probing Activeness-Probes Configuration for ICMP-Based and BFD-Based Probing
Note:

ICMP-based probing and BFD-based probes are mutually exclusive.

In hybrid mode and default gateway deployments, you can configure the activeness-probe interval and threshold at following two levels:

  • Global-level which is applicable to ICMP-based split-brain probing

  • BFD-Liveliness level which is specific to BFD split-brain probe. When you configure BFD-based probing, do not configure global minimum-interval and multiplier options under activeness-probe statement.

To configure activeness probe for default gateway deployments, use the primary virtual IP (VIP1) address interface on both nodes (local and peer) to set up your activeness probe. The destination IP is from the peer node, and the source IP is from your local node. Both VIPs must have the same index value. The IP addresses must be the inet addresses assigned to the LAN interface of the SRX Series Firewall.

You can configure split-brain probing on a Multinode Node High Availability setup in the following ways:

  • Routing and Hybrid mode —If you've configured the probe destination IP details for activeness determination (by using the activeness-probe dest-ip statement), then do not configure the multiplier and minimal-interval values. Configure these parameters when you are using VIP-based activeness probing.

  • Hybrid and switching mode—Layer 2 split-brain probing using ICMP. Use the probe type ICMP and set interval and timeout threshold using the following statement:

  • Hybrid and switching mode—Layer 2 split-brain probing using BFD. Use the probe type BFD and set the timeout threshold that can be sub-second based on the BFD minimum-interval configured.

Figure 17 shows the sample topology. Two SRX Series Firewalls are connected to adjacent routers on trust and untrust side forming a BGP neighborship. An encrypted logical interchassis link (ICL) connects the nodes over a routed network. The nodes communicate with each other using a routable IP address (floating IP address) over the network.

Figure 17: Multinode High Availability Configuration for Split-Brain Probing Multinode High Availability Configuration for Split-Brain Probing

Lets consider SRX-1 as a local node and SRX-2 a remote node. The local node is currently in active role and the upstream router has higher priority path for the local node.

For activeness-probe, you must configure the following options:

  • Source IP address: Use the virtual IP address 1 (VIP1) of SRG1 of the local node.

  • Destination IP address: Use the VIP1 of SRG1 of the peer node.

  • Interface: Interface associated with VIP1.

In this example, assign a virtual IP (VIP) address (192.168.21.1) and an interface xe-0/0/1.0 for BFD liveliness. Here, you configure BFD-based split-brain probing by specifying source and destination IP addresses and the interface.

The nodes use the family inet address of the interface associated to virtual IP address (VIP1) of SRG1.

Both nodes initiate an activeness determination probe (BFD-based probe) as soon as the SRGs start operating.

Note:

For BFD-based split-brain probing, you must:

  • Configure matching source and destination IP addresses for the same SRG on both nodes.
  • Configure the activeness-priority option to determine active node as a result of split-brain probing.

The following table shows how Multinode High Availability setup resolves split-brain situation with BFD-based probing when the ICL is down. Depending on node states and probe results, Multinode High Availability system selects the node to take up the active role.

Table 6 shows how Multinode High Availability setup resolves split-brain situation with BFD-based probing when the ICL is down. Depending on node states and probe results, Multinode High Availability system selects the node to take up the active role.

In this example, we assume that SRG1 of node 1 has the higher activeness-priority.

Table 6: Activeness Determination Using Split-Brain Probes and Node States
State of Node 1 Probing State of Node 1 State of Node 2 Probing State of Node 2 Node Transitioning to SRG1 Active State
Active Down Ineligible No probing Node 1
Active Up Backup Up Node 1
Active Up Active Up Node 1 (Tie breaker)
Backup Down Ineligible No probing Node 1
Backup Up Backup Up Node 1 (Tie breaker)
Backup Up Active Up Node 2
Ineligible No probing Ineligible No probing Neither Node
Ineligible No probing Backup Down Node 2
Ineligible No probing Active Down Node 2

Sample Configuration

Node 1:

Node 2:

Verification

  • Use the show chassis high-availability services-redundancy-group 1 command to see the type of split-brain probe configured on the device.

    (BFD-based Probing)(ICMP-based Probing)

  • Use the show bfd session command to see if BFD-based probe status.

    In the sample, you can notice that BFD-based split-brain probing is running for interface xe-0/0/1.0.

  • Use the show chassis high-availability services-redundancy-group 1 command to get the details pf BFD-based probes.