Optimized Intersubnet Multicast (OISM) with Assisted Replication (AR) for Edge-Routed Bridging Overlays

For example:

All SL and BL Devices:

S-ARR Devices:

See Figure 2 and Figure 3 for the device IP addresses, AS numbers, and the AE interface names and IP addresses in this example.

See Figure 2 and Figure 3 for the device IP addresses, AS numbers, and the AE interface names and IP addresses in this example.

For example:

SL-1 (device AS: 4200000011):

On each SL and BL device, set the spine devices S-ARR-1 (lo0:192.168.2.1) and S-ARR-2 (192.168.2.2) in the overlay AS as BGP neighbors. For example:

SL-1 (device lo0: 192.168.0.1):

Repeat this step for each SL and BL device, substituting that device loopback IP address for the local-address option. See Figure 2 and Figure 3.

On each S-ARR device, set all of the SL and BL devices in the overlay AS as BGP neighbors. For example:

S-ARR-1 (device lo0: 192.168.2.1):

Set these values on all SL and BL devices.

Set the NDP stale timer on all IRB interfaces for IPv6 neighbor reachability confirmation on all SL and BL devices for IPv6 unicast traffic.

Note:

You apply this setting to all IRB interfaces that might handle IPv6 unicast traffic, so you usually add this statement to a configuration group of common statements and apply the group to all interfaces.

For example:

SL-1:

Repeat this step on the remaining SL devices, BL-1, BL-2, S-ARR-1, and S-ARR-2. In the configuration on each device, substitute that device’s IP address as part of the route-distinguisher statement value so these values are unique across the devices.

1. On Junos OS devices in the QFX5000 line of switches , include the global shared tunnels configuration statement. This setting supports VXLAN tunneling with MAC-VRF instances on those devices in scaled environments:

   Note:

   If you need to configure the shared tunnels setting here, after you commit the configuration, you must reboot the device for the shared tunnels setting to take affect.

   The shared tunnels feature is set by default on devices that run Junos OS Evolved.

2. On QFX5130 and QFX5700 switches, configure the host-profile unified forwarding profile option to support an EVPN-VXLAN environment (see Layer 2 Forwarding Tables for details):

3. On QFX5110 and QFX5120 switches, configure next hop settings for scaled environments to ensure next hops table size is proportional to the device capacity, as follows:

   QFX5110 switches:

   QFX5120 switches:

Configure these elements on all SL, BL, and S-ARR devices.

Note:

The S-ARR spine devices in this example serve as standalone AR replicator devices, so you must enable OISM on them too. If the spine devices don’t run as AR replicators, you don’t need to enable OISM on those devices.

Note:

You enable IGMP snooping on the S-ARR devices because they act as AR replicator devices. AR replicators use IGMP snooping to optimize traffic forwarding.

In EVPN-VXLAN fabrics, we support IGMPv2 traffic with ASM reports only. We support IGMPv3 traffic with SSM reports only. When you enable IGMP snooping for IGMPv3 traffic, include the SSM-specific option evpn-ssm-reports-only configuration option as shown below. See Supported IGMP or MLD Versions and Group Membership Report Modes for more on ASM and SSM support with EVPN-VXLAN.

IGMP snooping with IGMPv2 on all VLANs:

IGMP snooping on VLANs with IGMPv3 receivers (VLAN-1 through VLAN-4, according to Table 1):

IGMPv2 is enabled by default when you configure PIM, but you need to explicitly enable IGMPv3 on the IRB interfaces that handle IGMPv3 traffic. According to Table 1, the IGMPv3 IRB interfaces are irb.1, irb.2, irb.3, and irb.4, which are associated with VRF-1:

For example, in this test environment, S-ARR-1 and S-ARR-2 are AR replicators, so we configure this statement in the MAC-VRF routing instance on those devices. Also, SL-4 and SL-5 are switches in the QFX10000 line switches, so if those devices are running earlier releases than Junos OS or Junos OS Evolved Release 23.4R1, you would also configure this statement on those devices.

Include the IRB interfaces associated with each tenant VRF instance. Set a route distinguisher based on the device IP address, and a route target for the instance.

For example:

SL-1:

Repeat this step on the remaining SL devices, BL-1, BL-2, S-ARR-1, and S-ARR-2. On each device, configure the route-distinguisher to be unique across the devices and tenant VRFs. Substitute the device IP address from Figure 2 or Figure 3, and the VRF instance number for each tenant VRF, as follows:

• On SL-2, use route-distinguisher 192.168.0.2:1 for VRF-1, and 192.168.0.2:101 for VRF-101.

  On SL-3, use route-distinguisher 192.168.0.3:1 for VRF-1, and 192.168.0.3:101 for VRF-101.

  And so on for the remaining SL devices.

• On BL-1, use route-distinguisher 192.168.5.1:1 for VRF-1, and 192.168.5.1:101 for VRF-101.

  On BL-2, use route-distinguisher 192.168.5.2:1 for VRF-1, and 192.168.5.2:101 for VRF-101.

• On S-ARR-1, use route-distinguisher 192.168.2.1:1 for VRF-1, and 192.168.2.1:101 for VRF-101.

  On S-ARR-2, use route-distinguisher 192.168.2.2:1 for VRF-1, and 192.168.2.2:101 for VRF-101.

Repeat the same configuration all SL, BL, and S-ARR devices.

Repeat the same configuration all SL, BL, and S-ARR devices.

On multihoming peer SL devices, configure the same ESI on the AE interface links to the multihomed TOR device. Also enable LACP and LACP hold timers on the interfaces. See Multihoming an Ethernet-Connected End System Design and Implementation for more details on configuring these interfaces and the corresponding Ethernet segment identifiers (ESIs).

Note:

See Step 4 in this section to add configuring the device to detect and take action upon network isolation events on these interfaces. If you include that configuration, make sure the "up" hold timer configured here in this step is greater than the "up" hold timer in the network isolation group profile in that step.

For example:

On SL-1 for the multihoming interfaces to TOR-1:

Repeat this configuration on SL-2, the multihoming peer to SL-1. Use the corresponding interface for the link on SL-2 to TOR-1, and the same ESI.

On SL-2 for the multihoming interfaces to TOR-1:

Use a different ESI for each corresponding multihoming peer SL device pair. See Figure 3. In this example, all SL devices use ae3 to link to the TOR devices. SL-4 and SL-5 use ae3 to link to TOR-3, and additionally use ae5 to link to TOR-4. As a result, you set one ESI for the ae3 links and a different ESI for the ae5 links on both of those SL devices.

Here we assign ESI values with the following conventions:

• The 6th segment from the right in the ESI value matches the number of the multihomed TOR.

• The last segment of the ESI value matches the AE interface number.

For example:

• On SL-4 and SL-5 for the interfaces to multihomed TOR-3 (link is ae3 on both SL devices):

• On SL-4 and SL-5 for the interfaces to multihomed TOR-4 (link is ae5 on both SL devices):

TOR-2 is single-homed to SL-3, so you don’t configure an ESI on the ae3 interface on SL-3.

Similarly, TOR-5 is single-homed to SL-6, so you don’t configure an ESI on the ae3 interface on SL-6.

Note:

The test environment also includes separate AE interfaces (ae4 on each SL device) and ESIs (for the multihomed TOR devices) that the fabric uses for unicast traffic. We only show configuring the multicast ESIs here.

For example:

On SL-1 for the interface to TOR-1 (and repeat this configuration on SL-2 for the interface to TOR-1, SL-3 for the interface to TOR-2, and SL-6 for the interface to TOR-5):

On SL-4 and SL-5 for the interfaces to TOR-3 and TOR-4:

• Set up a network isolation group net-isolation-grp-1 with core isolation service tracking using the network-isolation statement.

• Include an "up" hold time (in ms) that the device will wait upon detecting recovery from network isolation before acting to bring the interface up again.

  Note:

  The "up" hold time you configure here for core isolation event detection should be less than the general "up" hold timer on the interfaces for the ESI in Step 1. For example, in that step, the interface "up" hold timer is 300000 ms; here, we set the "up" hold timer for the network isolation group to 100000 ms.

• Configure the device to set either lacp-out-of-sync or link-down status on the interface as the core isolation action. For example, here we set the lacp-out-of-sync action.

• Apply the network isolation group to the interface using the network-isolation-profile statement.

• Because core isolation service tracking improves the convergence time for the ESI routes, we don't need to include a delay in the overlay BGP route advertisements. As a result, on devices where we configure the network isolation feature on TOR-facing interfaces, we remove the delay-route-advertisements settings from Step 4 in Configure the Underlay (with EBGP) and the Overlay (with IBGP).

For example:

On SL-1 for the multihoming interface(s) to TOR-1:

Repeat this configuration on SL-2 for ae3 with the same network isolation group profile parameters.

You can set up the same or similar network isolation group profiles on the other multihoming peer SL devices, SL-4 and SL-5, which use ae3 for the ESI to TOR-3 and ae5 for the ESI to TOR-4.

Also, when we enable the network isolation feature on the SL device interfaces, with that configuration we remove the overlay route advertisement delay setting on the device. We still need to retain the overlay delay-route-advertisements minimum-delay routing-uptime setting on S-ARR-1 and S-ARR-2 from Step 4 in Configure the Underlay (with EBGP) and the Overlay (with IBGP). However, with the network isolation feature configured in the network, we observed better results testing with a higher setting on the S-ARR devices than the setting in that step. As a result, in this case we recommend you also change the following:

On S-ARR-1 and S-ARR-2 from Step 4 in Configure the Underlay (with EBGP) and the Overlay (with IBGP):

change that configuration to:

In this example, both BL devices use interface ae3 for this purpose with a different subnet for each BL device connection per tenant VRF. The physical interfaces in the AE interface bundle might differ across the BL devices. Figure 2 shows the BL device and link information for this example.

The external L3 multicast connection uses the ae3 interface with VLAN tagging enabled. In the scaled environment, we configure logical units on the ae3 interface and corresponding VLANs per tenant VRF starting with unit 0 and VLAN-3001 for VRF-1. In this example we show configuring ae3.0 in VRF-1 and ae3.100 in VRF-101 on both BL devices. See Table 2 for a summary of the external L3 interface connection configuration parameters.

BL-1:

BL-2:

BL-1 and BL-2:

BL-1 and BL-2:

• In this environment, set a static PIM RP address corresponding to the external PIM router and RP router (one for each logical unit and associated tenant VRF).

• Configure PIM on the OISM revenue VLAN IRB interfaces and include the PIM distributed-dr option.

• Configure classic PIM (no distributed-dr option) on the SBD IRB interface, the logical unit L3 interface and logical loopback interface.

  With PIM on the SBD IRB interface, include the accept-remote-source option to enable the BL devices to accept multicast traffic from the SBD IRB interface as the source interface. This option handles situations where the BL devices might send source traffic to each other on the SBD. See Multicast Traffic from an External Source to Receivers Inside the EVPN Data Center—L3 Interface Method or Non-EVPN IRB Method in the EVPN User Guide for more information on those situations.

• With PIM on the SBD IRB interface, enable Bidirectional Forwarding Detection (BFD) and the stickydr option. The BFD settings improve convergence time with interface issues to help avoid traffic loss. The stickydr option eliminates designated router switchover convergence delay during reboot events.

Include the same configuration on BL-1 and BL-2.

For VRF-1:

For VRF-101:

See OISM Components in the EVPN User Guide for details on why we configure these PIM options on the BL devices.

You also define and include the export-direct export policy, which exports addresses of the directly-connected IRB interfaces on a particular BL device into the OSPF routing protocol.

The configuration is the same on both BL devices.

BL-1 and BL-2:

Include the L3 interfaces to BL-1 and BL-2 in the routing instances, as well as the device logical loopback interface and logical interfaces that connect to the external source for each routing instance.

See Table 2 for the PIM RP static addresses we use in this example.

1. Configure the device loopback interface lo0 with a secondary IP address specifically for AR functions. The AR replicator advertises this IP address to the network in EVPN Type 3 AR tunnel routes.

   We include the primary loopback address configuration statements here as well so you can more easily identify each S-ARR device.

   S-ARR-1:

   S-ARR-2:

2. Configure the AR replicator role on S-ARR-1 and S-ARR-2 in the MAC-VRF instance, using the secondary AR loopback interface you configured in the previous step.

   S-ARR-1:

   S-ARR-2:

3. On AR replicator devices in standalone mode, you must also configure the common OISM elements that you configure on the OISM SL and BL devices. You configure those elements in earlier steps in this example. See:

   • Configure an OISM-Enabled EVPN MAC-VRF Instance

   • Configure the Tenant VRF Instances for IGMPv2 and IGMPv3 Multicast Receivers

   You don't need to configure any of the PIM or external multicast elements specific to OISM BL or SL devices.

Include the replicator-activation-delay option as shown. By default, AR leaf devices delay 10 seconds after receiving an AR replicator advertisement before starting to send traffic to that AR replicator device. In a scaled environment, we recommend you make the delay longer to ensure the AR replicator devices have fully learned the current EVPN state from the network. The delay also helps in cases when an AR replicator goes down and comes up again.

BL-1, BL-2, SL-1, SL-2, SL-4, SL-5, and SL-6:

We skip this configuration on SL-3, which acts as a device that doesn’t support AR.

SL-1 (device lo0: 192.168.0.1):

Run this command on each SL, BL, and S-ARR device in the fabric.

You can use the show ethernet-switching vxlan-tunnel-end-point remote command, or its alias, the show mac-vrf forwarding vxlan-tunnel-end-point remote command (where supported).

For example:

SL-1:

Note:

You also see the AR role of the device on each remote VTEP in the RVTEP-Mode column of the output from this command, as follows

• The primary loopback IP address on an AR replicator device is the ingress replication (IR) IP address, which the device uses to forward traffic to RNVE devices. That’s why you see “RNVE” as the role corresponding to the S-ARR device primary loopback addresses in the RVTEP-IP column.

• The secondary loopback IP address you assign for the AR replicator role is the AR IP address. You see “Replicator” as the role for those RVTEP-IP addresses in this output.

• The AR leaf devices and the RNVE device only use IR tunnels, so this command displays “Leaf” or “RNVE” role corresponding to the primary loopback IP address for those devices in the RVTEP-IP column.

Run this command on each SL, BL, and S-ARR device in the fabric.

For example:

SL-1 to TOR-1:

SL-2 to TOR-1:

Repeat this command on each SL device for interface ae3, and on SL-4 and SL-5, repeat this command for ae5.

BL-1:

BL-2:

The AR leaf devices detect the advertised AR replicator devices and load-balance among them using different methods based on the leaf device platform. (See AR Leaf Device Load Balancing with Multiple Replicators for details.)

In this example, SL-1 is a QFX5120 switch, so as an AR leaf device, SL-1 load balances by assigning an AR replicator device to each VLAN or VNI.

Run the show evpn multicast-snooping assisted-replication next-hops instance mac-vrf-instance command on the AR leaf devices to see the overlay tunnels and load-balancing next hops to the available AR replicators. On SL devices that designate AR replicators by VNI, the output of this command tags the AR replicator as the (Designated Node). The output doesn’t include this tag on AR leaf devices that load balance based on active traffic flow.

For example, the output here for SL-1 shows that the device assigned:

• S-ARR-1 as the designated replicator for configured VNIs 110001 and 110003 (which corresponds to VLAN-1 and VLAN-3, respectively)

• S-ARR-2 as the designated replicator for configured VNI 110002 and 110004 (which correspond to VLAN-2 and VLAN-4, respectively)

SL-1:

Run this command on the SL and BL devices in the fabric.

With AR enabled, the ingress SL device forwards the multicast source traffic to the designated AR replicator. See Figure 4. The AR replicator forwards copies to the SL devices with subscribed receivers on the source VLAN, VLAN-1. Then the SL devices forward the traffic toward the receivers on VLAN-1.

In this step, you run the commands only for VRF-1, which is the tenant VRF that hosts the internal multicast source traffic in this example. Also, this stream is an IGMPv3 stream with SSM reports, so you see only (S,G) multicast routes. In this case, the output shows the source behind TOR-1 has source IP address 10.0.1.12.

In this step, we show running verification commands for:

• PIM join status on SL-1 and SL-2 for the source on multihomed device TOR-1.

  The show pim join summary output shows that the SL devices serving TOR-1 saw joins for a total of 6 multicast groups.

• IGMP snooping multicast group membership status for the receiver behind device TOR-4, which is multihomed to SL-4 and SL-5.

  The show igmp snooping membership output shows the multicast group joins from the receiver behind TOR-4. TOR-4 hashes the join messages to either of the multihoming peer SL devices. The number of joins on both devices together (3 on each device) equals the total number of joins in the show pim join summary output (6).

• PIM join status summary and details on SL-4 and SL-5 for the receiver behind multihomed device TOR-4.

  When the show pim join extensive output on SL-4 and SL-5 show the same upstream and downstream IRB interface, the device is bridging the multicast stream within the same VLAN. When the downstream IRB interfaces are different from the upstream IRB interface, the device is routing the multicast stream between VLANs.

• The designated forwarder among multihoming peers SL-4 and SL-5 that the device elected to forward the traffic to the receiver behind multihomed TOR-4.

  We run the show evpn instance MACVRF-1 designated-forwarder esi command for the ESI you configured on the ae5 interfaces from SL-4 and SL-5 to TOR-4.

SL-1: Internal source—PIM join status for VRF-1:

SL-2: Internal source—PIM join status for VRF-1:

SL-4: Receiver—Multicast group membership status on source VLAN VLAN-1 and receiver VLANs VLAN-1 through VLAN-4::

SL-5: Receiver—Multicast group membership status on source VLAN VLAN-1 and receiver VLANs VLAN-1 through VLAN-4:

SL-4: Receiver—PIM join status for VRF-1:

SL-5: Receiver—PIM join status for VRF-1:

SL-4: Check the designated forwarder to the receiver behind TOR-4:

Recall that in the SL device configuration, we assign ae5 as the link from both SL-4 and SL5 to TOR-4. We set ESI 00:00:00:ff:00:04:00:01:00:05 on those links. The following output shows that SL-4 is not the designated forwarder for this ESI.

SL-5: Check the designated forwarder to the receiver behind TOR-4:

The following output confirms that SL-5 (lo0.0 192.168.0.5) is the designated forwarder for ESI 00:00:00:ff:00:04:00:01:00:05.

With AR enabled, the BL device forwards the traffic on the SBD VLAN to an AR replicator (either the designated AR replicator or an AR replicator based on traffic load-balancing, depending on the BL device platform). See Figure 4. The AR replicator forwards copies on the SBD to the SL devices with subscribed receivers. Then the SL devices forward or locally route the traffic toward the receivers on the tenant VLANs.

In this step, you run the commands only for VRF-101, which is the tenant VRF that hosts external multicast source traffic in this example. Also, this stream is an IGMPv2 stream with ASM reports, so you see only (*,G) multicast routes.

In this step, you run verification commands for:

• PIM join status on BL-1 and BL-2 as the ingress devices for the external multicast source.

• IGMP snooping multicast group membership status for a receiver behind device TOR-1, which is multihomed to SL-1 and SL-2.

• PIM join status on SL-1 and SL-2 for the receiver on multihomed device TOR-1.

BL-1: Ingress BL device for external source—PIM join status for VRF-101:

BL-2: Ingress BL device for external source—PIM join status for VRF-101:

SL-1: Receiver—Multicast group membership status for VLANs associated with VRF-101 (VLAN-401 through VLAN-404):

SL-2: Receiver—Multicast group membership status for VLANs associated with VRF-101 (VLAN-401 through VLAN-404):

SL-1: Receiver—PIM join status for VRF-101:

SL-2: Receiver—PIM join status for VRF-101:

For example, here we view Type 6 routes for interested receivers behind TOR-4, which is multihomed to peer SL devices SL-4 and SL-5. We show results for parameters of the featured multicast stream in Table 1 for tenant VRF-1, with IGMPv3 traffic between an internal source and internal receivers:

• VLANs: VLAN-1 through VLAN-4, which map to VNIs 110001 through 110004

• SBD VLAN: VLAN-2001, which maps to VNI 992001

• Internal source: Behind SL-1 and SL-2 on TOR-1, with internal IP address 10.0.1.12

• Multicast groups: 233.252.0.121 through 233.252.0.123

These commands show:

• S-ARR-1 and SL-ARR-2 received Type 6 routes from SL-4 and SL-5 on the SBD (VLAN-2001 VNI 992001).

• SL-4 (lo0: 192.168.0.4) and SL-5 (lo0: 192.168.0.5) received joins from a receiver behind multihomed TOR-4 for multicast groups 233.252.0.121 through 233.252.0.123.

• Source (10.0.1.12) and Group information, because the receivers send IGMPv3 joins.

For S-ARR-1 — Type 6 routes from SL-4 and SL-5:

S-ARR-1 links to SL-4 on ae4 (172.16.7.0/31).

S-ARR-1 links to SL-5 on ae5 (172.16.8.0/31):

Run the same commands on S-ARR-2 to see similar output on that device. S-ARR-2 links to SL-4 on ae4 (172.16.9.0/31) and SL-5 on ae5 (172.16.10.0/31).

SL-4 — Locally generated Type 6 routes, and Type 6 routes from other SL devices by way of S-ARR-1 and S-ARR-2:

You see similar Type 6 routes as remote routes from the other SL devices, which also serve receivers for the same groups in the same tenant VRF for the internal source on TOR-1 (multihomed to SL-1 and SL-2).

Run these commands on SL-5 to see the Type 6 routes on that device. Match on prefix 6:192.168.0.5 to see the locally generated Type 6 routes for SL-5. Match on other device prefixes (such as 6*192.168.0.4 for SL-4) to see the remotely generated Type 6 routes.

Use the show route table __default_evpn__.evpn.0 command to see Type 7 route prefixes.

For example, here we show Type 7 routes generated by peer SL devices SL-4 and SL-5 for the receiver behind TOR-4, with an internal source and IGMPv3 joins (see the parameters in Table 1 for VRF-1). TOR-4 hashes join messages from its receivers to either SL-4 or SL-5. The devices each advertise Type 7 routes to their multihoming peers for the joins they receive, to synchronize the join status among them.

These commands show:

• SL-5 locally generated Type 7 routes to advertise to SL-4

• SL-5 received Type 7 route advertisements from SL-4 for joins that TOR-4 hashed to SL-4

• OISM devices advertise Type 7 and Type 8 routes on the OISM revenue VLANs, not the SBD.

  In this case, receivers joined groups on VLAN-2 (VNI 110002, hashed to SL-5) and VLAN-3 (VNI-110003, hashed to SL-4).

Note that the ESI we configured for the SL-4 and SL-5 links to TOR-4 is 00:00:00:ff:00:04:00:01:00:05, which you see in the Type 7 route entry in the routing table.

SL-5 — Locally generated Type 7 routes to advertise to SL-4:

This output shows that SL-5 locally generated three Type 7 routes for joins on VLAN-2 (VNI 110002) for multicast groups 233.252.0.121 through 233.252.0.123. .

SL-5 — Type 7 routes from SL-4:

This output shows that SL-5 received three Type 7 route advertisements from SL-4 for joins on VLAN-3 (VNI 110003) for multicast groups 233.252.0.121 through 233.252.0.123.

Run these commands on SL-4 to see Type 7 routes on that device. Match on prefix 7:192.168.0.5 to see the Type 7 routes from its multihoming peer, SL-5. Match on prefix 7:192.168.0.4 to see the locally generated Type 7 routes that SL-4 advertises to SL-5.

You can use these commands to see Type 7 routes on the other multihoming peer SL device pairs.