APPENDIX: Example ERB Fabric Creation

Note:

The configuration example shown below was made as part of phase 1 of this JVD. Please review the topology of phase 2. We’ve kept this example to introduce how to build a minimal ERB fabric.

Campus Fabric Core-Distribution ERB Components

This configuration example uses the following devices:

Two EX9204 switches as core devices, software version: Junos OS Release 22.4R2-Sx or later.
Two QFX5120 switches as distribution devices, software version: Junos OS Release 22.4R2-Sx or later.
Two access layer EX4400 switches, software version: Junos OS Release 22.4R2-Sx or later.
One MX80 WAN router, software version: Junos OS Release 21.2R3-Sx or later.
Juniper access points.
Two Linux desktops that act as wired clients.

Figure 1: Figure : Topology A diagram of a network Description automatically generated

Juniper Mist Wired Assurance

Juniper Mist Wired Assurance, through the Juniper Mist™ portal, can be used to centrally manage all Juniper switches. Juniper Mist Wired Assurance gives you full visibility into the devices that comprise your network’s access layer. The portal provides a user interface to access your architecture through the AI-driven cloud services with your Juniper Mist account. You can monitor, measure, and get alerts on key compliance metrics on the wired network. This includes switch version and PoE compliance, switch-AP affinity, and VLAN insights.

Juniper switch onboarding to the Juniper Mist cloud:

https://www.juniper.net/documentation/us/en/quick-start/hardware/cloud-ready-switches/topics/topic-map/step-1-begin.html

Juniper Mist Wired Assurance, through the portal, is used to build a Campus Fabric Core-Distribution ERB from the ground up. This includes the following:

Assignment of P2P links between the core and distribution layers.
Assignment of unique BGP AS numbers per device participating in the underlay and overlay.
The creation of VRF instances allows you to logically segment traffic. This also includes the assignment of new or existing VLANs to each representative VRF.
IP addressing of each Layer 3 gateway integrated routing and bridging (IRB) interface assigned to the distribution layer.
IP addressing of each loopback interface.
Configuration of routing policies for underlay and overlay connectivity.
Optimized maximum transmission unit (MTU) settings for P2P underlay, Layer 3 IRB, and ESI-LAG bundles.
Downloadable connection table (CSV format) that can be used by those involved in the physical buildout of the campus fabric.
Graphical interface depicting all devices with BGP peering and physical link status.

For more information on Juniper Mist Wired Assurance, see: https://www.mist.com/documentation/category/wired-assurance/

Juniper Mist Wired Assurance Switches

You must validate that each device participating in the campus fabric has been adopted or claimed and assigned to a site. The switches are named for respective layers in the fabric to facilitate building and operating the fabric.

Figure 2: Switch Inventory

Templates

A key feature of switch management through the Juniper Mist cloud is to use templates and a hierarchical model to group the switches and make bulk updates. Templates provide uniformity and convenience, while the hierarchy (site and switch) provides both scale and granularity.

Templates and the hierarchical model means that you can create a template configuration and then all the devices in each group inherit the template settings. When a conflict occurs, for example, when there are settings at both the site and organizational levels that apply to the same device, the narrower settings (in this case, the site settings) override the broader settings defined at the organization level.

Individual switches, at the bottom of the hierarchy, can inherit all or part of the configuration defined at the organization level, and again at the site level. Of course, individual switches can also have their own unique configurations.

You can include individual CLI commands at any level of the hierarchy, which are then appended to all the switches in that group on an “AND” basis—that is, individual CLI settings are appended to the existing configuration (existing settings might be replaced or appended).

Note:

If you run CLI commands for items not native to the portal, this configuration data is applied last; overwriting existing configuration data within the same stanza. You can access the CLI command option from the switch template or individual switch configuration.

Figure 3: Adding Additional CLI

Under organization and switch templates, we use the following template:

Figure 4: Switch Templates Background pattern Description automatically generated

We provide a copy of the following template in JSON format for importing into your own system for verification:

Topology

Juniper Mist Wired Assurance provides the template for LAN and loopback IP addressing for each core and distribution device once the device’s management IP address is reachable. Each device is provisioned with a /32 loopback address and /31 point-to-point interfaces that interconnect core and distribution devices within the Campus Fabric Core-Distribution. Devices such as the access layer switches connect to the distribution layer using standard LAGs; while the distribution uses ESI-LAG in a multihoming, load balancing manner.

The WAN router can be provisioned through the portal but is separate from the campus fabric workflow. The WAN router has a southbound LAG configured to connect to the ESI-LAG on the core switches. WAN routers can be standalone or built as a high availability cluster. In this document, a single MX router is used as the WAN router.

Note:

There is a JVD extension available covering more details on WAN router integration especially for production grade installations. What is shown here is a quick method that has known limits not feasible for production usage.

Create the Campus Fabric

From Organization on the left-hand side of the portal, select Campus Fabric.
Figure 5: Campus Fabric Creation

Mist provides the option of deploying a campus fabric at the organizational or site level noted in the upper-left side of the campus fabric menu shown below. Both designs now allow you to build fabrics with just a single PoD or multiple PoDs based on customer requirements to connect multiple buildings.

In our example here, the fabric was built on the site level with a single PoD only.

Figure 6: Fabric Site Level Creation
Choose the Campus Fabric Topology

Select the Campus Fabric Core-Distribution option below:

Figure 7: ERB Fabric Creation

Mist provides a section to name the Campus Fabric Core-Distribution ERB:
- Configuration—Provide a name in accordance with company standards
- Topology Sub-type:
  - CRB
  - ERB
Note:
ERB uses anycast addressing which provides shared IP addresses among all distribution layer devices participating in the L3 IRB. Deployments that require a routing protocol on the IRB must use CRB with virtual gateway addressing.You must choose CRB if most network traffic is north-south while ERB should be selected if mainly east-west traffic patterns exist.

Topology Settings
- BGP Local AS—Represents the starting point of private BGP AS numbers that are automatically allocated per device. You can use whatever private BGP AS number range suits your deployment, routing policy is provisioned by Mist to ensure the AS numbers are never advertised outside of the fabric.
- Subnet—Represents the pool of IP addresses used for point-to-point links between devices. You can use whatever range suits your deployment. Mist breaks this subnet into /31 subnet addressing per link. This number can be modified to suit the specific deployment scale. For example, /24 provides up to 128 P2P /31 subnets.
- Auto Router ID Subnet—Represents the pool of IP addresses associated with each device’s loopback address. Each device will automatically get a loopback IP address of /32 assigned from this pool. You can use whatever range suits your deployment. VXAN tunnelling using a VTEP is associated with this address. The loopback IP addresses assigned here are only visible in the underlay transport network. The definition of these underlay loopback IP addresses is critical for the operation of the EVPN-VXLAN fabric to function at all.
- Loopback per-VRF-subnet—Represents a second pool of loopback IP addresses which are each associated with an L3 VRF and switch of the overlay fabric network. It is designed for scale-out services in the overlay network where some services, like DHCP relay, share a single IP address external to the fabric. This is the case for anycast fabrics like ERB and IP Clos. If those L3 VRFs use a dedicated loopback IP address per VRF and switch, it is easy to send return answers to an originating VRF/switch.
Note:
In previous documentation, you did not have the default configuration fields for auto router ID subnet and loopback per-VRF subnet. Instead, you had a field for loopback prefix definition like shown below and then you had to assign the loopback IPs for each fabric node manually. This has changed in favor of automatic loopback assignments via the configuration of the prefix pool.

Figure 8: Older Fabric Configuration Options

Note:
We recommend default settings for all options unless it conflicts with other networks attached to the campus fabric. The P2P links between each layer utilize /31 addressing to conserve addresses.
Select Campus Fabric Nodes

Select devices to participate in each layer of the Campus Fabric Core-Distribution ERB. We recommend that you validate each device’s presence in the site switch inventory prior to the creation of the campus fabric.

The next step is to assign the switches to the layers. Since the switches were named relative to target layer functionality, they can be quickly assigned to their roles.

The services block router is where the campus fabric interconnects external devices such as firewalls, routers, or critical devices. For example, DHCP and RADIUS servers. Devices to which external services connect to the campus fabric are known as border leafs. If you want to connect these services or devices to the Campus Fabric Core-Distribution ERB in a separate device or pair of devices, clear the Use Core as border option and select the Select Switches option to choose the devices.

Figure 9: Select the Fabric Nodes

Note:
Placing the services block router on a dedicated pair of switches (or single switch) alleviates the encapsulation and de-capsulation of VXLAN headers from the core layer. If you want to combine this capability within the core devices, you must select the Use Core as border option.
Once all layers have selected the appropriate devices, you must provide an underlay loopback IP address for each device (with the exception of the access switches). This loopback interface is associated with a logical construct called a VTEP; used to source the VXLAN tunnel. The Campus Fabric Core-Distribution ERB has VTEPs for VXLAN tunnelling on the distribution switches and the core switches when enabling the core border option.

When defining an auto router ID subnet prefix, the underlay loopback IP address and router ID assignments happens automatically. There is no need to manually assign them. You may still see warnings like the one below about a 5 router ID, you can ignore those as the automatic assignment happens at a later phase.

Figure 10: Router ID Not Assigned Yet

Note:
If the auto router ID subnet field is not configured or is empty, you can use the previous mode of operation and manually assign the underlay loopback IP addresses as router IDs to each device needing one. Make sure that all IP addresses are in the same subnet as required by the Mist cloud fabric configuration.
Configuring Networks

Enter the network information such as VLANs and VRF options. VLANs are mapped to VNIs and can optionally be mapped to VRFs to provide a way to logically separate traffic such as IoT device traffic from Corp IT traffic.

Figure 11: Configure Networks
Networks

VLANs can be created or imported under this section including the IP subnet and default gateway per each VLAN.

The Shared Elements section of the campus fabric template includes the networks section mentioned above where VLANs are created.

Figure 12: Networks Inherited by Switch Template
Back to the campus fabric build, select the existing template which includes Layer 2 VLAN information. All VLAN and IP information is inherited from the template.

Figure 13: Network Import from Template

Networks can be edited, newly added or added from an existing template:

Figure 14: Edit a Network

Other IP Configuration

Juniper Mist Wired Assurance provides automatic IP addressing for IRB interfaces for each of the VLANs. Port profiles and port configurations then associate the VLAN with specified ports. In this case, we selected campus fabric ERB at the onset of the campus fabric build.

Figure 15: ERB Selection

This option uses anycast addressing for all devices participating in the L3 subnet. In this case, Dist1 and Dist2 switches are configured with the same IP address for each L3 subnet.

More information on anycast gateways can be found here: https://www.juniper.net/documentation/us/en/software/junos/evpn-vxlan/topics/concept/evpn-mclag-irb-gateway-anycast-address.html

Figure 16: Anycast GW on Distribution Switch 1

Figure 17: Anycast GW on Distribution Switch 2

By default, all VLANs are placed in the default VRF. The VRF option allows you to group common VLANs into the same VRF or separate VRFs depending on traffic isolation requirements. This example includes three VRFs or routing instances: corp-it, developers, and guest-wifi.
Here, you build the first corp-it VRF and select the pre-defined vlan 1099.

Figure 18: Enable VRF

Figure 19: Assign Network to VRF

By default, inter-VRF communications are not supported within the campus fabric. If inter-VRF communications is required, each VRF can include extra routes such as a default route that instructs the campus fabric to use an external router or firewall for further security inspection or routing capabilities. In this example, all traffic is trunked over the ESI-LAG and the MX router handles inter-VRF routing. See Figure 11: Topology.

Notice the MX router participates in the VLANs defined within the campus fabric and is the gateway of last resort for all traffic leaving the subnet.
Select the Add Extra Routes option to inform Mist to forward all traffic leaving 10.99.99.0/24 to use the next hop of the MX router: 10.99.99.254.

Figure 20: Add Default Route
Create two additional VRFs:
1. The developers VRF using vlan 1088 with 0.0.0.0/0 utilizing 10.88.88.254
2. The guest-wifi VRF using vlan 1033 with 0.0.0.0/0 utilizing 10.33.33.254
Figure 21: Entire Network and VRF Configuration
As a next step, you need to provide a name like “erb-lag” that the fabric will use to establish the redundant LAG interfaces between all access and distribution switches. All created VLANs should be automatically added already as future trunk networks.
Figure 22: Fabric LAG Configuration
The section configures the active-active ESI-LAG trunks between distribution and access switches. Here, we name the port configuration and include VLANs associated with this configuration. The advanced tab provides additional configuration options:

Figure 23: Fabric LAG

Note:
We recommend default settings unless specific requirements are needed.
Now that all VLANs are configured and assigned to each VRF, and the distribution and access ESI-LAGs have been built, click the Continue button in the upper-right corner of the portal to move to the next step.

Configure Campus Fabric Ports

The final step is the selection of physical ports among core, distribution, and access switches.

Figure 24: Port Overview
Core Switches

Core1:

Starting with Core1, select xe-1/0/5 and xe-1/0/6 terminating on distribution switches 1 and 2, respectively.

Figure 25: First Port Core1

Figure 26: Figure :Second port core1
Core2:

On Core2, select xe-1/0/4 and xe-1/0/5 terminating on distribution switches 1 and 2, respectively.

Figure 27: Figure :First Port Core2

Figure 28: Second Port Core2
Distribution Switches

Now moving on to the distribution switches, you notice two interconnect options exist:
- Link to Core
- Link to Access
Dist1:

Select Link to Core and choose xe-0/0/5 and xe-0/0/4 terminating on Core Switches 1 and 2, respectively.

Figure 29: First Uplink Port Dist1

Figure 30: Second Uplink Port Dist1
Select Link to Access and choose ge-0/0/36 and ge-0/0/37 terminating on access switches 1 and 2, respectively.

Figure 31: First Downlink Port Dist1

Figure 32: Second Downlink Port Dist1
Next, select the following interconnects off Dist2:
1. Link to Core
  - 1. xe-0/0/6 – Core1
    2. xe-0/0/5 – Core2
2. Link to Access
  - 1. ge-0/0/36 – Access2
    2. ge-0/0/37 – Access1
Access Switches

You only need to know which interfaces are used to interconnect with the distribution switch but do not need to know the specific mapping. The system bundles all interfaces into a single Ethernet bundle through the AE index option. This greatly simplifies the physical port build for each access switch.

Access1 and 2:

Select both uplinks and interface speed, while allowing Mist to define each AE index. In this case, uplinks ge-0/0/36+37 are selected as Links to Distribution on both access switches and AE Index 11 and 12 on Access1 and 2, respectively.

Figure 33: Uplink and AE# on Access1

Figure 34: Uplink and AE# on access2
Once you have completed selecting all requisite port combinations, select the Continue button in the upper-right corner of the portal.

Campus Fabric Configuration Confirmation

This last section provides the ability to confirm each device’s configuration as shown below:

Figure 35: Fabric Confirmation View

Note:
As we have configured the usage of auto router ID subnet, the underlay loopback IP addresses may still not be assigned on this page and warnings may appear like the ones shown above. Please ignore this for now as the assignments happen when you apply the configuration for the first time.
Once you have completed verification, select the Apply Changes option at the upper-right corner of the portal.

Figure 36: Apply Changes to Fabric

You must complete the second stage confirmation to create the fabric.

Mist displays the following banner including the estimated time for the campus fabric to be built. The process includes the following:
- Mist builds the point-to-point interfaces between distribution and core devices with IP addresses chosen from the range presented at the onset of the build.
- Each device is configured with a loopback address from the range presented at the onset of the build.
- eBGP is provisioned on each device with unique BGP autonomous system numbers. The primary goal of the underlay is to leverage ECMP for load balancing traffic on a per packet level for device loopback reachability. The primary goal of the eBGP overlay is support of customer traffic using EVPN-VXLAN.
- IP addressing of each L3 gateway IRB located on Dist1 and Dist2.
- IP addressing of each lo0.0 loopback, which is done automatically in this case.
- Configuration of routing policies for underlay and overlay connectivity.
- Optimized MTU settings for P2P underlay, L3 IRB, and ESI-LAG bundles.
- VXLAN to VLAN mapping using VNI addresses that are automatically assigned.
- VRF creation of corp-it, developers, and guest-wifi and VLAN associated with each VRF.
- VXLAN tunnelling creation between distribution devices and distribution-core devices (in support of the northbound MX router that is configured in subsequent steps).
- Downloadable connection table (CSV format) that can be used by those involved in the physical buildout of the campus fabric.
- Graphical interface depicting all devices with BGP peering and physical link status.
Figure 37: Applying Changes
Once you click Close Campus Fabric Configuration, you can view a summary of the newly created Campus Fabric Core-Distribution ERB.

Figure 38: Created ERB Fabric View

With Juniper Mist Wired Assurance, you can download a connection table (CSV format) representing the physical layout of the campus fabric. This can be used to validate all switch interconnects for those participating in the physical campus fabric build. Once the campus fabric is built or in the process of being built, you can download the connection table.

Figure 39: Download Connection Table CSV

Connection Table spreadsheet:

Figure 40: Downloaded Connection Table

Apply VLANs to Access Ports

As previously discussed, Mist provides the ability to templatize well known services such as RADIUS, NTP, DNS, and so on that can be used across all devices within a site. These templates can also include VLANs and port profiles that can be targeted at each device within a site. The last step before verification is to associate VLANs with the requisite ports on each access switch.

In this case, Desktop1 and 2 are associated with different ports on each access switch which requires the configuration to be applied to Access1 and 2, respectively. See Figure 1.

It is also noteworthy that Juniper access points connect to the same port on Access1 and 2 allowing the switch template to be customized with this configuration. For example, the following found under the switch template option is customized to associate each switch with its role: core, distribution, and access. Furthermore, all access switches (defined by the Juniper Networks® EX4400 Switch, as an example) associated the AP port profile named “myaccess” with ge-0/0/16 without needing to configure each independent switch.

Figure 41: Port Configuration Via Switch Template

Using Access1 as an example, we apply vlan1099 to port ge-0/0/11 under the Port Configuration section on Access1. In this example, vlan1099 (corp-it), vlan1088 (developers), and vlan1033 (guest-wifi) are defined in the switch template. Here, vlan1099 is selected under the configuration profile.

Figure 42: Assign Port Profile to a Port

The switch template definition for vlan1099 is shown below, representing attributes associated with VLANs such as dot1x authentication, QoS, and PoE. Vlan1088 and vlan1033 need to be configured in a similar fashion.

Figure 43: Port Profile Example A screenshot of a phone Description automatically generated with medium confidence

ON THIS PAGE

APPENDIX: Example ERB Fabric Creation

Campus Fabric Core-Distribution ERB Components

Juniper Mist Wired Assurance

Juniper Mist Wired Assurance Switches

Templates

Topology

Create the Campus Fabric

Apply VLANs to Access Ports