close

keyboard_arrow_left

Table of Contents Expand all

list Table of Contents

English

Static VXLAN Tunnels with Q-in-Q

date_range 10-Oct-24

For small MC-LAG networks, you can use static VXLAN to reduce the control plane complexity in your network. Configuring VTEPs on a static VXLAN is straightforward. Use this example to configure static VXLAN tunnels with Q-in-Q tagging (VLAN translation) between data centers. In this example, we focus on the following features:

Static VXLAN—Static VXLAN connects servers in different data centers by creating a Layer 2 path (tunnel). For more information about static VXLAN, see Static VXLAN.
Q-in-Q tunnels—Q-in-Q tunnels segregates and bundles different customer VLAN (C-VLAN) traffic into a single service provider VLAN.

For more information about Q-in-Q tunnels, see Configuring Q-in-Q Tunneling and VLAN Q-in-Q Tunneling and VLAN Translation.
MC-LAG—MC LAG provides redundancy and load balancing. We configure ICL and ICCP connection between two peer devices to create the MC-LAG. For more information about MC-LAG, see Understanding Multichassis Link Aggregation Groups

Figure 1 shows a portion of a spine-leaf data center (POD). Within the POD, the TOR devices (TOR1 and TOR2) collect VLANs from the servers below and also manages the VLAN translations (Q-in-Q tunnels). The aggregators collect VLANs from different TOR devices and function as the gateway for the POD. We use a static VXLAN tunnel as a gateway between two PODs. We configure MC-LAG between the peer TOR devices and the peer aggregators. In our reference test environment, we tested a configuration with 64 pods. For this example, we describe how to configure the aggregators and TOR devices in a single pod.

Figure 1: Data Center POD with Q-in-Q and Static VXLAN Tunnels

This example is configured on top of an existing IP Fabric. See IP Fabric Underlay Network Design and Implementation.

Configuring the Aggregators

The following section describes how to configure the aggregators.

Configure the aggregators to support aggregated Ethernet and MC-LAG.
- Set the maximum number of aggregated Ethernet interfaces.
- Set the service identifier (SID) for the LAG.
- Configure a loopback address.
- Configure a management port. We use the management interface as an "always up" port to support the keepalive communication between ICCP peers.
AGG1 and AGG2
```
set chassis aggregated-devices ethernet device-count 64
set switch-options service-id 1
```
AGG1
```
set interfaces lo0 unit 0 family inet address 192.168.1.4/32 primary
set interfaces em0 unit 0 family inet address 10.48.49.69/221
```
AGG2
```
set interfaces lo0 unit 0 family inet address 192.168.1.5/32 primary
set nterfaces em0 unit 0 family inet address 10.48.49.117/22
```
Assign the aggregated Ethernet interfaces.
- ae0 and ae1 forms the ICL and ICCP links between the aggregators.
- ae3 connects the aggregator to the spine devices.
- ae4 connects the aggregators to the TOR devices.
AGG1 and AGG2
```
set interfaces xe-0/0/49:0 ether-options 802.3ad ae0
set interfaces xe-0/0/49:1 ether-options 802.3ad ae1
set interfaces xe-0/0/48:0 ether-options 802.3ad ae3
set interfaces xe-0/0/50:0 ether-options 802.3ad ae4
set interfaces xe-0/0/50:1 ether-options 802.3ad ae4
```

Enable LACP on the aggregated Ethernet interfaces. Enable LACP with the fast periodic interval to send a packet every second.

AGG1 and AGG2

content_copy zoom_out_map
set interfaces ae4 aggregated-ether-options minimum-links 1
set interfaces ae4 aggregated-ether-options lacp active
set interfaces ae4 aggregated-ether-options lacp periodic fast
set interfaces ae4 aggregated-ether-options lacp system-id 00:00:00:00:04:01
set interfaces ae4 aggregated-ether-options lacp admin-key 4

Configure the MC-LAG interfaces from the aggregators to the TOR devices and set it to active-active mode. Set a unique chassis ID for each peer.

AGG1 and AGG2

content_copy zoom_out_map
set interfaces ae4 aggregated-ether-options mc-ae mc-ae-id 4
set interfaces ae4 aggregated-ether-options mc-ae redundancy-group 1
set interfaces ae4 aggregated-ether-options mc-ae mode active-active
set interfaces ae4 aggregated-ether-options mc-ae status-control active
set interfaces ae4 aggregated-ether-options mc-ae init-delay-time 300

AGG1

content_copy zoom_out_map
set interfaces ae4 aggregated-ether-options mc-ae chassis-id 0

AGG2

content_copy zoom_out_map
set interfaces ae4 aggregated-ether-options mc-ae chassis-id 1

Configure the ICCP peers (AGG1 and AGG2) across the ICL. We use the IP address of the management link when we configure backup-liveness-detection to exchange keepalive messages.

AGG1

content_copy zoom_out_map
set interfaces ae0 description "ICCP link Connected to MCLAG peer"
set interfaces ae0 unit 0 family inet address 172.16.10.1/30
set multi-chassis multi-chassis-protection 172.16.10.2 interface ae1
set protocols iccp local-ip-addr 172.16.10.1
set protocols iccp peer 172.16.10.2 session-establishment-hold-time 600
set protocols iccp peer 172.16.10.2 redundancy-group-id-list 1
set protocols iccp peer 172.16.10.2 backup-liveness-detection backup-peer-ip 10.48.49.69
set protocols iccp peer 172.16.10.2 liveness-detection minimum-interval 1000

AGG2

content_copy zoom_out_map
set interfaces ae0 description "ICCP link Connected to MCLAG peer"
set interfaces ae0 unit 0 family inet address 172.16.10.2/30
set multi-chassis multi-chassis-protection 172.16.10.1 interface ae1
set protocols iccp local-ip-addr 172.16.10.2
set protocols iccp peer 172.16.10.1 session-establishment-hold-time 600
set protocols iccp peer 172.16.10.1 redundancy-group-id-list 1
set protocols iccp peer 172.16.10.1 backup-liveness-detection backup-peer-ip 10.48.49.117
set protocols iccp peer 172.16.10.1 liveness-detection minimum-interval 1000

Configure the interfaces to support VLANs.

AGG1 and AGG2

content_copy zoom_out_map
set vlans SP-VLAN-3000 vlan-id 3000
set vlans SP-VLAN-3000 interface ae1.3000
set vlans SP-VLAN-3000 interface ae4.3000
set vlans SP-VLAN-3001 vlan-id 3001
set vlans SP-VLAN-3001 interface ae1.3001
set vlans SP-VLAN-3001 interface ae4.3001

set interfaces ae1 unit 3000 encapsulation vlan-bridge
set interfaces ae1 unit 3000 vlan-id 3000
set interfaces ae1 unit 3001 encapsulation vlan-bridge
set interfaces ae1 unit 3001 vlan-id 3001

set interfaces ae4 description "Connected to TOR1 TOR2"
set interfaces ae4 flexible-vlan-tagging
set interfaces ae4 encapsulation flexible-ethernet-services

set interfaces ae4 unit 3000 encapsulation vlan-bridge
set interfaces ae4 unit 3000 vlan-id 3000
set interfaces ae4 unit 3001 encapsulation vlan-bridge
set interfaces ae4 unit 3001 vlan-id 3001

Configure the interface to the Spine devices.

AGG1

content_copy zoom_out_map
set interfaces ae3 description "Connected to Spine-1"
set interfaces ae3 unit 0 family inet address 192.168.100.2/24

AGG2

content_copy zoom_out_map
set interfaces ae3 description "Connected to Spine-1"
set interfaces ae3 unit 0 family inet address 192.168.200.2/24

Enable static VXLAN by configuring the local and remote VTEP interfaces.
AGG1 and AGG2
```
set switch-options vtep-source-interface lo0.0
set switch-options remote-vtep-list 192.168.1.6
```

Map the VLANs to the remote VTEP.

AGG1 and AGG2

content_copy zoom_out_map
set vlans SP-VLAN-3000 vxlan vni 103000
set vlans SP-VLAN-3000 vxlan ingress-node-replication
set vlans SP-VLAN-3000 vxlan static-remote-vtep-list 192.168.1.6
set vlans SP-VLAN-3001 vxlan vni 103001
set vlans SP-VLAN-3001 vxlan ingress-node-replication
set vlans SP-VLAN-3001 vxlan static-remote-vtep-list 192.168.1.6

Configuring the TOR Devices

The following section describes how to configure the TOR devices.

Configure the TOR device to support aggregated Ethernet and MC-LAG.
- Set the maximum number of aggregated Ethernet interfaces.
- Set the SID for the LAG.
- Configure a loopback address.
- Configure a management port. We use the management interface as an "always up" port to support the keepalive communication between ICCP peers.
TOR1 and TOR2
```
set chassis aggregated-devices ethernet device-count 64
set switch-options service-id 1
```
TOR1
```
set interfaces lo0 unit 0 family inet address 192.168.1.8/32 primary
set interfaces em0 unit 0 family inet address 10.48.49.197/22
```
TOR2
```
set interfaces lo0 unit 0 family inet address 192.168.1.9/32 primary
set interfaces em0 unit 0 family inet address 10.48.49.196/22
```

Assign the aggregated Ethernet interfaces.

ae0 and ae1 form the ICL and ICCP link between the TOR devices.
ae4 connects the TOR devices to the aggregators.
ae7 and ae8 connect the TOR devices to the servers.

TOR1 and TOR2

content_copy zoom_out_map
set interfaces xe-0/0/0:0 ether-options 802.3ad ae0
set interfaces xe-0/0/0:1 ether-options 802.3ad ae0
set interfaces xe-0/0/0:2 ether-options 802.3ad ae1
set interfaces xe-0/0/0:3 ether-options 802.3ad ae1
set interfaces xe-0/0/1:0 ether-options 802.3ad ae4
set interfaces xe-0/0/1:1 ether-options 802.3ad ae4
set interfaces xe-0/0/2:1 ether-options 802.3ad ae7
set interfaces xe-0/0/2:3 ether-options 802.3ad ae8

Enable LACP on the aggregated Ethernet interfaces. Enable LACP with a fast periodic interval to send a packet every second.

TOR1 and TOR2

content_copy zoom_out_map
set interfaces ae4 aggregated-ether-options minimum-links 1
set interfaces ae4 aggregated-ether-options lacp active
set interfaces ae4 aggregated-ether-options lacp periodic fast
set interfaces ae4 aggregated-ether-options lacp system-id 00:00:00:00:04:02
set interfaces ae4 aggregated-ether-options lacp admin-key 4

set interfaces ae7 aggregated-ether-options minimum-links 1
set interfaces ae7 aggregated-ether-options lacp active
set interfaces ae7 aggregated-ether-options lacp periodic fast
set interfaces ae7 aggregated-ether-options lacp system-id 00:00:00:00:07:01
set interfaces ae7 aggregated-ether-options lacp admin-key 7

set interfaces ae8 aggregated-ether-options minimum-links 1
set interfaces ae8 aggregated-ether-options lacp active
set interfaces ae8 aggregated-ether-options lacp periodic fast
set interfaces ae8 aggregated-ether-options lacp system-id 00:00:00:00:08:01
set interfaces ae8 aggregated-ether-options lacp admin-key 8

Configure the interfaces to support VLANs and Q-in-Q translation.

Note:

When you configure Q-in-Q mapping, the device selects the lowest value in the VLAN ID range as the outer tag. For example, with a range of 3000-3001 in our vlan-id-list, our device uses the VLAN 3000 as the outer tag. When the device receives an outgoing packet with a VLAN in the 3000 to 3001 range, the device pushes an outer tag of with a VLAN ID of 3000. Conversely, the device strips the outer tag for incoming packets with a VLAN ID of 3000 in its outer tag.

TOR1 and TOR2

content_copy zoom_out_map

set interfaces ae1 description "ICL link Connected to MCLAG peer"
set interfaces ae1 flexible-vlan-tagging
set interfaces ae1 encapsulation encapsulation extended-vlan-bridge
set interfaces ae1 unit 3000 vlan-id 3000

set interfaces ae4 description "Connected to AGG1 AGG2"
set interfaces ae4 flexible-vlan-tagging
set interfaces ae4 encapsulation extended-vlan-bridge
set interfaces ae4 unit 3000 vlan-id-list 3000-3001
set interfaces ae4 unit 3000 input-vlan-map push
set interfaces ae4 unit 3000 output-vlan-map pop

set interfaces ae7 description "Connected to Server1"
set interfaces ae7 flexible-vlan-tagging
set interfaces ae7 encapsulation extended-vlan-bridge
set interfaces ae7 unit 3000 vlan-id-list 3000-3001
set interfaces ae7 unit 3000 input-vlan-map push
set interfaces ae7 unit 3000 output-vlan-map pop

set interfaces ae8 description "Connected to Server2"
set interfaces ae8 flexible-vlan-tagging
set interfaces ae8 encapsulation extended-vlan-bridge
set interfaces ae8 unit 3000 vlan-id-list 3000-3001
set interfaces ae8 unit 3000 input-vlan-map push
set interfaces ae8 unit 3000 output-vlan-map pop

set vlans SP-VLAN-3000 interface ae1.3000
set vlans SP-VLAN-3000 interface ae4.3000
set vlans SP-VLAN-3000 interface ae7.3000
set vlans SP-VLAN-3000 interface ae8.3000
set vlans SP-VLAN-3000 service-id 3000
set vlans SP-VLAN-3001 interface ae4.3001
set vlans SP-VLAN-3001 interface ae7.3001
set vlans SP-VLAN-3001 interface ae8.3001
set vlans SP-VLAN-3001 service-id 3001

Configure the MC-LAG interfaces from the TOR devices to the aggregators and servers and set it to active-active mode. Set a unique chassis ID for each peer.

TOR1 and TOR2

content_copy zoom_out_map
set interfaces ae4 aggregated-ether-options mc-ae mc-ae-id 4
set interfaces ae4 aggregated-ether-options mc-ae redundancy-group 1
set interfaces ae4 aggregated-ether-options mc-ae mode active-active
set interfaces ae4 aggregated-ether-options mc-ae status-control active
set interfaces ae4 aggregated-ether-options mc-ae init-delay-time 300

set interfaces ae7 aggregated-ether-options mc-ae mc-ae-id 7
set interfaces ae7 aggregated-ether-options mc-ae redundancy-group 1
set interfaces ae7 aggregated-ether-options mc-ae mode active-active
set interfaces ae7 aggregated-ether-options mc-ae status-control active
set interfaces ae7 aggregated-ether-options mc-ae init-delay-time 300

set interfaces ae8 aggregated-ether-options mc-ae mc-ae-id 8
set interfaces ae8 aggregated-ether-options mc-ae redundancy-group 1
set interfaces ae8 aggregated-ether-options mc-ae mode active-active
set interfaces ae8 aggregated-ether-options mc-ae status-control active
set interfaces ae8 aggregated-ether-options mc-ae init-delay-time 300

TOR1

content_copy zoom_out_map
set interfaces ae4 aggregated-ether-options mc-ae chassis-id 0
set interfaces ae7 aggregated-ether-options mc-ae chassis-id 0
set interfaces ae8 aggregated-ether-options mc-ae chassis-id 0

TOR2

content_copy zoom_out_map
set interfaces ae4 aggregated-ether-options mc-ae chassis-id 1
set interfaces ae7 aggregated-ether-options mc-ae chassis-id 1
set interfaces ae8 aggregated-ether-options mc-ae chassis-id 1

Configure ICCP across the ICL between the two TOR peers (TOR1 and TOR2). We use the IP address of the management link when we configure backup-liveness-detection to exchange keepalive messages.

TOR1

content_copy zoom_out_map
set interfaces ae0 description "ICCP link Connected to MCLAG peer"
set interfaces ae0 unit 0 family inet address 172.16.20.1/30
set multi-chassis multi-chassis-protection 172.16.20.2 interface ae1
set protocols iccp local-ip-addr 172.16.2.1
set protocols iccp peer 172.16.20.2 session-establishment-hold-time 600
set protocols iccp peer 172.16.20.2 redundancy-group-id-list 1
set protocols iccp peer 172.16.20.2 backup-liveness-detection backup-peer-ip 10.48.49.197
set protocols iccp peer 172.16.20.2 liveness-detection minimum-interval 1000

TOR2

content_copy zoom_out_map
set interfaces ae0 description "ICCP link Connected to MCLAG peer"
set interfaces ae0 unit 0 family inet address 172.16.20.2/30
set multi-chassis multi-chassis-protection 172.16.20.1 interface ae1
set protocols iccp local-ip-addr 172.16.20.2
set protocols iccp peer 172.16.20.1 session-establishment-hold-time 600
set protocols iccp peer 172.16.20.1 redundancy-group-id-list 1
set protocols iccp peer 17.16.20.1 backup-liveness-detection backup-peer-ip 10.48.49.196
set protocols iccp peer 172.16.20.1 liveness-detection minimum-interval 1000

Verify the Static VXLAN Tunnels with Q-in-Q on Aggregators

This section shows how to verify the operation of the aggregators as it manages VLANs through the static VXLAN tunnels. All commands are issued on AGG1.

Display the VLAN information.

content_copy zoom_out_map
user@agg1> show vlans
default-switch          SP-VLAN-3000          NA       
                                                           ae1.3000*
                                                           ae4.3000*
                                                           vtep.32769*

Verify the operational status of the multichassis aggregated Ethernet link.

content_copy zoom_out_map
user@agg1> show interfaces mc-ae 
 Member Link                  : ae4
 Current State Machine's State: mcae active state
 Configuration Error Status   : No Error
 Local Status                 : active
 Local State                  : up
 Peer Status                  : active
 Peer State                   : up
     Logical Interface        : ae4.3000
     Topology Type            : bridge
     Local State              : up
     Peer State               : up
     Peer Ip/MCP/State        : 172.16.10.2  ae1.3000 up
    

Verify the MC-LAG status between AGG1 and AGG2.

content_copy zoom_out_map
user@agg1> show iccp 

Redundancy Group Information for peer 172.16.10.2 
  TCP Connection       : Established
  Liveliness Detection : Up
  Backup liveness peer status: Up
  Redundancy Group ID          Status
    1                           Up   

Client Application: lacpd
  Redundancy Group IDs Joined: 1 

Client Application: l2ald_iccpd_client
  Redundancy Group IDs Joined: 1 

Verify the LACP status on the aggregated Ethernet interface.

content_copy zoom_out_map
user@agg1> show lacp interfaces 

Aggregated interface: ae4
    LACP state:           Role   Exp   Def  Dist  Col  Syn  Aggr  Timeout  Activity
      xe-0/0/50:0        Actor    No    No   Yes  Yes  Yes   Yes     Fast    Active
      xe-0/0/50:0      Partner    No    No   Yes  Yes  Yes   Yes     Fast    Active
      xe-0/0/50:1        Actor    No    No   Yes  Yes  Yes   Yes     Fast    Active
      xe-0/0/50:1      Partner    No    No   Yes  Yes  Yes   Yes     Fast    Active
    LACP protocol:        Receive State  Transmit State          Mux State 
      xe-0/0/50:0               Current   Fast periodic Collecting distributing
      xe-0/0/50:1               Current   Fast periodic Collecting distributing

Verify the local and remote VTEP interfaces for the Static VXLAN is operational.

content_copy zoom_out_map
user@agg1> show ethernet-switching vxlan-tunnel-end-point source 
Logical System Name       Id  SVTEP-IP         IFL   L3-Idx    SVTEP-Mode    ELP-SVTEP-IP
<default>                 0   192.168.1.4      lo0.0    0  
    L2-RTT                   Bridge Domain              VNID     Translation-VNID    MC-Group-IP    Interface
    default-switch           SP-VLAN-3000               103000                       0.0.0.0             vtep.32768 
   


user@agg1> show ethernet-switching vxlan-tunnel-end-point remote 
Logical System Name       Id  SVTEP-IP         IFL   L3-Idx    SVTEP-Mode    ELP-SVTEP-IP
<default>                 0   192.168.1.4      lo0.0    0  
 RVTEP-IP         L2-RTT                   IFL-Idx   Interface    NH-Id   RVTEP-Mode  ELP-IP        Flags
 192.168.1.5      default-switch           825       vtep.32769   1784    RNVE      
    VNID          MC-Group-IP      
    103000        0.0.0.0         
   

Verify Q-in-Q Tunnels on the TOR Devices

This section shows how you verify the operation of VLANs on one of the TOR Device. All commands are issued on TOR1

Display the VLAN information.

content_copy zoom_out_map
user@tor1> show vlans
default-switch          SP-VLAN-3000          NA       
                                                           ae1.3000*
                                                           ae4.3000*
                                                           ae7.3000*
                                                           ae8.3000*
                                                          

Verify the operational status of the multichassis aggregated Ethernet link.

content_copy zoom_out_map
user@tor1> show interfaces mc-ae 
 Member Link                  : ae4
 Current State Machine's State: mcae active state
 Configuration Error Status   : No Error
 Local Status                 : active
 Local State                  : up
 Peer Status                  : active
 Peer State                   : up
     Logical Interface        : ae4.3000
     Topology Type            : bridge
     Local State              : up
     Peer State               : up
     Peer Ip/MCP/State        : 172.16.20.2  ae1.3000 up

Member Link                  : ae7
 Current State Machine's State: mcae active state
 Configuration Error Status   : No Error
 Local Status                 : active
Peer State                   : up
     Logical Interface        : ae7.3000
     Topology Type            : bridge
     Local State              : up
     Peer State               : up
     Peer Ip/MCP/State        : 172.16.20.2  ae1.3000 up

 Member Link                  : ae8     
 Current State Machine's State: mcae active state
 Configuration Error Status   : No Error
 Local Status                 : active
 Local State                  : up
 Peer Status                  : active
 Peer State                   : up
     Logical Interface        : ae8.3000
     Topology Type            : bridge
     Local State              : up
     Peer State               : up
     Peer Ip/MCP/State        : 172.16.20.1 ae1.3000 up

Verify the LACP status on the aggregated Ethernet interface.

content_copy zoom_out_map
user@tor1> show lacp interfaces
Aggregated interface: ae4
    LACP state:           Role   Exp   Def  Dist  Col  Syn  Aggr  Timeout  Activity
      xe-0/0/1:0         Actor    No    No   Yes  Yes  Yes   Yes     Fast    Active
      xe-0/0/1:0       Partner    No    No   Yes  Yes  Yes   Yes     Fast    Active
      xe-0/0/1:1         Actor    No    No   Yes  Yes  Yes   Yes     Fast    Active
      xe-0/0/1:1       Partner    No    No   Yes  Yes  Yes   Yes     Fast    Active
    LACP protocol:        Receive State  Transmit State          Mux State 
      xe-0/0/1:0                Current   Fast periodic Collecting distributing
      xe-0/0/1:1                Current   Fast periodic Collecting distributing
	  
Aggregated interface: ae7
    LACP state:           Role   Exp   Def  Dist  Col  Syn  Aggr  Timeout  Activity
      xe-0/0/2:1         Actor    No    No   Yes  Yes  Yes   Yes     Fast    Active
      xe-0/0/2:1       Partner    No    No   Yes  Yes  Yes   Yes     Fast    Active
    LACP protocol:        Receive State  Transmit State          Mux State 
      xe-0/0/2:1                Current   Fast periodic Collecting distributing

Aggregated interface: ae8
    LACP state:           Role   Exp   Def  Dist  Col  Syn  Aggr  Timeout  Activity
      xe-0/0/2:3         Actor    No    No   Yes  Yes  Yes   Yes     Fast    Active
      xe-0/0/2:3       Partner    No    No   Yes  Yes  Yes   Yes     Fast    Active
    LACP protocol:        Receive State  Transmit State          Mux State 
      xe-0/0/2:3                Current   Fast periodic Collecting distributing

Verify the MC-LAG status between TOR1 and TOR2.

content_copy zoom_out_map
user@tor1> show iccp 

Redundancy Group Information for peer 172.16.20.2
  TCP Connection       : Established
  Liveliness Detection : Up
  Backup liveness peer status: Up
  Redundancy Group ID          Status
    1                           Up   

Client Application: l2ald_iccpd_client
  Redundancy Group IDs Joined: 1 

Client Application: lacpd
  Redundancy Group IDs Joined: 1 

arrow_backward PREVIOUS Configure EVPN-VXLAN Data Center Stitching Through Interconnected EVPN-MPLS WAN Gateways

NEXT arrow_forward Configure L2PT over VXLAN Tunnels

Data Center EVPN-VXLAN Fabric Architecture Guide

ON THIS PAGE

Static VXLAN Tunnels with Q-in-Q

Configuring the Aggregators

Configuring the TOR Devices

Verify the Static VXLAN Tunnels with Q-in-Q on Aggregators

Verify Q-in-Q Tunnels on the TOR Devices

Stay in touch

Helped me install or use the product
Accelerated my deployment

Discuss the product with a co-worker
Make a purchase inquiry