Example: Configuring an FCoE Transit Switch
You can use an EX4500 CEE-enabled switch as a Fibre Channel over Ethernet (FCoE) transit switch, enabling it to transport both FCoE and Ethernet LAN traffic. Using the same switch to support both your storage network and traditional IP-based data communications reduces the costs of powering, cooling, provisioning, maintaining, and managing your network.
This example includes:
FIP snooping for security
Priority-based flow control (PFC) for lossless transport
The FCoE forwarding class for the DCBX application protocol type, length, value (TLV) exchange
A trusted port connecting to the FCoE forwarder (FCF)
Enlarged maximum transmission unit (MTU) size for handling FCoE traffic
This example shows how to configure an FCoE transit switch:
Requirements
This example uses the following hardware and software components:
One EX4500 switch (CEE-capable model)
Junos OS Release 12.1 or later for EX Series switches
One FCoE Node (ENode)
One FCoE forwarder (FCF)
Before you begin, be sure you have:
Configured the VLAN
fcoe-vlanon the switch. See Configuring VLANs for EX Series Switches.
Overview and Topology
FCoE transmissions are vulnerable to address spoofing and man-in-the-middle attacks, because they are not actually sent through point-to-point links. This example describes how to configure the switch so that it provides security similar to that provided by traditional Fibre Channel (FC) networks. The switch is transparent to the ENode and the FCF, so the ENode and FCF communicate just as they would for a point-to-point link.
FIP snooping is disabled by default. You enable FIP snooping on a per-VLAN basis for VLANs that carry FCoE traffic. Ensure that a VLAN that carries FCoE traffic carries only FCoE traffic, because enabling FIP snooping denies access for all other Ethernet traffic.
This example shows how to configure FIP snooping on a VLAN of
the EX4500 switch that is connected with one ENode, that is, a server
equipped with converged network adapters (CNAs). The setup for this
example includes the VLAN fcoe-vlan on the switch.
This example also shows how to configure PFC on the interfaces that are being used for FCoE traffic and how to configure an FCoE trusted port to handle traffic between the switch and the FCF gateway to the storage area network (SAN).
You must configure PFC properties for the interfaces that are carrying FCoE traffic, because flow control must be implemented on the link level for this type of traffic.
Data Center Bridging Capability Exchange protocol (DCBX) is enabled by default on all 10-Gigabit Ethernet interfaces on EX4500 switches. DCBX automatically controls whether PFC is enabled or disabled on the interface. However, you must configure the PFC properties selecting the traffic class and queue. See Configuring Priority-Based Flow Control for an EX Series Switch (CLI Procedure).
You configure trunk interfaces that connect to the FCF as trusted interfaces. The switch must use the same FCoE MAC Address Prefix (FC-MAP) value that is being used by the FCF. Therefore, if the FCF is using a nondefault FC-MAP value, you must configure the FC-MAP value on the switch to match that value.
You must also enlarge the MTU size for all interfaces (both access and trunk) that are handling FCoE traffic to accommodate the maximum FC frame and Ethernet header sizes.
This example also includes configuring the fcoe forwarding
class to be used for the FCoE traffic, so that it can take advantage
of DCBX support for the Application Protocol TLV Exchange. See Understanding Data Center Bridging Capability Exchange
Protocol for EX Series Switches for additional information.
Configuring and applying PFC and a forwarding class fcoe on the DCBX interfaces automatically enables the DCBX
FCoE application protocol exchange on those interfaces. Do not explicitly
configure an FCoE application map, because doing that generates a
commit error. See Understanding Data Center
Bridging Capability Exchange Protocol for EX Series Switches for additional information.
PFC is supported only on 10-Gigabit Ethernet interfaces.
We recommend that you also:
Configure the PFC congestion notification profile for the same 802.1p code points that you are using for the
fcoeforwarding class. We recommend code point011, because this is the conventional IEEE 802.1p code point for FCoE traffic.Configure at least 20 percent of the buffer for the queue that is using PFC.
Do not specify the
exactoption when configuring the buffer for the queue that is using PFC.Configure the
loss-prioritystatement tolowfor a traffic class that is using PFC.Configure an appropriate percent of the buffer for any other forwarding classes (default forwarding classes and the user-defined forwarding classes) that you are using
Topology
The components of the topology for this example are shown in Table 1.
| Properties | Settings |
|---|---|
Switch hardware |
One EX4500 CEE-enabled switch |
VLAN name and ID |
|
Forwarding class for FCoE traffic |
|
Interfaces in |
|
FCoE trusted port to the FCF |
|
PFC interfaces |
|
CoS forwarding-class interface |
|
CoS scheduler-map interface |
|
Interfaces configured with MTU of 2500 |
|
In this example, the switch has already been configured as follows:
All access ports are untrusted, which is the default setting.
DCBX is enabled by default on all 10-Gigabit Ethernet interfaces.
The port connecting the switch to the FCF is configured as a trunk port.
Configuration
To configure an FCoE transit switch, perform these tasks:
Procedure
CLI Quick Configuration
To quickly configure an FCoE transit switch, copy the following commands and paste them into the switch terminal window:
[edit] set ethernet-switching-options secure-access-port vlan fcoe-vlan examine-fip fc-map 0x0EFC03 set ethernet-switching-options secure-access-port interface xe-0/0/30 fcoe-trusted set interfaces xe-0/0/1 ether-options no-flow-control set interfaces xe-0/0/2 ether-options no-flow-control set interfaces xe-0/0/3 ether-options no-flow-control set interfaces xe-0/0/30 ether-options no-flow-control set class-of-service congestion-notification-profile cn-profile input ieee-802.1 code-point 011 pfc set class-of-service interfaces xe-0/0/1 congestion-notification-profile cn-profile set class-of-service interfaces xe-0/0/2 congestion-notification-profile cn-profile set class-of-service interfaces xe-0/0/3 congestion-notification-profile cn-profile set class-of-service interfaces xe-0/0/30 congestion-notification-profile cn-profile set class-of-service classifiers ieee-802.1 pfc-class import default set class-of-service classifiers ieee-802.1 pfc-class forwarding-class fcoe loss-priority low code-points 011 set class-of-service interfaces xe-0/0/1 unit 0 classifiers ieee-802.1 pfc-class set class-of-service interfaces xe-0/0/2 unit 0 classifiers ieee-802.1 pfc-class set class-of-service interfaces xe-0/0/3 unit 0 classifiers ieee-802.1 pfc-class set class-of-service interfaces xe-0/0/30 unit 0 classifiers ieee-802.1 pfc-class set class-of-service forwarding-classes class fcoe queue-num 3 set class-of-service schedulers pfc-sched buffer-size percent 25 set class-of-service schedulers default-sched buffer-size percent 17 set class-of-service scheduler-maps pfc-map forwarding-class fcoe scheduler pfc-sched set class-of-service scheduler-maps pfc-map forwarding-class assured-forwarding scheduler default-sched set class-of-service scheduler-maps pfc-map forwarding-class best-effort scheduler default-sched set class-of-service scheduler-maps pfc-map forwarding-class network-control scheduler default-sched set class-of-service scheduler-maps pfc-map forwarding-class expedited-forwarding scheduler default-sched set class-of-service interfaces xe-0/0/30 scheduler-map pfc-map set interfaces xe-0/0/1 mtu 2500 set interfaces xe-0/0/2 mtu 2500 set interfaces xe-0/0/3 mtu 2500 set interfaces xe-0/0/30 mtu 2500
Step-by-Step Procedure
To configure an FCoE transit switch:
Enable FIP snooping on the VLAN and modify the FC-MAP value to match the FC-MAP value being used by the FCF:
[edit ethernet-switching-options secure-access-port] user@switch# set vlan fcoe-vlan examine-fip fc-map 0x0EFC03
Set the FCF-facing interface (
xe-0/0/30) as FCoE-trusted:[edit ethernet-switching-options secure-access-port] user@switch# set interface xe-0/0/30 fcoe-trusted
Configure a congestion notification profile, specifying the name of the profile and applying it to the traffic class that is indicated by the User Priority bits in the 802.1Q tagged frame of an incoming packet:
Note:The ENode and the switch must use the same traffic class for the FCoE traffic. DCBX advertises the traffic class being used by the switch and detects the traffic class being used by the ENode. If there is a mismatch, the switch disables the PFC capability of the switch interface.
[edit class-of-service] user@switch# set congestion-notification-profile cn-profile input ieee-802.1 code-point 011 pfc
Note:The configuration of PFC includes two different
ieee-802.1configuration statements:ieee-802.1 (Congestion Notification)—Use to configure the congestion notification profile.
ieee-802.1—Use to configure the CoS classifier.
Disable standard flow control on the interfaces that you want to use for the FCoE VLAN.
Note:PFC and standard flow control cannot be enabled on the same interface, and you must use PFC for FCoE traffic.
[edit interfaces] user@switch# set xe-0/0/1 ether-options no-flow-control user@switch# set xe-0/0/2 ether-options no-flow-control user@switch# set xe-0/0/3 ether-options no-flow-control user@switch# set xe-0/0/30 ether-options no-flow-control
Bind the congestion notification profile to all interfaces of the FCoE VLAN:
[edit class-of-service] user@switch# set interface xe-0/0/1 congestion-notification-profile cn-profile user@switch# set interface xe-0/0/2 congestion-notification-profile cn-profile user@switch# set interface xe-0/0/3 congestion-notification-profile cn-profile user@switch# set interface xe-0/0/30 congestion-notification-profile cn-profile
Create a CoS classifier for the fcoe forwarding class:
[edit class-of-service] user@switch# set forwarding-classes fcoe queue-num 3
Configure this forwarding class (fcoe) to use a low loss priority value and to use the same code point that is used for PFC:
Note:We recommend that you use code point 011, because this is the conventional IEEE 802.1p code point for FCoE traffic.
[edit class-of-service] user@switch# set classifiers ieee-802.1 pfc-class forwarding-class fcoe loss-priority low code-points 011
Bind the
pfc-classclassifier to all interfaces of the FCoE VLAN:[edit class-of-service] user@switch# set interfaces xe-0/0/1 unit 0 classifiers ieee-802.1 pfc-class user@switch# set interfaces xe-0/0/2 unit 0 classifiers ieee-802.1 pfc-class user@switch# set interfaces xe-0/0/3 unit 0 classifiers ieee-802.1 pfc-class user@switch# set interfaces xe-0/0/30 unit 0 classifiers ieee-802.1 pfc-class
Assign forwarding-class
fcoeto an egress queue:[edit class-of-service] user@switch# set forwarding-classes fcoe queue-num 3
Set a scheduler for this queue, allocating at least 20 percent of the buffer to
pfc-sched:[edit class-of-service] user@switch# set schedulers pfc-sched buffer-size percent 25
Set a scheduler for the default queue, allocating 17 percent of the buffer to that queue:
[edit class-of-service] uuser@switch# set schedulers default-sched buffer-size percent 17
Configure a scheduler map (
pfc-map) that associates the scheduler (pfc-sched) with thefcoeforwarding class and associates the default forwarding classes (assured-forwarding, best-effort and network-control) with the default schedule:[edit class-of-service] user@switch# set scheduler-maps pfc-map forwarding-class fcoe scheduler pfc-sched user@switch# set scheduler-maps pfc-map forwarding-class assured-forwarding schedulerdefault-sched user@switch# set scheduler-maps pfc-map forwarding-class best-effort scheduler default-sched user@switch# set scheduler-maps pfc-map forwarding-class network-control scheduler default-sched user@switch# set scheduler-maps pfc-map forwarding-class expedited-forwarding scheduler default-sched
Assign the scheduler map (
pfc-map) to the FCF-facing interface (xe-0/0/30):[edit class-of-service] user@switch# set interfaces xe-0/0/30 scheduler-map pfc-map
Enlarge the MTU size to 2500 bytes for all the interfaces (both access and trunk) that are handling FCoE traffic:
[edit interfaces] user@switch# set xe-0/0/1 mtu 2500 user@switch# set xe-0/0/2 mtu 2500 user@switch# set xe-0/0/3 mtu 2500 user@switch# set xe-0/0/30 mtu 2500
Results
Display the results of the configuration:
[edit]
user@switch#show interfaces {
xe-0/0/1 {
mtu 2500;
ether-options {
no-flow-control;
}
unit 0 {
family ethernet-switching {
vlan {
members fcoe-vlan;
}
}
}
}
xe-0/0/2 {
mtu 2500;
ether-options {
no-flow-control;
}
unit 0 {
family ethernet-switching {
vlan {
members fcoe-vlan;
}
}
}
}
xe-0/0/3 {
mtu 2500;
ether-options {
no-flow-control;
}
unit 0 {
family ethernet-switching {
vlan {
members fcoe-vlan;
}
}
}
}
xe-0/0/30 {
mtu 2500;
ether-options {
no-flow-control;
}
unit 0 {
family ethernet-switching {
port-mode trunk;
vlan {
members fcoe-vlan;
}
}
}
}
}
class-of-service {
classifiers {
ieee-802.1 pfc-class {
import default;
forwarding-class fcoe {
loss-priority low code-points 011;
}
forwarding-classes {
class fcoe queue-num 3;
}
congestion-notification-profile {
cn-profile {
input {
ieee-802.1 {
code-point 011 {
pfc;
}
}
}
}
}
interfaces {
xe-0/0/1 {
congestion-notification-profile cn-profile;
unit 0 {
classifiers {
ieee-802.1 pfc-class;
}
}
}
xe-0/0/2 {
congestion-notification-profile cn-profile;
unit 0 {
classifiers {
ieee-802.1 pfc-class;
}
}
xe-0/0/3 {
congestion-notification-profile cn-profile;
unit 0 {
classifiers {
ieee-802.1 pfc-class;
}
}
}
xe-0/0/30 {
congestion-notification-profile cn-profile;
scheduler-map pfc-map;
unit 0 {
classifiers {
ieee-802.1 pfc-class;
}
}
}
scheduler-maps {
pfc-map {
forwarding-class fcoe scheduler pfc-sched;
forwarding-class assured-forwarding scheduler default-sched;
forwarding-class best-effort scheduler default-sched;
forwarding-class network-control scheduler default-sched;
forwarding-class expedited-forwarding scheduler default-sched;
}
}
schedulers {
pfc-sched {
buffer-size percent 25;
}
default-sched {
buffer-size percent 17;
}
}
}
}
ethernet-switching-options {
secure-access-port {
interface xe-0/0/30.0 {
fcoe-trusted;
}
vlan fcoe-vlan {
examine-fip {
fc-map 0x0EFC03;
}
}
}
}
Verification
Confirm that the configuration of the FCoE transit switch is working properly:
- Verifying That FIP Snooping Is Working Correctly on the Switch
- Verifying That PFC is Enabled, That the FCoE Application Is Advertised, and That the Switch Interface and DCB Peer Are Using the Same 802.1p Code Points
Verifying That FIP Snooping Is Working Correctly on the Switch
Purpose
Verify that FIP snooping is being implemented on the appropriate VLAN.
Action
Send some requests from ENodes to the switch.
Display the FIP snooping information :
user@switch> show fip snooping vlan detail fcoe-vlan
VLAN: fcoe-vlan, FC-MAP: 0e:fc:03
FCF Information
FCF-MAC : 30:10:94:01:00:00
Active Sessions : 2
Configured FKA-ADV : 195
Running FKA-ADV : 73
Enode Information
Enode-MAC: 10:10:94:01:00:01, Interface: xe-0/0/1
Configured FKA-ADV : 195
Running FKA-ADV : 103
Session Information
VN-Port MAC: 0E:FC:03:01:0A:01, FKA-ADV : 178
VN-Port MAC: 0E:FC:03:01:0B:01, FKA-ADV : 194
FCF Information
FCF-MAC : 40:10:94:01:00:00
Active Sessions : 2
Configured FKA-ADV : 258
Running FKA-ADV : 212
Enode Information
Enode-MAC: 20:10:94:01:00:02, Interface: xe-0/0/0
Configured FKA-ADV : 258
Running FKA-ADV : 242
Session Information
VN-Port MAC: 0E:FC:03:02:0C:02, FKA-ADV : 254
VN-Port MAC: 0E:FC:03:02:0D:02, FKA-ADV : 269
Meaning
The output for this VLAN (fcoe-vlan) includes
the FC MAP value that you configured. It shows the MAC addresses of
the FCF and the ENode that are transmitting FCoE traffic through the
switch.
Verifying That PFC is Enabled, That the FCoE Application Is Advertised, and That the Switch Interface and DCB Peer Are Using the Same 802.1p Code Points
Purpose
Verify that PFC is enabled on the local switch interface and on the peer interface, and that the local interface and the peer interface are using the same code point.
Action
Send some requests from ENodes to the switch.
Display the DCBX information advertised by the configured CoS
forwarding class interface (xe-0/0/30) and detected by
the switch:
user@switch> show dcbx neighbors interface xe-0/0/30
Interface : xe-0/0/30.0
Protocol-State: in-sync
Local-Advertisement:
Operational version: 0
sequence-number: 1, acknowledge-id: 1
Peer-Advertisement:
Operational version: 0
sequence-number: 1, acknowledge-id: 1
Feature: PFC, Protocol-State: in-sync
Operational State: Enabled
Local-Advertisement:
Enable: Yes, Willing: No, Error: No
Maximum Traffic Classes capable to support PFC: 6
Code Point Admin Mode
000 Disabled
001 Disabled
010 Disabled
011 Disabled
100 Disabled
011 Enabled
110 Disabled
111 Disabled
Peer-Advertisement:
Enable: Yes, Willing: No, Error: No
Maximum Traffic Classes capable to support PFC: 6
Code Point Admin Mode
000 Disabled
001 Disabled
010 Disabled
011 Disabled
100 Disabled
011 Enabled
110 Disabled
111 Disabled
Feature: Application, Protocol-State: in-sync
Local-Advertisement:
Enable: Yes, Willing: No, Error: No
Appl-Name Ethernet-Type Socket-Number Priority-Map Status
FCoE 0x8906 00001000 Enabled
Peer-Advertisement:
Enable: Yes, Willing: No, Error: No
Appl-Name Ethernet-Type Socket-Number Priority-Map Status
FCoE 0x8906 00001000 Enabled
Meaning
PFC is a requirement for transmitting FCoE traffic and PFC works only when the local and peer devices are both enabled for PFC and are both using the same traffic class (code point) for transmitting the PFC traffic.
In the output for Feature: PFC, check the status
of Local-Advertisement to verify that PFC is enabled. If
DCBX detects a misconfiguration with the DCB peer, it disables the
PFC capability. In this example, the PFC Operational State is enabled, because PFC is configured symmetrically on
the switch and the DCB peer. Both devices are using code point 011 for forwarding the traffic.
If the results show that PFC is disabled, you van use the information provided by this command to reconfigure the congestion notification profile to match the code point being used for PFC by the peer device. See Configuring Priority-Based Flow Control for an EX Series Switch (CLI Procedure).
Appl-Name shows the default FCoE application. The
FCoE application always indicates Ethernet-Type 0x8906.
The Priority-Map for the FCoE application shows the 8-bit
format of the code-point setting that was specified for the PFC congestion
notification profile. In this case, the three bit code point is 3, 011. So the Priority-Map for the default FCoe application
is 00001000.
The fcoe forwarding-class and PFC were configured;
and the configuration of the application on the switch and on the
DCB are synchronized. Therefore, the Status of the FCoE
application is Enabled.
If the configuration of the FCoE application on the switch did
not match the FCoE application of the DCB peer, the status of the
application would appear as Disabled.