Understanding FCoE LAGs

Why a Standard LAG Does Not Work for FCoE Traffic

Each physical link that carries FCoE traffic connects to a CNA port on an FCoE device. The connection that the FIP process creates between the CNA and the FC SAN switch emulates a point-to-point connection between that CNA and the SAN switch through the QFabric system Node device. If a connection to an FCoE device is not on a point-to-point link, communication from the FC SAN switch to the FCoE device CNA might not reach the CNA.

In a LAG, two (or more) physical links connect to the same device. Standard LAGs use a hashing algorithm to determine which physical LAG link to use for each transmission. Because the hashing algorithm might choose any LAG link for a given transmission, there is no way a standard LAG can guarantee that a response from the FC SAN will use the same LAG link on a Node device as the request from the CNA.

To ensure that communication between the CNA and the FC SAN is successful, communication from the SAN to the CNA must use the same physical link. If the FCoE CNA sends a request to the FC SAN, the response from the FC SAN must come on the same link the FCoE device CNA used to send the request. For example, if a if a request from the CNA goes out on Node device LAG member interface RSNG1:xe-0/0/20, then the response from the FC SAN must be received on interface RSNG1:xe-0/0/20.

If the FC SAN switch response to the FCoE CNA uses a different physical link on the Node device LAG, the response arrives at a different CNA port than the CNA port on which the request was sent. This breaks the virtual point-to-point link and the SAN switch response does not reach the correct requestor, so the response is lost. This is why a standard LAG does not work for FCoE traffic.

How an FCoE LAG Works

For FIP and FCoE transactions with the FC SAN to work properly, a LAG for FIP and FCoE traffic must allow the FC SAN switch to respond to the FCoE CNA device on the same link that the CNA used to communicate with the FC SAN switch.

To accomplish this, an FCoE LAG selects the member interface that the CNA used to communicate with the FC SAN switch as the link for the SAN switch response to the CNA. This preserves the virtual point-to-point link across the LAG and ensures that traffic from the FC SAN reaches the correct CNA port.

In a standard LAG, other devices learn the MAC address of the LAG interface, not the MAC address of the physical member interface that actually carries the traffic. However, for FCoE communication, other devices need to learn and use the VN_Port MAC address that the SAN switch assigns to the virtual node port (VN_Port) on the FCoE device’s CNA. The VN_Port MAC address uniquely identifies the CNA port used for FCoE transmission. (The VN_Port MAC address is based on the Fibre Channel ID and the FC-MAP value, which the FC SAN switch provides to the FCoE CNA as a unique port identifier.)

In an FCoE LAG, the Node device performs FIP snooping to learn the VN_Port MAC address of the CNA (in addition to other information). The Node device assigns the VN_Port MAC address to the particular interface that was used to connect to the CNA. For FCoE traffic, this replaces the normal LAG hashing logic, so instead of using an arbitrary LAG interface on the Node device for FCoE communication between the SAN switch and the CNA, an FCoE LAG uses the same physical LAG link for all FCoE transactions based on the VN_Port MAC address.

VLAN discovery traffic is untagged, so it must use a native VLAN. When you configure an FCoE LAG, VLAN discovery traffic on a native VLAN in the LAG also automatically uses the same physical link, preserving the virtual point-to-point link.

For multicast packets such as multicast discovery advertisements (MDAs), the advertisement is forwarded on all member links of the FCoE LAG. This ensures that multicast advertisements reach all of the FCoE devices attached to FCoE LAG member interfaces.

Behavior on FCoE LAG Link Failure

If an FCoE LAG link goes down, FCoE traffic and regular Ethernet traffic are treated differently.

If an FCoE LAG link goes down, the FCoE sessions on that link also go down. The Node device cannot simply move a session to another LAG link because that breaks the virtual point-to-point link. FCoE LAGs do not provide link redundancy for FCoE traffic.

As on a normal LAG, an FCoE LAG provides link redundancy for regular Ethernet traffic. Regular Ethernet sessions on the down FCoE LAG link are moved to other member links of the FCoE LAG (assuming that other member links are up).

FIP Snooping Session Scaling on QFabric System Node Devices

When the switch is on the FCoE access edge, you must enable FIP snooping on the FCoE VLAN to provide secure access when connecting to the FC SAN. (You can also enable FIP snooping on FCoE VLANs on switches that are not at the access edge if you want to collect FIP snooping statistics on the switch or if you are not confident that the edge switch is properly snooping traffic.)

FIP snooping VLANs support scaling up to 2,500 sessions by default, which is called enhanced FIP snooping scaling mode. Software releases before Junos OS Release 12.3 limited VN2VF_Port FIP snooping session scaling to 376 sessions on untrusted interfaces and untrusted FC fabrics, but scaled to 2,500 sessions on trusted interfaces and trusted FC fabrics. Starting with Junos OS Release 12.3, by default, all VN2VF_Port FIP snooping VLANs used enhanced FIP snooping scaling (2,500 sessions) for both trusted and untrusted interfaces and FC fabrics. The old limit of 376 sessions for untrusted interfaces and untrusted FC fabrics was deprecated and could not be configured.

The FCoE LAG feature introduces the ability to disable FIP snooping session scaling so that only 376 sessions are supported instead of the default 2,500 sessions. The reason for reintroducing FIP snooping session scaling limits is that when a Node device is configured as an FCoE-FC gateway that has one or more untrusted gateway Fibre Channel fabric (fc-fabric), placing FCoE traffic in a LAG forces the TCAM to store additional session data to ensure that the virtual point-to-point link between the FCoE device and the FC SAN is maintained. This case is described later in this document.

FCoE LAG Configuration on an FCoE Transit Switch

To create an FCoE LAG on an FCoE transit switch, you include the fcoe-lag option in the [edit interfaces interface-name aggregated-ether-options] hierarchy.

In addition to creating the FCoE LAG, you also need to:

• Add interfaces to the FCoE LAG.

• Configure at least one dedicated VLAN for FCoE traffic (an FCoE VLAN).

• Configure a native VLAN to carry untagged FIP traffic.

• Configure the FCoE LAG interfaces as a member of both the FCoE VLAN and the native VLAN.

• Enable FIP snooping on the FCoE VLAN.

FCoE LAG Configuration and FIP Snooping Scaling on an FCoE-FC Gateway

There are differences in the way you configure an FCoE LAG on an FCoE-FC gateway compared to configuring an FCoE LAG on an FCoE transit switch.

• Configuring an FCoE LAG on an FCoE-FC Gateway
• FIP Snooping Session Scaling on an FCoE-FC Gateway
• Summary of FCoE LAG and FIP Snooping Scaling on an FCoE-FC Gateway

FCoE Blade Switches

If you are using an FCoE blade switch, you need to configure an FCoE LAG only if the blade switch uses a passthrough module instead of an integrated switch.

Limitations

There are several limitations to configuring FCoE LAGs:

1. All FCoE LAG member links must belong to the same QFabric system Node group.

2. On an FCoE-FC gateway, you must disable FIP snooping scaling on untrusted fc-fabrics. Disabling FIP snooping scaling is global to the gateway Node device. If all of the fc-fabrics on an FCoE-FC gateway are trusted fabrics, you do not need to disable FIP snooping scaling.

3. FCoE LAGs with enhanced FIP snooping scaling enabled are not supported on untrusted FCoE-FC gateway fc-fabrics.