Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

 
ON THIS PAGE
 

Understanding Load Balancing in an FCoE-FC Gateway Proxy Fabric

You can balance or rebalance the load on the ports in an FCoE-FC gateway proxy fabric in order to avoid overutilizing or underutilizing the links. Load balancing is distributing sessions across the available native Fibre Channel (FC) interfaces (NP_Ports) that belong to a local gateway FC fabric to create a relatively equal load on all the fabric links. Load rebalancing is redistributing the existing sessions across the available NP_Port links on a local gateway FC fabric.

Note:

A session is a fabric login (FLOGI) or fabric discovery (FDISC) login to the FC SAN fabric. Session does not refer to end-to-end server-to-storage sessions.

The fabric-facing NP_Port links of an FCoE-FC gateway use different load-balancing methods than the FCoE-network-facing Ethernet links.

Balancing the load on FCoE-FC gateway NP_Port links consists of two steps:

  1. Choosing the algorithm used to balance and rebalance the link load

  2. Choosing whether to rebalance link loads automatically or only when you explicitly request a rebalance (load-rebalancing method)

You can configure a different load-balancing algorithm and use a different rebalancing method for each local FC fabric on the FCoE-FC gateway. The load-balancing algorithm and automated rebalancing, if configured, apply to all NP_Port interfaces in the local FC fabric.

This topic describes:

Load-Balancing Algorithms

You can choose one of three load-balancing algorithms to configure the way the switch balances the link loads. The switch uses the configured algorithm to balance the link loads when NP_Ports are initialized and whenever link loads are rebalanced. Regardless of whether you configure automated load rebalancing or use on-demand load rebalancing, the switch uses the configured algorithm to balance the link load:

  • Simple load balancing—The switch assigns each ENode FLOGI session and VN_Port FDISC session to the least-loaded link. The switch can place FDISC sessions on a different link than the parent FLOGI session (an ENode FLOGI session and its subsequent FDISC sessions can be placed on different links). Simple load balancing is the default load-balancing algorithm. Rebalancing the link load disrupts only selected sessions to minimize the impact (the switch uses an algorithm to log out only the sessions that need to be moved to other links to balance the load when those sessions log in again).

  • ENode-based load balancing—When an ENode logs in to the fabric, the switch places all subsequent VN_Port FDISC sessions associated with that ENode on the same link as the ENode FLOGI session, regardless of the link load. New ENode FLOGIs are placed on the least-loaded link. The switch calculates the link load based on the combined total of FLOGIs and FDISCs on each NP_Port link. Rebalancing the link load disrupts all sessions (all sessions log out and then log in again).

  • FLOGI-based load balancing—Similar to ENode-based load balancing; when an ENode logs in to the fabric, the switch places all subsequent VN_Port FDISC sessions associated with that ENode on the same link as the ENode FLOGI session, regardless of the link load. New ENode FLOGIs are placed on the least-loaded link.

    One difference between ENode-based load balancing and FLOGI-based load balancing is that the switch calculates the link load based only on the number of FLOGIs on each NP_Port link. The algorithm does not count FDISCs. Another difference is that instead of disrupting all sessions on a link load rebalance, the system disrupts only selected sessions to minimize the impact (the switch uses an algorithm to log out only the sessions that need to be moved to other links to balance the load when those sessions log in again).

Note:

Changing the load-balancing algorithm when FCoE sessions are running forces the FCoE sessions to log out and then log in again.

If you do not explicitly configure the load-balancing algorithm, the switch uses simple load balancing by default on the all NP_Port interfaces that belong to a given local FC fabric.

The following sections describe how each algorithm works, its advantages and disadvantages, and what happens when NP_Port links come up for the first time, when an NP_Port link is added to existing links, and when you rebalance the link load:

Simple Load Balancing

Simple load balancing provides the most equal load balancing across links because each VN_Port FDISC session can be assigned to the least-loaded link, regardless of whether the parent ENode FLOGI session is on that link. (The parent ENode is the ENode that originates the logins to the fabric. After the parent ENode logs in, the VN_Ports on that ENode can log in to the fabric using FDISC.)

The FCoE-FC gateway performs simple load balancing by default on the NP_Ports that connect the gateway to the FC SAN. When an ENode sends a FLOGI request to the gateway, the gateway checks the NP_Ports that connect it to the FC SAN and assigns the new session to the least-loaded link.

Every time an ENode sends a FLOGI or an FDISC request, the gateway assigns the new session to the least-loaded NP_Port link. After the gateway assigns an ENode FLOGI session to an NP_Port, subsequent FDISC requests by the same ENode can result in sessions being assigned to different NP_Ports, because the gateway always assigns the new session to the least-loaded interface.

Note:

Because VN_Port sessions might be placed on a different link than their parent ENode, if the link that contains the ENode goes down, only the ENode session and any of its VN_Port sessions that are on that link go down. VN_Port sessions on other links remain active as long as the link is up and the VN_Port is not logged out.

When a new link comes up, the switch logs out enough sessions so that when the sessions log in again, they are placed on the new link and the link loads are balanced. The switch uses an algorithm to log out sessions in the least disruptive manner by first logging out FDISCs whose FLOGI is not on the same link, then the least-loaded FLOGIs (loaded in terms of related FDISC logins).

Similarly, when you rebalance an existing link load, the switch logs out only enough sessions so that when the sessions log in again, they balance the load on the existing links. In this case (rebalance without a new link up), the switch takes into account the dependencies between FLOGIs and FDISCs when selecting sessions to log out.

The simple load-balancing algorithm uses the sum of the FLOGI and FDISC sessions to determine the session load on each link for both initial load balancing and load rebalancing.

ENode-Based Load Balancing

ENode-based load balancing can result in a less balanced load across the NP_Port links because the VN_Port FDISC sessions are assigned to the same link as the parent ENode FLOGI session, regardless of how many FDISC sessions are associated with the ENode. However, ENode-based load balancing has the advantage of keeping all of the sessions associated with a particular ENode on one link, which provides better control and predictability.

When you use the ENode-based load-balancing algorithm, the gateway assigns the ENode to an NP_Port link when the ENode sends its FLOGI message to the gateway. The gateway places the ENode session on the least-loaded link at that time. The VN_Port FDISC sessions associated with an ENode are placed on the same link as the ENode FLOGI session, regardless of the link load. Essentially, the ENode sessions are load-balanced, but the VN_Port sessions are not.

ENode-based load balancing ensures that each ENode and its associated VN_Port sessions are assigned to the same NP_Port link. ENode-based load balancing provides more control and predictability and ensures that if the link carrying an ENode goes down, all of the ENodes associated VN_Port sessions also go down.

The disadvantage of ENode-based load balancing is that if one ENode has a large number of sessions and the other ENodes do not, the link that carries the ENode with the large number of sessions might have a much larger load than the other NP_Port links in the gateway proxy fabric.

For example, if a gateway fabric has two NP_Ports connected to the FC fabric, and two ENodes log in to the fabric, one ENode session is placed on each link. If two VN_Port sessions are initiated on one of the ENodes, those sessions are placed on the same link as the parent ENode. If 1000 VN_Port sessions are initiated on the other ENode, all of the 1000 VN_Port sessions are placed on the same link as that ENode. In this case, one link has 3 sessions (1 ENode FLOGI session and 2 VN_Port FDISC sessions) and the other link has 1001 sessions (1 ENode FLOGI session and 1000 VN_Port FDISC sessions).

When a new link comes up or when you rebalance an existing load, the switch logs out all sessions (FLOGIs and FDISCs) in the fabric. As the sessions log in again, the switch assigns them to NP_Ports in a balanced manner, with all FDISCs assigned to the same link as the parent FLOGI. A new link coming up or a rebalance disrupts all of the existing sessions.

The ENode-based load-balancing algorithm uses the sum of the FLOGI and FDISC sessions to determine the session load on each link for both initial load balancing and load rebalancing.

FLOGI-Based Load Balancing

FLOGI-based load balancing is similar to ENode-based load balancing in most ways:

  • It can result in a less balanced load across the NP_Port links because the VN_Port FDISC sessions are assigned to the same link as the parent ENode FLOGI session, regardless of how many FDISC sessions are associated with the ENode.

  • When an ENode logs in with a FLOGI, the gateway places the session on the least-loaded link, and the FDISC logins associated with the FLOGI are placed on the same link, regardless of link load.

  • Provides control and predictability because each ENode and its associated VN_Port (FDISC) sessions are assigned to the same link, so if the link an ENode is on goes down, all of its associated sessions also go down.

  • If one ENode has a large number of sessions and the other ENodes do not, the link that carries the ENode with the large number of sessions might have a much larger load than the other NP_Port links in the gateway proxy fabric.

FLOGI-based load balancing differs from ENode-based load balancing in two important ways:

  1. The switch uses the sum of the FLOGI sessions on a link to determine the link load. The switch does not use FDISC sessions when calculating the number of sessions on a link. (ENode-based load balancing uses the sum of the FLOGI and FDISC sessions to calculate the number of sessions on a link.)

  2. When a new link comes up or when you rebalance an existing load, the switch logs out enough FLOGI (and FDISC) sessions so that when the FLOGI sessions log in again, the load is balanced. The switch balances the load based only on the number of FLOGI sessions, not the sum of FLOGI and FDISC sessions. However, the FDISC sessions associated with a FLOGI follow the FLOGI to the new link if the FLOGI session is part of the rebalancing.

The FLOGI-based load-balancing algorithm uses only the FLOGI sessions to determine the session load on each link for both initial load balancing and load rebalancing.

Load-Balancing Algorithm Comparison

Table 1 compares the three load-balancing algorithms and summarizes their differences, advantages, and disadvantages.

Table 1: Load-Balancing Algorithm Comparison

Load-Balancing Algorithm

Session Assignment

Session Disruption on Rebalance

Session Count Method

Advantages

Disadvantages

Simple (default algorithm)

FDISC sessions can be placed on different links than the parent FLOGI session

Minimum number of selected sessions logged out (FDISC sessions can be logged out independent of the parent FLOGI session)

Sum of FLOGI and FDISC sessions

  • Most equal session distribution across links

  • Minimum number of sessions logged out when rebalancing

  • Least disruptive algorithm

  • Less session control and predictability

ENode-based

FDISC sessions are always placed on the same link as the parent FLOGI session

All sessions are logged out

Sum of FLOGI and FDISC sessions

  • Better session control and predictability (on link down, all sessions associated with an ENode go down)

  • Most disruptive algorithm; all sessions logged out on rebalance

  • Might result in less balanced link load because FDISCs are placed on the same link as parent FLOGI

FLOGI-based

FDISC sessions are always placed on the same link as the parent FLOGI session

Minimum number of selected sessions logged out (but FDISC sessions logged out when parent FLOGI session is logged out)

FLOGI sessions only (FDISC sessions not included in the session count)

  • Better session control and predictability (on link down, all sessions associated with an ENode go down)

  • Minimum number of sessions logged out when rebalancing

  • Might result in less balanced link load because FDISCs are placed on the same link as parent FLOGI

Load-Rebalancing Methods

The load-rebalancing method determines the way the system redistributes sessions to balance the load on the NP_Ports that belong to a local FC fabric on an FCoE-FC gateway.

You can rebalance the existing load on existing NP_Port links using either of two methods:

  • Automated load rebalancing—When a load rebalancing trigger occurs, the switch automatically rebalances the link loads by redistributing the sessions across the active NP_Port links. There are three possible load rebalancing triggers:

    1. When you enable automated load rebalancing, the switch checks the load balance on the existing NP_Port links. If the links are already balanced, the switch does not rebalance the link load. If the links are not balanced, the switch rebalances the link loads using the configured load-balancing algorithm.

      Enabling automated load rebalancing causes sessions to be logged out in accordance with the configured load-balancing algorithm if the link load is unbalanced. If the link load is already balanced when you enable automated load rebalancing, the links are not rebalanced. (Disabling automated load rebalancing is not disruptive because the link load is already balanced.)

    2. When a new NP_Port link comes up on a local FCoE-FC gateway fabric, the switch rebalances the link load using the configured load-balancing algorithm if automated load balancing is enabled.

    3. When the port speed is changed (unless the port speed change does not change the actual port speed, for example, changing the port speed from auto to 8 Gbps).

    Use automated load rebalancing if you want link loads to be rebalanced automatically when a load-balancing trigger occurs, instead of at times of your choosing. Keep in mind that load rebalancing is a disruptive event (sessions are logged out).

  • On-demand load rebalancing—You choose when to rebalance the NP_Port links by explicitly requesting a load rebalance using an operational command. The system rebalances the link load only when you issue the rebalancing command.

    Use on-demand load rebalancing if you only want to rebalance the link load once or if you want to rebalance the link loads at controlled times instead of automatically.

    You can also request a load rebalancing dry run. A dry run simulates rebalancing and lists the sessions that might be affected if you choose to perform an actual load-rebalancing operation. The link loads are not rebalanced when you request a dry run.

NP_Port Interface FIP Session Limit Effect on Load Balancing

The maximum number of FIP login sessions configured for each NP_Port interface affects load balancing. When an interface reaches its maximum number of FIP login sessions, that interface is removed from the list of interfaces used for load balancing. The other interfaces in the gateway fabric continue to accept ENode login sessions until they reach their configured maximum session limit. Only interfaces that have not reached their maximum session limit are included in the load-balancing calculations.

Note:

If all NP_Port interfaces in a gateway fabric reach their FIP login session limits, the fabric sends subsequent multicast discovery advertisements (MDAs) with the availability bit set to 0 (zero) to prevent additional ENode login attempts. While the maximum number of sessions is running on the gateway fabric, ENodes cannot use that fabric to log in to the FC switch. When the number of sessions falls below the maximum, the gateway sets the availability bit in MDAs to 1 so that ENodes can log in to the fabric again.

Load-Balancing Triggers and Timing

Several events trigger load balancing. Some of the events trigger load balancing only when automated load balancing is enabled. Other events trigger load rebalancing whether or not automated rebalancing is enabled.

This section describes the load-balancing triggers, what happens when the trigger action occurs, and how the switch determines if and when to balance the link load:

Load-Balancing Triggers

Table 2 describes the four different events can trigger load balancing or load rebalancing. In every case, link load rebalancing uses the configured load-balancing algorithm to determine the placement of sessions on links.

Table 2: Load-Balancing Triggers and Actions

Trigger Event

Action

New link comes up

Triggers a load-rebalancing operation regardless of whether or not automated load rebalancing is enabled. (The new link has no sessions, so the sessions on other links must be redistributed to balance the load.)

The link load is not rebalanced if there are no sessions on the existing links or if there are so few sessions on the existing links that they cannot be redistributed.

On-demand load rebalancing request issued from CLI

The switch checks the NP_Port link load. If the load is not balanced across the links, the switch rebalances the link load. If the load is already balanced, nothing happens.

Note:

Requesting a dry run displays sessions that might be disrupted if you rebalance the link load, but does not rebalance the link load.

Automated load balancing configured for the first time

The switch checks the NP_Port link load. If the load is not balanced across the links, the switch rebalances the link load. If the load is already balanced, nothing happens.

NP_Port speed change

If automated rebalancing is enabled, changing the port speed brings the port up and down (flaps the port) and causes the switch to rebalance the link loads. If the port speed change does not change the actual port speed (for example, changing the port speed from auto to 8 Gbps), the link loads are not rebalanced.

If automated rebalancing is not enabled, port speed changes do not cause link load rebalancing.
Note:

When an NP_Port link goes down, it does not trigger load rebalancing. The loads on the remaining active links are already balanced, and as the sessions logged out from the down link log in again, they are they assigned to links in a balanced manner determined by the configured load-balancing algorithm.

Load-Balancing Timer

When you trigger load balancing from the CLI, the load-balancing action occurs immediately after you execute the command. However, when a load-balancing trigger occurs that is not a CLI command, the switch does not balance the link loads immediately. Instead, the switch follows an intelligent timer process:

  1. The switch checks the current load balance on the NP_Port links in the local gateway FC fabric. If the load is already balanced, the switch does nothing, and there is no session disruption.

  2. If the check shows that the link load is not balanced, the switch starts a 10-second timer. If no other load-balancing triggers occur during the 10-second interval, the switch rebalances the load.

    If another load-balancing trigger occurs during the 10-second interval, the timer resets to 10 seconds. The 10-second timer prevents the switch from performing multiple disruptive load-rebalancing actions in a short period of time.

    Note:

    The switch processes new sessions that log in after the timer starts in the normal manner. The new sessions are considered in the load-balancing evaluation and operation.

  3. At a maximum of 30 seconds after the first load-balancing trigger occurs, the switch checks the link load balance again. If the links are already balanced, the switch cancels the load-rebalancing operation. If the links are not balanced, the switch rebalances the link loads.

Note:

If the trigger event that started the load-rebalancing timer is no longer valid when the timer elapses, the switch cancels the rebalancing operation. For example, if a new NP_Port link comes up and triggers the timer, then goes down before the timer expires, the original link up event is no longer valid, and the switch cancels the rebalancing operation (unless another valid rebalancing trigger occurs in that time frame).

When a link load rebalancing operation is in progress, the switch defers any load-rebalancing triggers that occur until the load-rebalancing operation is complete. The new rebalancing operation begins after the current rebalancing operation finishes if a check shows that rebalancing is required.

If you explicitly request load rebalancing from the CLI using the request fibre-channel proxy load-rebalance operational command, the switch rejects the command and displays an error message stating that rebalancing is already in progress.

Load Rebalancing Behavior When a Link Goes Down

If an NP_Port link goes down, the ENode and VN_Port sessions on that link are logged out. The ENodes and VN_Port sessions log in again and are assigned to NP_Port links based on the link load and the load-balancing algorithm. If a link goes down, the switch does not rebalance the remaining load on the remaining links to avoid disrupting the existing ENode and VN_Port sessions. (Also, it is not necessary to rebalance the links in that manner because after a link goes down, the sessions on the remaining links are already balanced. As the logged out sessions log back in, the switch places them on the remaining active links in a balanced manner, according to the configured load-balancing algorithm.)

Note:

When you use the simple load-balancing algorithm, an ENode and its associated VN_Port sessions might be on different links. In that case, if the NP_Port with the ENode goes down, only the VN_Ports on the same link are logged out. VN_Ports on other links remain up and running.

Interface Load Calculation Algorithm

A weighted round-robin (WRR) algorithm determines the interface load based on:

  • The current number of sessions on the interface

    Note:

    The configured load-balancing algorithm determines how the switch counts the number of sessions. For simple and ENode-based load balancing, the number of sessions is the sum of the FLOGI and FDISC sessions on each link. For FLOGI-based load balancing, the number of sessions is the sum of the FLOGI sessions on each link.

  • The interface weight, which is the speed of the Fibre Channel link (2 Gbps, 4 Gbps, or 8 Gbps)

The interface load algorithm is:

(number-of-sessions * max-weight) / weight

where max-weight is an internal constant.

If the load on the FC interfaces is equal, the session is assigned to the interface with the highest link speed (the greatest weight).

For example, if the three FC interfaces have the characteristics shown in Table 3, the loads of the interfaces are not equal:

Table 3: FC Interface Session-Based Load-Balancing Characteristics for Unequal Loads

Interface

Number of Sessions

Weight (Speed)

fc-0/0/0

4

4 Gbps

fc-0/0/1

1

2 Gbps

fc-0/0/2

8

8 Gbps

In this example, interfaces fc-0/0/0 and fc-0/0/2 have a greater load than fc-0/0/1. For simple load balancing, the gateway assigns the next new FLOGI or FDISC to fc-0/0/1 because it is the least-loaded interface. For both ENode-based and FLOGI-based load balancing, the gateway assigns the next new FLOGI to fc-0/0/1 because it is the least-loaded interface. Then all VN_Port FDISCs from that ENode follow the ENode FLOGI and are also assigned to fc-0/0/1 regardless of the link load.

For another example, if the three FC interfaces have the characteristics shown in Table 4, the loads of the interfaces are equal:

Table 4: FC Interface Session-Based Load-Balancing Characteristics for Equal Loads

Interface

Number of Sessions

Weight (Speed)

fc-0/0/0

4

4 Gbps

fc-0/0/1

2

2 Gbps

fc-0/0/2

8

8 Gbps

In this case, all interfaces have the same relative load. For simple load balancing, the gateway assigns the next new FLOGI or FDISC to fc-0/0/2 because although the loads of the three interfaces are equal, fc-0/0/2 has the greatest weight. For both ENode-based and FLOGI-based load balancing, the gateway assigns the next new FLOGI to fc-0/0/2, and all VN_Port FDISCs from that ENode follow the ENode FLOGI and are also assigned to fc-0/0/2 regardless of the link load.

After the gateway establishes a session between an ENode or a VN_Port and an FC switch on an NP_Port, the session remains on that NP_Port until the ENode or VN_Port performs a LOGO.

If the physical FC interface link goes down, the FLOGI and FDISC sessions on the down link are logged out. The ENodes and VN_Ports log in again to start new sessions on other NP_Ports in the local gateway FC fabric in accordance with the configured load-balancing algorithm (assuming there is more than one NP_Port connected to the FC fabric).

Load-Balancing Scenarios

The configured load-balancing algorithm, the sequence in which ENodes log in to the FC network, the current session count (number of sessions per interface) and the interface speed determine the way the session load is balanced across the native FC interfaces (NP_Ports) in a gateway FC fabric. Whether you are balancing the link load for the first time or rebalancing an existing link load, the way the load is distributed across the active links is the same.

Note:

The way the switch counts the number of sessions on a port depends on the load-balancing algorithm. For simple and ENode-based load balancing, the sum of the FLOGI and FDISC sessions equals the session count. For FLOGI-based load balancing, only the FLOGI sessions are counted in the total session count.

The following scenarios demonstrate how sessions are assigned to links for each load-balancing algorithm:

All of the scenarios use the topology shown in Figure 1.

Figure 1: Sample Load-Balancing TopologySample Load-Balancing Topology

Simple Load-Balancing Algorithm Scenario

Simple load balancing results in the most equal load distribution among the NP_Ports connected to an FC SAN fabric because VN_Port FDISC sessions do not need to “follow” the parent ENode FLOGI session on the same link between the gateway and the FC fabric. When a new FLOGI or FDISC session is initiated, it is assigned to the least-loaded link.

The simple load-balancing algorithm example uses the topology shown in Figure 1 and has the following characteristics:

  • QFX3500 switch configured as an FCoE-FC gateway

  • Two gateway NP_Ports, fc-0/0/0 and fc-0/0/1, connected to an FC SAN fabric switch at a speed of 8 Gbps

  • Three ENodes, ENode_A, ENode_B, and ENode_C connected to the gateway

  • NP_Ports fc-0/0/0 and fc-0/0/1, and ENode_A, ENode_B, and ENode_C, belong to the same local FC fabric on the gateway

When the NP_Ports initialize, they send FLOGI messages to the FC switch and log in to the FC SAN fabric. The gateway then advertises the fabric to the ENodes on the Ethernet side of the network. At this point, the load on both of the NP_Ports is equal. Now the ENodes and VN_Ports start to log in to the fabric:

  1. ENode_A sends a FLOGI to log in to the fabric. Because the loads on the two NP_Ports are equal, the session for ENode_A is randomly placed on one of the links. In this example, the ENode_A FLOGI session is placed on port fc-0/0/0.

  2. Enode_B logs in. Because the load is less on port fc-0/0/1, the Enode_B FLOGI session is placed on port fc-0/0/1.

  3. ENode_C logs in. Because the link loads are equal, the ENode_C login session is randomly placed on one of the links. In this example, the ENode_C login session is placed on port fc-0/0/0.

  4. A VN_Port on ENode_A sends an FDISC to log in to the fabric. Because port fc-0/0/1 currently is the least-loaded link, the VN_Port session is placed on port fc-0/0/1, even though its parent ENode session is on port fc-0/0/0.

  5. As each new VN_Port session comes up, it is placed on the least-loaded link, regardless of the link on which its parent ENode session is placed.

ENode-Based Load-Balancing Algorithm Scenarios

ENode-based load balancing ensures that VN_Port FDISC sessions are placed on the same link as their parent ENode FLOGI sessions, regardless of the link load. ENode-based load balancing can result in a less-balanced load among the NP_Port links, but it provides the control and predictability of keeping ENodes and their VN_Port sessions on the same link.

The examples in this section use the topology shown in Figure 1.

  • QFX3500 switch configured as an FCoE-FC gateway

  • Two gateway NP_Ports, fc-0/0/0 and fc-0/0/1, connected to an FC SAN fabric switch at a speed of 8 Gbps

  • Three ENodes connected to the gateway:

    • ENode_A, which has 2 VN_Port FDISC sessions

    • ENode_B, which has 20 VN_Port FDISC sessions

    • ENode_C, which has 100 VN_Port FDISC sessions

  • NP_Ports fc-0/0/0 and fc-0/0/1, and ENode_A, ENode_B, and ENode_C, belong to the same local FC fabric on the gateway

When the NP_Ports initialize, they send FLOGI messages to the FC switch and log in to the FC SAN fabric. The gateway then advertises the fabric to the ENodes on the Ethernet side of the network. At this point, the load on both of the NP_Ports is equal. Now the ENodes and VN_Ports start to log in to the fabric. As the following two scenarios show, how these sessions are placed on the links depends on the sequence in which they log in to the fabric.

Scenario 1:

  1. ENode_A sends a FLOGI to log in to the fabric. Because the loads on the two NP_Ports are equal, the session for ENode_A is randomly placed on one of the links. In this example, the ENode_A FLOGI session is placed on port fc-0/0/0.

  2. ENode_B logs in. Because the load is less on port fc-0/0/1, the ENode_B FLOGI session is placed on port fc-0/0/1.

  3. The two VN_Ports on ENode_A log in to the fabric. Their sessions are placed on port fc-0/0/0, following ENode_A on the link. Now port fc-0/0/0 has a greater load (one FLOGI session plus two FDISC sessions) than port fc-0/0/1 (one FLOGI session).

  4. The 20 VN_Ports on ENode_B log in to the fabric. Their sessions are placed on port fc-0/0/1, following ENode_B on the link. Now port fc-0/0/0 has a lesser load (one FLOGI, two FDISC) than port fc-0/0/1.

  5. ENode_C logs in. Because the load is less on port fc-0/0/0, the ENode_C FLOGI session is placed on port fc-0/0/0.

  6. The 100 VN_Ports on ENode_C log in to the fabric. Their sessions follow the ENode_C session onto port fc-0/0/0.

  7. If more VN_Ports come up, their FDISC sessions are placed on the same link as the corresponding parent ENode session.

Scenario 2:

  1. ENode_A sends a FLOGI to log in to the fabric. Because the loads on the two NP_Ports are equal, the session for ENode_A is randomly placed on one of the links. In this example, the ENode_A FLOGI session is placed on port fc-0/0/0.

  2. ENode_B logs in. Because the load is less on port fc-0/0/1, the ENode_B FLOGI session is placed on port fc-0/0/1.

  3. The two VN_Ports on ENode_A log in to the fabric. Their sessions are placed on port fc-0/0/0, following ENode_A on the link. Now port fc-0/0/0 has a greater load (one FLOGI session plus two FDISC sessions) than port fc-0/0/1 (one FLOGI session).

  4. In this step, the login sequence in Scenario 2 differs from the login sequence in Scenario 1, resulting in a different placement of sessions on the links, and therefore a different load on the links. ENode_C logs in before the ENode_B VN_Ports log in, which changes the session count on the links compared to the first scenario. Because the load in this scenario is less on port fc-0/0/1, the ENode_C FLOGI session is placed on port fc-0/0/1 (instead of port fc-0/0/0 as in the first scenario).

  5. The 20 VN_Ports on ENode_B log in to the fabric. Their sessions are placed on port fc-0/0/1, following ENode_B on the link. Now port fc-0/0/0 carries one FLOGI and two FDISC sessions, and port fc-0/0/1 carries two FLOGI and 20 FDISC sessions.

  6. The 100 VN_Ports on ENode_C log in to the fabric. Their sessions follow the ENode_C session onto port fc-0/0/1. Now port fc-0/0/1 carries 2 FLOGI and 120 FDISC sessions, whereas port fc-0/0/0 carries one FLOGI and two FDISC sessions.

  7. If more VN_Ports come up, their FDISC sessions are placed on the same link as the corresponding parent ENode session.

Because of the sequence of ENode logins in Scenario 2, port fc-0/0/1 carries a greater load than port fc-0/0/0. If the simple load-balancing algorithm had been used, the FLOGI and FDISC sessions would be allocated to the two links evenly. However, because the FDISC sessions are placed on the same link as their parent FLOGI sessions, this example demonstrates how using the ENode-based load-balancing algorithm can lead to scenarios in which the link loads are not equal.

FLOGI-Based Load-Balancing Algorithm Scenarios

FLOGI-based load balancing is similar in many ways to ENode-based load balancing. An important difference that affects how the switch places sessions on links is that for FLOGI-based load balancing, only the FLOGI sessions are counted when the link load is calculated. FDISC sessions are not counted to determine the link load. Because ENode-based load balancing uses the sum of the FLOGI and FDISC sessions to determine the link load, an interface with exactly the same combination of FLOGI and FDISC sessions can have a different session count depending on the algorithm used. A different session count can change the interface to which the switch assigns the next session.

As with ENode-based load balancing, FLOGI-based load balancing ensures that VN_Port FDISC sessions are placed on the same link as their parent ENode FLOGI sessions, regardless of the link load. FLOGI-based load balancing can result in a less-balanced load among the NP_Port links, but it provides the control and predictability of keeping ENodes and their VN_Port sessions on the same link.

The examples in this section use the topology shown in Figure 1.

  • QFX3500 switch configured as an FCoE-FC gateway

  • Two gateway NP_Ports, fc-0/0/0 and fc-0/0/1, connected to an FC SAN fabric switch at a speed of 8 Gbps

  • Three ENodes connected to the gateway:

    • ENode_A, which has 2 VN_Port FDISC sessions

    • ENode_B, which has 20 VN_Port FDISC sessions

    • ENode_C, which has 100 VN_Port FDISC sessions

  • NP_Ports fc-0/0/0 and fc-0/0/1, and ENode_A, ENode_B, and ENode_C, belong to the same local FC fabric on the gateway

When the NP_Ports initialize, they send FLOGI messages to the FC switch and log in to the FC SAN fabric. The gateway then advertises the fabric to the ENodes on the Ethernet side of the network. At this point, the load on both of the NP_Ports is equal. Now the ENodes and VN_Ports start to log in to the fabric.

Because FLOGI-based load balancing does not count FDISC sessions when calculating the link load, how the sessions are placed on the link depends only on the number of FLOGI sessions per interface, not on the number of FLOGI sessions plus FDISC sessions. This means that an ENode with a FLOGI session and many FDISC sessions is counted as having the same load as an ENode with a FLOGI session and no FDISC sessions.

Scenario 1:

  1. ENode_A sends a FLOGI to log in to the fabric. Because the loads on the two NP_Ports are equal, the session for ENode_A is randomly placed on one of the links. In this example, the ENode_A FLOGI session is placed on port fc-0/0/0.

  2. ENode_B logs in. Because the load is less on port fc-0/0/1, the ENode_B FLOGI session is placed on port fc-0/0/1.

  3. The two VN_Ports on ENode_A log in to the fabric. Their sessions are placed on port fc-0/0/0, following ENode_A on the link. However, unlike simple load balancing or ENode-based load balancing, the session count of the two ports is still equal (one session each) because the FDISC sessions are not used in the session count.

  4. The 20 VN_Ports on ENode_B log in to the fabric. Their sessions are placed on port fc-0/0/1, following ENode_B on the link. Again, unlike simple load balancing or ENode-based load balancing, the session count of the two ports is still equal (one session each) because the FDISC sessions are not used in the session count.

  5. ENode_C logs in. Because the link loads are counted as equal, the ENode_C login session is randomly placed on one of the links. In this example, the ENode_C login session is placed on port fc-0/0/0.

  6. The 100 VN_Ports on ENode_C log in to the fabric. Their sessions follow the ENode_C session onto port fc-0/0/0.

  7. If more VN_Ports come up, their FDISC sessions are placed on the same link as the corresponding parent ENode session.

If a fourth ENode, ENode_D, sends a FLOGI to log in to the fabric, it is placed on port fc-0/0/1 because port fc-0/0/0 has a session count of two (two FLOGIs from ENode_A and ENode_C, FDISCs not counted) and port fc-0/0/1 has a session count of one (one FLOGI from ENode_B, FDISCs not counted), so port fc-0/0/1 is the least-loaded port.

With FLOGI-based load balancing, it is possible for ENodes with many FDISC sessions to be placed on the same link, whereas ENodes with few FDISC sessions are placed on different links because only FLOGIs are used in the session count.

Related Documentation