Rate and give feedback:  Feedback Received. Thank You!

Rate and give feedback: 

X

This document helped resolve my issue

Yes No

Additional Comments

800 characters remaining

May we contact you if necessary?

Name:  

E-mail: 

Enter a valid Email ID

Need product assistance? Contact Juniper Support

Submit

This site is protected by hCaptcha and its Privacy Policy and Terms of Service apply.

English  

English

Understanding CoS Fabric Forwarding Class Sets

Default Fabric Forwarding Class Sets

Interconnect devices have 12 default fabric fc-sets, including five visible default fabric fc-sets, four for unicast traffic and one for multidestination (multicast, broadcast, and destination lookup failure) traffic.

There are also seven hidden default fabric fc-sets. There are three hidden default fabric fc-sets for multidestination traffic that you can use if you want to map different multidestination forwarding classes to different multidestination fabric fc-sets. There are four hidden default fabric fc-sets for lossless traffic that you can use to map different lossless forwarding classes (priorities) to different lossless fabric fc-sets. Table 1 shows the default fabric fc-sets:

The five default forwarding classes (best-effort, fcoe, no-loss, network-control, and mcast) are mapped to the fabric fc-sets by default as shown in Table 2.

The maximum fiber cable length between the QFabric system Node device and the QFabric system Interconnect device is 150 meters.

Tip: If you explicitly configure lossless forwarding classes, we recommend that you map each user-configured lossless forwarding class to an unused fabric fc-set (fabric_fcset_noloss3 through fabric_fcset_noloss6) on a one-to-one basis: one lossless forwarding class mapped to one lossless fabric fc-set. The reason for one-to-one mapping is to avoid fate sharing of lossless flows. Because each fabric fc-set is mapped statically to an output queue, when you map more than one forwarding class to a fabric fc-set, all of the traffic in all of the forwarding classes that belong to the fabric fc-set uses the same output queue. If that output queue becomes congested due to congestion caused by one of the flows, the other flows are also affected. (They share fate because the flow that congests the output queue affects flows that are not experiencing congestion.) However, it is important to understand that fabric_fcset_noloss1 and fabric_fcset_noloss2 have a scheduling weight of 35, while the other fabric fc-sets have a scheduling weight of 1. The scheduling weights mean that fabric_fcset_noloss1 and fabric_fcset_noloss2 receive most of the bandwidth available to lossless fabric fc-sets if the amount of traffic on fabric_fcset_noloss1 and fabric_fcset_noloss2 requires the bandwidth. If you believe that the traffic on fabric_fcset_noloss1 and fabric_fcset_noloss2 will consume most of that bandwidth, then you should place all lossless traffic on fabric_fcset_noloss1 and fabric_fcset_noloss2. If you believe that the traffic on fabric_fcset_noloss1 and fabric_fcset_noloss2 will <emphasis>not</emphasis> consume most of that bandwidth, then you can map lossless forwarding classes in a one-to-one manner to lossless fabric fc-sets to avoid fate sharing.

If you want to map different multidestination forwarding classes to different multidestination fabric fc-sets, use one or more of the hidden multidestination fabric fc-sets.

Note: The global mapping of forwarding classes to fabric fc-sets is independent of the mapping of forwarding classes to Node device fc-sets. Global mapping of forwarding classes to fabric fc-sets occurs only on the Interconnect device. The Node device mapping of forwarding classes to fc-sets does not affect the global mapping of forwarding classes to fabric fc-sets on the Interconnect device, and vice versa.

When you define new forwarding classes on a Node device, you explicitly map those forwarding classes to Node device fc-sets. However, new (user-created) forwarding classes are mapped by default to fabric fc-sets. (You can override the default mapping if you want to configure the forwarding class to fabric fc-set mapping explicitly, as described in the next section.)

By default:

• All best-effort traffic forwarding classes that you create are mapped to the fabric_fcset_be fabric fc-set.
• All lossless traffic forwarding classes that you create are mapped to the fabric_fcset_noloss1 or fabric_fcset_noloss2 fabric fc-set.
• All multidestination traffic forwarding classes that you create are mapped to the fabric_fcset_multicast1 fabric fc-set.
• All strict-high priority traffic and network-control forwarding classes that you create are mapped to the fabric_fcset_strict_high fabric fc-set.

Fabric Forwarding Class Set Configuration and Implementation

You can map forwarding classes to fabric fc-sets, but no other attributes of fabric fc-sets are user-configurable, including CoS. This section describes:

• Mapping Forwarding Classes to Fabric Forwarding Class Sets
• Fabric Forwarding Class Set Implementation

Mapping Forwarding Classes to Fabric Forwarding Class Sets

If you do not want to use the default mapping of forwarding classes to fabric fc-sets, you can map forwarding classes to fabric fc-sets the same way as you map forwarding classes to Node device fc-sets. To do this, use exactly the same statement that you use to map forwarding classes to fc-sets, but instead of specifying a Node device fc-set name, specify a fabric fc-set name.

Note: The global mapping of forwarding classes to fabric fc-sets does not affect the mapping of forwarding classes to Node device fc-sets. The global forwarding class mapping to fabric fc-sets pertains to the traffic only when it enters, traverses, and exits the fabric. The forwarding class mapping to fc-sets on a Node device is valid within that Node device. Mapping forwarding classes to fabric fc-sets does not affect the scheduling configuration of the forwarding classes or fc-sets on Node devices. Fabric fc-set scheduling (which is not user-configurable) pertains to traffic only when it enters, traverses, and exits the Interconnect device fabric. If you change the mapping of a forwarding class to a fabric fc-set, the new mapping is global and applies to all traffic in that forwarding class, regardless of which Node device forwards the traffic to the Interconnect device.

• To assign one or more forwarding classes to a fabric fc-set:
  [edit class-of-service]
user@switch# set forwarding-class-sets fabric-forwarding-class-set-name class forwarding-class-name

  For example, to map a user-defined forwarding class named best-effort-2 to the fabric fc-set fabric_fcset_be:

  [edit class-of-service]
user@switch# set forwarding-class-sets fabric_fcset_be class best-effort-2

Note: Because fabric fc-set configuration is global, in this example all forwarding classes with the name best-effort-2 on all of the Node devices attached to the fabric use the fabric_fcset_be fabric fc-set to transport traffic across the fabric.

Fabric Forwarding Class Set Implementation

The following rules apply to fabric fc-sets:

• You cannot create new fabric fc-sets. Only the twelve default fabric fc-sets are available.
• You cannot delete a default fabric fc-set.
• You cannot attach a fabric fc-set to a Node device interface. Fabric fc-sets are used only on the Interconnect device fabric, not on Node devices.
• You can map only multidestination forwarding classes to multidestination fabric fc-sets.
• You cannot map multidestination forwarding classes to unicast fabric fc-sets.
• You cannot map unicast forwarding classes to multidestination fabric fc-sets.
• You cannot configure CoS for fabric fc-sets. (However, default CoS scheduling properties are applied to traffic on the fabric, and the fabric interfaces use link layer flow control (LLFC) for flow control.)

Fabric Forwarding Class Set Scheduling (CoS)

Although fabric fc-set CoS is not user-configurable, CoS is applied to traffic on the fabric. (In addition, fabric interfaces use LLFC to ensure lossless transport for lossless traffic flows.) This section describes how the fabric applies CoS scheduling to traffic:

• Class Groups for Fabric Forwarding Class Sets
• Class Group Scheduling
• QFabric System CoS

Class Groups for Fabric Forwarding Class Sets

To transport traffic across the fabric, the QFabric system organizes the fabric fc-sets into three classes called class groups. The three class groups are:

• Strict-high priority—All traffic in the fabric fc-set fabric_fcset_strict_high. This class group includes the traffic in strict-high priority and network-control forwarding classes and in any forwarding classes you create on a Node device that consist of strict-high priority or network-control forwarding class traffic.
• Unicast—All traffic in the fabric fc-sets fabric_fcset_be, fabric_fcset_noloss1, and fabric_fcset_noloss2. This class group includes the traffic in the best-effort, fcoe, and no-loss forwarding classes and in any forwarding classes you create on a Node device that consist of best-effort or lossless traffic. If you use any of the hidden no loss fabric fc-sets (fabric_fcset_noloss3, fabric_fcset_noloss4, fabric_fcset_noloss5, or fabric_fcset_noloss6), that traffic is part of this class group.
• Multidestination—All traffic in the fabric fc-set fabric_fcset_multicast1. This class group includes the traffic in the mcast forwarding class and in any forwarding classes you create on a Node device that consist of multidestination traffic. If you use any of the hidden multidestination fabric fc-sets (fabric_fcset_multicast2, fabric_fcset_multicast3, or fabric_fcset_multicast4), that traffic is also classified as part of this class group.

Class Group Scheduling

You cannot configure CoS for class groups or for fabric fc-sets (that is, you cannot attach a traffic control profile to a fabric fc-set—you attach traffic control profiles to Node device fc-sets to apply scheduling to the traffic that belongs to the Node device fc-set). By default, the fabric uses weighted round-robin (WRR) scheduling in which each class group receives a portion of the total available fabric bandwidth based on its type of traffic, as shown in Table 3:

The fabric fc-sets within each class group are weighted equally and receive bandwidth using round-robin scheduling. For example:

• If the unicast class group has three member fabric fc-sets, fabric_fcset_be, fabric_fcset_noloss1, and fabric_fcset_noloss2, then each of the three fabric fc-sets receives one-third of the bandwidth available to the unicast class group.
• If the multidestination class group has one member fc-set, fabric_fcset_multicast1, then that fc-set receives all of the bandwidth available to the multidestination class group.
• If the multidestination class group has two member fc-sets, fabric_fcset_multicast1 and fabric_fcset_multicast2, then each of the two fabric fc-sets receives one-half of the bandwidth available to the multidestination class group.

QFabric System CoS

When traffic enters and exits the same Node device, CoS works the same as it works on a standalone QFX3500 switch.

However, when traffic enters a Node device, crosses the Interconnect device, and then exits a different Node device, CoS is applied differently:

1. Traffic entering the ingress Node device receives the CoS configured at the Node ingress (packet classification, congestion notification profile for PFC).
2. When traffic goes from the ingress Node device to the Interconnect device, the fabric fc-set CoS is applied as described in the discussion of fabric forwarding class set scheduling.
3. When traffic goes from the Interconnect device to the egress Node device, the egress Node device applies CoS at the egress port (egress queue scheduling, WRED, IEEE 802.1p or DSCP code-point rewrite).

Support for Flow Control and Lossless Transport Across the Fabric

The Interconnect device incorporates flow control mechanisms to support lossless transport during periods of congestion on the fabric. To support the priority-based flow control (PFC) feature on the Node devices, the fabric interfaces use LLFC to support lossless transport for up to six IEEE 802.1p priorities when the following two configuration constraints are met:

1. The IEEE 802.1p priority used for the traffic that requires lossless transport is mapped to a lossless forwarding class on the Node devices.
2. The lossless forwarding class must be mapped to a lossless fabric fc-set on the Interconnect device (fabric_fcset_noloss1, fabric_fcset_noloss2, fabric_fcset_noloss3, fabric_fcset_noloss4, fabric_fcset_noloss5, or fabric_fcset_noloss6).

When traffic meets the two configuration constraints, the fabric propagates the back pressure from the egress Node device across the fabric to the ingress Node device during periods of congestion. However, to achieve end-to-end lossless transport across the switch, you must also configure a congestion notification profile to enable PFC on the Node device ingress ports.

For all other combinations of IEEE 802.1p priority to forwarding class mapping and all other combinations of forwarding class to fabric fc-set mapping, the congestion control mechanism is normal packet drop. For example:

• Case 1—If the IEEE 802.1p priority 5 is mapped to the lossless fcoe forwarding class, and the fcoe forwarding class is mapped to the fabric_fcset_noloss1 fabric fc-set, then the congestion control mechanism is PFC.
• Case 2—If the IEEE 802.1p priority 5 is mapped to the lossless fcoe forwarding class, and the fcoe forwarding class is mapped to the fabric_fcset_be fabric fc-set, then the congestion control mechanism is packet drop.
• Case 3—If the IEEE 802.1p priority 5 is mapped to the lossless no-loss forwarding class, and the no-loss forwarding class is mapped to the fabric_fcset_noloss2 fabric fc-set, then the congestion control mechanism is PFC.
• Case 4—If the IEEE 802.1p priority 5 is mapped to the lossless no-loss forwarding class, and the no-loss forwarding class is mapped to the fabric_fcset_be fabric fc-set, then the congestion control mechanism is packet drop.
• Case 5—If the IEEE 802.1p priority 5 is mapped to the best-effort forwarding class, and the best-effort forwarding class is mapped to the fabric_fcset_be fabric fc-set, then the congestion control mechanism is packet drop.
• Case 6—If the IEEE 802.1p priority 5 is mapped to the best-effort forwarding class, and the best-effort forwarding class is mapped to the fabric_fcset_noloss1 fabric fc-set, then the congestion control mechanism is packet drop.

Note: Lossless transport across the fabric also must meet the following two conditions: The maximum cable length between the Node device and the Interconnect device is a 150 meters of fiber cable. The maximum frame size is 9216 bytes. If the MTU is 9216 KB, in some cases the QFabric system supports only five lossless forwarding classes instead of six lossless forwarding classes because of headroom buffer limitations.

The number of IEEE 802.1p priorities (forwarding classes) the QFabric system can support for lossless transport across the Interconnect device fabric depends on several factors:

• Approximate fiber cable length—The longer the fiber cable that connects Node device fabric (FTE) ports to the Interconnect device fabric ports, the more data the connected ports need to buffer when a pause is asserted. (The longer the fiber cable, the more frames are traversing the cable when a pause is asserted. Each port must be able to store all of the “in transit” frames in the buffer to preserve lossless behavior and avoid dropping frames.)
• MTU size—The larger the maximum frame sizes the buffer must hold, the fewer frames the buffer can hold. The larger the MTU size, the more buffer space each frame consumes.
• Total number of Node device fabric ports connected to the Interconnect device—The higher the number of connected fabric ports, the more headroom buffer space the Node device needs on those fabric ports to support the lossless flows that traverse the Interconnect device. Because more buffer space is used on the Node device fabric ports, less buffer space is available for the Node device access ports, and a lower total number of lossless flows are supported.

The QFabric system supports six lossless priorities (forwarding classes) under most conditions. The priority group headroom that remains after allocating headroom to lossless flows is sufficient to support best-effort and multidestination traffic.

Table 4 shows how many lossless priorities the QFabric system supports under different conditions (fiber cable lengths and MTUs) in cases when the QFabric system supports fewer than six lossless priorities. The number of lossless priorities is the same regardless of how many Node device FTE ports are connected to the Interconnect device. However, the higher the number of FTE ports connected to the Interconnect device, the lower the number of total lossless flows supported. In all cases that are not shown in Table 4, the QFabric system supports six lossless priorities.

Note: The system does not perform a configuration commit check that compares available system resources with the number of lossless forwarding classes configured. If you commit a configuration with more lossless forwarding classes than the system resources can support, frames in lossless forwarding classes might be dropped.

Note: The total number of lossless flows decreases as resource consumption increases. For a Node device, the higher the number of FTE ports connected to the Interconnect device, the larger the MTU, and the longer the fiber cable length, the fewer total lossless flows the QFabric system can support.

Viewing Fabric Forwarding Class Set Information

You can display information about fabric fc-sets using the same CLI command you use to display information about Node device fc-sets:

Forwarding class set: fabric_fcset_be, Type: fabric-type, Forwarding class set index: 1
  Forwarding class                       Index
  best-effort                              0               

Forwarding class set: fabric_fcset_mcast1, Type: fabric-type, Forwarding class set index: 5
  Forwarding class                       Index
  mcast                                    8               

Forwarding class set: fabric_fcset_mcast2, Type: fabric-type, Forwarding class set index: 6

Forwarding class set: fabric_fcset_mcast3, Type: fabric-type, Forwarding class set index: 7

Forwarding class set: fabric_fcset_mcast4, Type: fabric-type, Forwarding class set index: 8

Forwarding class set: fabric_fcset_noloss1, Type: fabric-type, Forwarding class set index: 2
  Forwarding class                       Index
  fcoe                                     1               
                                        
Forwarding class set: fabric_fcset_noloss2, Type: fabric-type, Forwarding class set index: 3
  Forwarding class                       Index
  no-loss                                  2               

Forwarding class set: fabric_fcset_noloss3, Type: fabric-type, Forwarding class set index: 9

Forwarding class set: fabric_fcset_noloss4, Type: fabric-type, Forwarding class set index: 10

Forwarding class set: fabric_fcset_noloss5, Type: fabric-type, Forwarding class set index: 11

Forwarding class set: fabric_fcset_noloss6, Type: fabric-type, Forwarding class set index: 12

Forwarding class set: fabric_fcset_strict_high, Type: fabric-type, Forwarding class set index: 4
  Forwarding class                       Index
  network-control                          3

Table 5 describes the meaning of the show class-of-service forwarding-class-set output fields when you display fabric fc-set information.

Summary of Fabric Forwarding Class Set and Node Device Forwarding Class Set Differences

Table 6 summarizes the differences between fabric fc-sets and Node device fc-sets: