Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

Navigation

Understanding CoS Buffer Configuration

The QFX3500 switch has 9 MB of packet forwarding engine (PFE) wide common packet buffer memory that is used to store packets as the packets traverse the switch. The buffer memory has separate ingress and egress accounting to make accept, drop, or pause decisions. Because the switch has a single pool of memory with separate ingress and egress accounting, the full amount of buffer memory is available from both ingress and egress perspective. Packets are accounted for as they enter and leave the switch, but there is no concept of a packet arriving at an ingress buffer and then being moved to an egress buffer.

The buffers are divided into two pools from both an ingress and an egress perspective:

  1. Shared buffers are a global memory pool that the switch allocates dynamically to ports as needed, so the buffers are shared among the switch ports. The global buffer is further partitioned into buffers for best-effort unicast and multidestination (broadcast, multicast, and destination lookup fail), and PFC (lossless) traffic types.
  2. Dedicated buffers are a memory pool divided equally among the switch ports. Each port receives a minimum guaranteed amount of buffer space, dedicated to each port, not shared among ports. You can divide the dedicated buffer allocation for a port among the port queues on a per-port, per-queue basis. (For example, this enables you to dedicate more buffer space to queues that transport lossless traffic.)

The switch reserves nonconfigurable buffer space to ensure that ports and queues receive a minimum allocation. You can configure how the system uses the rest of the buffer space to optimize the allocation for your mix of network traffic. You can configure the percentage of available buffer space used as shared buffer space versus dedicated buffer space. You can also configure how shared buffer space is allocated to different types of traffic. You can optimize the buffer settings for the traffic on your network.

The default buffer configuration is designed for networks that have a balance of best-effort and lossless traffic.

Note: Lossless traffic is traffic on which you enable priority-based flow control (PFC) to ensure lossless transport. Lossless traffic does not refer to best-effort traffic on a link enabled for Ethernet PAUSE (IEEE 802.3x).

The default class-of-service configuration provides two lossless queues (fcoe and no-loss), a best-effort unicast queue, a network control traffic queue, and one multidestination—multicast, broadcast, and destination lookup fail—queue. The default configuration allocates the buffers in a manner that supports a moderate amount of lossless traffic while still providing the ability to absorb bursts in best-effort traffic transmission.

Changing the buffer settings changes the abilities of the buffers to absorb traffic bursts and handle lossless traffic. For example, networks with mostly best-effort traffic require allocating most of the shared buffer space to best-effort buffers. This provides deep, flexible buffers that can absorb traffic bursts with minimal packet loss, at the expense of buffer availability for lossless traffic.

Conversely, networks with mostly lossless traffic require allocating most of the shared buffer space to lossless headroom buffers. This prevents packet loss on lossless flows at the expense of absorbing bursty best-effort traffic efficiently.

Caution: Changing the buffer configuration is a disruptive event. Traffic stops on all ports until buffer reprogramming is complete.

This topic describes the buffer architecture and settings:

Buffer Pools

Both from an ingress and an egress perspective, the 9 MB buffer is split into two main pools:

  • Shared buffer pool—A global memory space that all of the ports on the switch share dynamically as they need buffers. You can allocate global shared memory space to different buffer partitions to better support different mixes of network traffic. The larger the shared buffer pool, the better the switch can absorb traffic bursts because more shared memory is available for the traffic.
  • Dedicated buffer pool—A reserved global memory space allocated equally to each port. The switch reserves a minimum dedicated buffer pool that is not user-configurable. On each egress port, you can configure the amount of the port dedicated buffer that is allocated to each of the port queues. A larger dedicated buffer pool means a larger amount of dedicated buffer space for each port, so congestion on one port is less likely to affect traffic on another port because the traffic does not need to use as much shared buffer space. However, the larger the dedicated buffer pool, the less bursty traffic the switch can handle because there is less dynamic shared buffer memory.

A portion of the total 9 MB of buffer space is reserved so that there is always a minimum amount of shared and dedicated buffer space available to each port. You can configure the way the rest of the buffer space is allocated to the global shared buffer pool and the dedicated shared buffer pool by configuring the ingress and egress shared buffer percentages. The buffer space that is not reserved is the available buffer space.

By default, 100 percent of the remaining available buffer space is allocated to the shared buffer pool. If you change the percentage, the available buffer space that is not allocated to the shared buffer is allocated to the dedicated buffer. For example, if you configure the ingress shared buffer pool as 80 percent, the remaining 20 percent of the available buffer space is allocated to the dedicated buffer pool and divided equally across the ports.

Note: If 100 percent of the available (user-configurable) buffers are allocated to the shared buffer pool, the switch still has a reserved minimum dedicated buffer pool that is necessary for proper operation.

You can separately configure ingress and egress shared buffer pool allocations. You can also partition the ingress and egress shared buffer pool to allocate percentages of the shared buffer pool to specific types of traffic. If you do not use the default configuration or one of the recommended configurations, pay particular attention to the ingress configuration of the lossless and lossless headroom buffers (to handle PFC pause during periods of congestion) and to the egress configuration of the best-effort buffers to handle incast congestion.

In addition to the shared buffer pool and the dedicated buffer pool, there is also a small ingress global headroom buffer pool that is reserved and is not configurable.

When contention for buffer space occurs, the switch uses an internal algorithm to ensure that the buffer pools are distributed fairly among competing flows. When traffic for a given flow exceeds the amount of dedicated buffer reserved for that flow, the flow begins to consume memory from the dynamic shared buffer pool. The flow is a competing flow because it competes with other flows that have exhausted their dedicated buffers for shared buffer memory. When there is no congestion, there are no competing flows.

Buffer Handling of Lossless Flows (PFC) Versus Ethernet PAUSE

When we discuss lossless buffers in the following sections, we mean buffers that handle traffic on which you enable PFC to ensure lossless transport. The lossless buffers are not used for best-effort traffic on a link on which you enable Ethernet PAUSE (IEEE 802.3x). The lossless ingress and egress shared buffers, and the ingress lossless headroom shared buffer, are used only for traffic on which you enable PFC.

Note: To support lossless flows, you must configure the appropriate data center bridging capabilities (PFC, DCBX, or ETS) and scheduling.

Shared Buffer Pool

The shared buffer pool is a global memory space that all of the ports on the switch share dynamically as they need buffers. The switch uses the shared buffer pool to absorb traffic bursts after the dedicated buffer pool for a port is exhausted.

You can divide both the ingress shared buffer pool and the egress shared buffer pool into three partitions to allocate percentages of the buffer pool to different types of traffic. When you partition the ingress or egress shared buffer:

  • If you explicitly configure one ingress shared buffer partition, you must explicitly configure all three ingress shared buffer partitions.
  • The total percentages of the three ingress shared buffer partitions must total exactly 100 percent.
  • If you explicitly configure one egress shared buffer partition, you must explicitly configure all three egress shared buffer partitions.
  • The total percentages of the three egress shared buffer partitions must total exactly 100 percent.
  • If you explicitly partition one set of shared buffers, you do not have to explicitly partition the other set of shared buffers. For example, you can explicitly configure ingress shared buffer partitions and use the default egress shared buffer partitions (no explicit egress shared buffer partition configuration). However, if you change the buffer partitions for the ingress buffer pool to match the expected types of traffic flows, you probably would also want to change the buffer partitions for the egress buffer pool to match those traffic flows.

The switch returns a commit error if you do not explicitly configure all three partitions when configuring the ingress or egress shared buffer partitions.

When you explicitly configure the shared buffer partitions, the switch returns a commit error if the total percentage of the three partitions does not equal 100 percent.

You can configure the percentage of available (user-configurable) buffer space allocated to the global ingress and egress shared buffers. Any space that you do not allocate to the global shared buffer pool is added to the dedicated buffer pool. The default configuration allocates 100 percent of the available ingress and egress buffer space to the global shared buffers.

You can partition the ingress and egress shared buffer pools to allocate more buffers to the types of traffic your network predominantly carries, and fewer buffers to other traffic. From the buffer space allocated to the ingress shared buffer pool, you can allocate space to:

  • Lossless buffers—Percentage of shared buffer pool for all lossless ingress traffic. The recommended minimum value for lossless buffers is 5 percent.
  • Lossless headroom buffers—Percentage of shared buffer pool for packets received while a pause is asserted. If PFC is configured on priorities on a port, when the port sends a pause message to the connected peer, the port uses the headroom buffers to store the packets that arrive between the time the port sends the pause message and the time the last packet arrives after the peer pauses traffic. The minimum value for lossless headroom buffers is 0 (zero) percent. (Lossless headroom buffers are the only buffers for which the recommended value can be less than 5 percent.)
  • Lossy buffers—Percentage of shared buffer pool for all best-effort ingress traffic (best-effort unicast, multidestination, and strict-high priority traffic). The recommended minimum value for best-effort buffers is 5 percent.

The combined percentage values of the ingress lossless, lossless headroom, and best-effort buffer partitions must total exactly 100 percent. If the buffer percentages total more than 100 percent or less than 100 percent, the switch returns a commit error. All ingress buffer partitions must be explicitly configured, even when the lossless headroom buffer partition has a value of 0 (zero) percent.

From the buffer space allocated to the egress shared buffer pool, you can allocate space to:

  • Lossless buffers—Percentage of shared buffer pool for all lossless egress queues. The recommended minimum value for lossless buffers is 5 percent.
  • Lossy buffers—Percentage of shared buffer pool for all best-effort egress queues (best-effort unicast, and strict-high priority queues). The recommended minimum value for best-effort buffers is 5 percent.
  • Multicast buffers—Percentage of shared buffer pool for all multidestination (multicast, broadcast, and destination lookup fail) egress queues. The recommended minimum value for multicast buffers is 5 percent.

The combined percentage values of the egress lossless, lossy , and multicast buffer partitions must total exactly 100 percent. If the buffer percentages total more than 100 percent or less than 100 percent, the switch returns a commit error. All egress buffer partitions must be explicitly configured and must have a value of at least 5 percent.

Dedicated Buffer Pool

The global dedicated buffer pool is memory that is allocated equally to each port, so each port receives a guaranteed minimum amount of buffer space. Dedicated buffers are not shared among ports. Each port receives an equal proportion of the total dedicated buffer pool. The amount of dedicated buffers per port is not user-configurable and depends on the total size of the dedicated buffer pool.

When traffic enters and exits the switch, the switch ports first use their dedicated buffer space to store packets. If the dedicated buffers are not sufficient to handle the traffic, the switch uses shared buffers. The only way to increase the dedicated buffer pool is to decrease the shared buffer pool from its default value of 100 percent of available buffers.

Note: If 100 percent of the available (user-configurable) buffers are allocated to the shared buffer pool, the switch still has a reserved minimum dedicated buffer pool.

A smaller dedicated buffer pool means a larger shared buffer pool, and better burst absorption across the ports. A larger dedicated buffer pool means a larger amount of dedicated buffer space for each port, so congestion on one port is less likely to affect traffic on another port because the traffic does not need to use as much shared buffer space.

You can divide the dedicated buffer allocation for an egress port among the port queues by including the buffer-size statement in the scheduler configuration. This enables you to control the egress dedicated buffer allocation on a per-port, per-queue basis. (For example, this enables you to dedicate more buffer space to queues that transport lossless traffic, or to stop the port from reserving buffers for queues that do not carry traffic.) Egress dedicated buffer allocation is a hierarchical structure that allocates a global dedicated buffer pool evenly among ports, and then divides the allocation for each port among the port queues.

By default, ports divide their allocation of dedicated buffers among the egress queues in the same proportion as the default scheduler sets the minimum guaranteed transmission rates (transmit-rate) for traffic. Only the queues included in the default scheduler receive bandwidth and dedicated buffers, in the proportions shown in Table 1:

Table 1: Default Dedicated Buffer Allocation to Egress Queues (Based on Default Scheduler)

Forwarding Class

Queue

Minimum Guaranteed Bandwidth

Proportion of Reserved Dedicated Buffers

best-effort

0

5%

5%

fcoe

3

35%

35%

no-loss

4

35%

35%

network-control

7

5%

5%

mcast

8

20%

20%

In the default configuration, no other egress queues than the ones shown in Table 1 receive an allocation of dedicated port buffers.

Note: The switch uses hierarchical scheduling to control port and queue bandwidth allocation, as described in Understanding CoS Hierarchical Port Scheduling (ETS) and shown in Example: Configuring CoS Hierarchical Port Scheduling (ETS). For egress queue buffer size configuration, when you associate a traffic control profile (includes the queue scheduler information) with a port, the dedicated egress buffers on the port are divided among the queues as configured in the scheduler.

If you do not want to use the default allocation of dedicated port buffers to queues, you can configure the dedicated buffer allocation for each egress queue on a particular port. Configure the dedicated buffer allocation to a queue in the scheduler that is mapped to the egress queue by using the buffer-size option . You can configure the dedicated buffer allocation to queues in two ways:

  • As a percentage—The queue receives the specified percentage of dedicated port buffers when the queue is mapped to the scheduler and the scheduler is mapped to a port.
  • As a remainder—After the port services the queues that have an explicit percentage buffer size configuration, the remaining port dedicated buffer space is divided equally among the other queues to which a scheduler is attached. (No default or explicit scheduler means no dedicated buffer allocation for the queue.) If you configure a scheduler and you do not specify a buffer size as a percentage, remainder is the default setting.

Note: The total of all of the explicitly configured buffer size percentages for all of the queues on a port cannot exceed 100 percent.

In a port configuration that includes multiple forwarding class sets, with multiple forwarding classes mapped to multiple schedulers, the allocation of port dedicated buffers to queues depends on the mix of queues with buffer sizes configured as explicit percentages and queues configured with (or defaulted to) the remainder option.

The best way to demonstrate how using the percentage and remainder options affects port buffer allocation to queues is by showing an example of queue buffer allocation, and then showing how the queue buffer allocation changes when you add another forwarding class (queue) to the port.

Table 2 shows an initial configuration that includes four forwarding class sets, the five default forwarding classes (mapped to the five default queues for those forwarding classes), the buffer-size option configuration, and the resulting buffer allocation for each queue. Table 3 shows the same configuration after we add another forwarding class (queue) to the best-effort forwarding class set. Comparing the buffer allocations in each table shows you how adding another queue affects buffer allocation when you use remainders and explicit percentages to configure the buffer allocation for different queues.

Table 2: Example 1: Egress Queue Dedicated Buffer Allocation

Forwarding Class Set (Priority Group)

Forwarding Class

Queue

Scheduler Buffer Size Configuration

Buffer Allocation per Queue (Percentage)

fc-set-be

best-effort

0

10%

10%

fc-set-lossless

fcoe

3

20%

20%

no-loss

4

40%

40%

fc-set-strict-high

network-control

7

remainder

15%

fc-set-mcast

mcast

8

remainder

15%

In this first example, 70 percent of the egress port dedicated buffer pool is explicitly allocated to the best-effort, fcoe, and no-loss queues. The remaining 30 percent of the port dedicated buffer pool is split between the two queues that use the remainder option (network-control and mcast), so each queue receives 15 percent of the dedicated buffer pool.

Now we add another forwarding class (queue) to the best-effort priority group (fc-set-be) and configure it with a buffer size of remainder instead of configuring a specific percentage. Because a third queue now shares the remaining dedicated buffers, the queues that share the remainder receive fewer dedicated buffers, as shown in Table 3. The queues with explicitly configured percentages receive the configured percentage of dedicated buffers.

Table 3: Example 2: Egress Queue Dedicated Buffer Allocation with Another Remainder Queue

Priority Group (fc-set)

Forwarding Class

Queue

Scheduler Buffer Size Configuration

Buffer Allocation per Queue (Percentage)

fc-set-be

best-effort

0

10%

10%

best-effort-2

1

remainder

10%

fc-set-lossless

fcoe

3

20%

20%

no-loss

4

40%

40%

fc-set-strict-high

network-control

7

remainder

10%

fc-set-mcast

mcast

8

remainder

10%

The two tables show how the port divides the dedicated buffer space that remains after servicing the queues that have an explicitly configured percentage of dedicated buffer space.

Tradeoff Between Shared Buffer Space and Dedicated Buffer Space

The tradeoff between shared buffer space and dedicated buffer space is:

  • Shared buffers provide better absorption of traffic bursts because there is a larger pool of dynamic buffers that ports can use as needed to handle traffic bursts. However, all flows that exhaust their dedicated buffer space compete for the shared buffer pool. A larger shared buffer pool means a smaller dedicated buffer pool, and therefore more competition for the shared buffer pool because more flows exhaust their dedicated buffer allocation.
  • Dedicated buffers provide guaranteed buffer space to each port. The larger the dedicated buffer pool, the less likely that congestion on one port affects traffic on another port, because the traffic does not need to use as much shared buffer space. However, less shared buffer space means less ability to dynamically absorb traffic bursts.

When there is too much shared buffer space and too little dedicated buffer space, the result can be too much competition for the shared buffers because the flows exhaust their dedicated buffers too quickly. To maintain fairness, no single flow receives very much shared buffer space.

When there is too much dedicated buffer space and too little shared buffer space, there is less competition for the shared buffer pool because the dedicated buffer space is enough to service most flows. However, a smaller shared buffer pool leaves fewer buffers available to absorb traffic bursts.

For optimal burst absorption, the switch needs enough dedicated buffer space to avoid persistent competition for the shared buffer space. When fewer flows compete for the shared buffers, the flows that need shared buffer space to absorb bursts receive more of the shared buffer because fewer flows exhaust their dedicated buffer space.

The default configuration and all of the recommended configurations allocate 100 percent of the user-configurable memory space to the global shared buffer pool because the amount of nonconfigurable buffer space reserved for dedicated buffers provides enough dedicated buffers to avoid persistent competition for dynamic shared buffers. This results in fewer flows competing for the shared buffers, so the competing flows receive more of the buffer space.

Order of Buffer Consumption

The total buffer pool is divided into ingress and egress shared buffer pools and dedicated buffer pools. When traffic flows through the switch, the buffer space is used in a particular order that depends on the type of traffic.

On ingress, the order of buffer consumption is:

  • Best-effort unicast traffic:
    1. Dedicated buffers
    2. Shared buffers
    3. Global headroom buffers (very small)
  • Lossless unicast traffic:
    1. Dedicated buffers
    2. Shared buffers
    3. Lossless headroom buffers
    4. Global headroom buffers (very small)
  • Multidestination traffic:
    1. Dedicated buffers
    2. Shared buffers
    3. Global headroom buffers (very small)

On egress, the order of buffer consumption is the same for unicast best-effort, lossless unicast, and multidestination traffic:

  • Dedicated buffers
  • Shared buffers

In all cases on all ports, the switch uses the dedicated buffer pool first and the shared buffer pool only after the dedicated buffer pool for the port or queue is exhausted. This reserves the maximum amount of dynamic shared buffer space to absorb traffic bursts.

Shared Buffer Configuration Recommendations for Different Network Traffic Scenarios

The way you configure the shared buffer pool depends on the mix of traffic on your network. This section provides shared buffer configuration recommendations for four basic network traffic scenarios:

  • Balanced traffic—The network carries a balanced mix of unicast best-effort, lossless, and multicast traffic.
  • Best-effort unicast traffic—The network carries mostly unicast best-effort traffic.
  • Best-effort multicast traffic—The network carries mostly multicast best-effort traffic.
  • Lossless traffic—The network carries mostly lossless traffic (traffic on which PFC is enabled).

Note: Lossless traffic is defined as traffic on which you enable PFC to ensure lossless transport. Lossless traffic does not refer to best-effort traffic on a link on which you enable Ethernet PAUSE (IEEE 802.3X).

The lossless traffic configuration recommendation does not apply to networks that carry large amounts of best-effort traffic that is subject to Ethernet PAUSE. The best-effort traffic configuration also does not apply to networks that carry large amounts of Ethernet PAUSE traffic. You might have to adjust the buffer settings from the default configuration to optimize buffer utilization for networks with heavy Ethernet PAUSE traffic.

Start with the recommended shared buffer configuration for your network traffic conditions:

Balanced Traffic (Default Configuration)

The default shared buffer configuration is optimized for networks that carry a balanced mix of best-effort unicast, lossless, and multidestination (multicast, broadcast, and destination lookup fail) traffic. The default class of service (CoS) configuration is also optimized for networks that carry a balanced mix of traffic.

We recommend that you use the default shared buffer configuration for networks that carry a balanced mix of traffic, especially if you are using the default CoS settings. Table Table 4 shows the default ingress shared buffer allocations:

Table 4: Default Ingress Shared Buffer Configuration

Total Shared Ingress Buffer

Lossless Buffer

Lossless-Headroom Buffer

Lossy Buffer

100%

9%

45%

46%

Table Table 5 shows the default egress shared buffer allocations:

Table 5: Default Egress Shared Buffer Configuration

Total Shared Egress Buffer

Lossless Buffer

Lossy Buffer

Multicast Buffer

100%

50%

31%

19%

Best-Effort Unicast Traffic

If your network carries mostly best-effort (lossy) unicast traffic, then the default shared buffer configuration allocates too much buffer space to support lossless transport. Instead of wasting those buffers, we recommend that you use the following ingress shared buffer settings (see Table 6) and egress shared buffer settings (see Table 7):

Table 6: Recommended Ingress Shared Buffer Configuration for Networks with Mostly Best-Effort Unicast Traffic

Total Shared Ingress Buffer

Lossless Buffer

Lossless-Headroom Buffer

Lossy Buffer

100%

5%

0%

95%

Table 7: Recommended Egress Shared Buffer Configuration for Networks with Mostly Best-Effort Unicast Traffic

Total Shared Egress Buffer

Lossless Buffer

Lossy Buffer

Multicast Buffer

100%

5%

75%

20%

See Example: Recommended Configuration of the Shared Buffer Pool for Networks with Mostly Best-Effort Unicast Traffic for an example that shows you how to configure the recommended buffer settings shown in Table 6 and Table 7.

Best-Effort Multicast (Multidestination) Traffic

If your network carries mostly best-effort (lossy) multicast traffic, then the default shared buffer configuration allocates too much buffer space to support lossless transport. Instead of wasting those buffers, we recommend that you use the following ingress shared buffer settings (see Table 8) and egress shared buffer settings (see Table 9):

Table 8: Recommended Ingress Shared Buffer Configuration for Networks with Mostly Best -Effort Multicast Traffic

Total Shared Ingress Buffer

Lossless Buffer

Lossless-Headroom Buffer

Lossy Buffer

100%

5%

0%

95%

Table 9: Recommended Egress Shared Buffer Configuration for Networks with Mostly Best-Effort Multicast Traffic

Total Shared Egress Buffer

Lossless Buffer

Lossy Buffer

Multicast Buffer

100%

5%

20%

75%

See Example: Recommended Configuration of the Shared Buffer Pool for Networks with Mostly Multicast Traffic for an example that shows you how to configure the recommended buffer settings shown in Table 8 and Table 9.

Lossless Traffic

If your network carries mostly lossless traffic, then the default shared buffer configuration allocates too much buffer space to support best-effort traffic. Instead of wasting those buffers, we recommend that you use the following ingress shared buffer settings (see Table 10) and egress shared buffer settings (see Table 11):

Table 10: Recommended Ingress Shared Buffer Configuration for Networks with Mostly Lossless Traffic

Total Shared Ingress Buffer

Lossless Buffer

Lossless-Headroom Buffer

Lossy Buffer

100%

15%

80%

5%

Table 11: Recommended Egress Shared Buffer Configuration for Networks with Mostly Lossless Traffic

Total Shared Egress Buffer

Lossless Buffer

Lossy Buffer

Multicast Buffer

100%

90%

5%

5%

See Example: Recommended Configuration of the Shared Buffer Pool for Networks with Mostly Lossless Traffic for an example that shows you how to configure the recommended buffer settings shown in Table 10 and Table 11.

Optimizing the Buffer Configuration

Instead of using the default buffer configuration or one of the recommended buffer configurations, you can optimize the buffer configuration to best support the mix of traffic on your network:

Optimizing Buffer Configuration

If you feel that the recommended shared buffer configurations are not optimized enough for your network, adjust the settings gradually to fine-tune the shared buffer allocation. Use caution when adjusting the shared buffer configuration, not just when you fine-tune the ingress and egress buffer partitions, but also when you fine-tune the total ingress and egress shared buffer percentage. (Remember that if you allocate less than 100 percent of the available buffers to the shared buffers, the remaining buffers are added to the dedicated buffers). Tuning the buffers incorrectly can cause problems such as head-of-line blocking (HOLB).

Caution: Changing the buffer configuration is a disruptive event. Traffic stops on all ports until buffer reprogramming is complete.

The relationship between the sizes of the ingress buffer pool and the egress buffer pool affects when and where packets are dropped. The buffer pool sizes include the shared buffers and the dedicated buffers. In general, if there are more ingress buffers than egress buffers, the switch can experience HOLB because egress queues fill before ingress queues can empty.

Use the show class-of-service shared-buffer operational command to see the sizes in kilobytes (KB) of the dedicated and shared buffers and of the shared buffer partitions.

For best-efforttraffic (unicast and multidestination), the combined ingress lossy shared buffer partition and ingress dedicated buffers must be less than the combined egress lossy and multicast shared buffer partitions plus the egress dedicated buffers. This prevents HOLB and ensures that if packets are dropped, they are dropped at the egress queues. (Packets dropping at the ingress prevents the egress schedulers from working properly.)

For lossless traffic (traffic on which you enable PFC), the combined ingress lossless shared buffer partition and a reasonable portion of the ingress headroom buffer partition, plus the dedicated buffers, must be less than the total egress lossless shared buffer partition and dedicated buffers. (A reasonable portion of the ingress headroom buffer is approximately 20 to 25 percent of the buffer space, but this varies depending on how much buffer headroom is required to support the lossless traffic.) When these conditions are met, if there is HOLB, the HOLB triggers PFC on the ingress port to prevent packet loss. If the total lossless ingress buffers exceed the total lossless egress buffers, packets could be dropped at the egress instead of PFC being applied at the ingress to prevent packet loss.

Note: If you commit a buffer configuration that the switch cannot satisfy, the switch might log an error instead of returning a commit error. After you commit a buffer configuration, check the syslog messages to ensure that the new buffer configuration did not fail to commit.

If the buffer configuration commits but you receive a syslog message that indicates the configuration cannot be satisfied, you can:

  • Reconfigure the buffers or reconfigure other parameters (for example, the PFC configuration, which affects the need for lossless headroom buffers and lossless buffers—the more priorities you pause, the more lossless and lossless headroom buffer space you need), then attempt the commit operation again.
  • Rollback the switch to the last successful configuration.

If you receive a syslog message that says the buffer configuration cannot be satisfied, you must take corrective action. If you do not fix the configuration or rollback to a previous successful configuration, the system behavior is unpredicable.

General Rules and Considerations

Keep the following rules and considerations in mind when you configure the buffers:

  • Changing the buffer configuration is a disruptive event. Traffic stops on all ports until buffer reprogramming is complete.
  • If you configure the ingress or egress shared buffer percentages as less than 100 percent, the remaining percentage of buffer space is added to the reserved (not user-configurable) dedicated buffer pool.
  • The sum of all of the ingress shared buffer partitions must equal 100 percent. Each partition must be configured with a value of at least 5 percent except the lossless headroom buffer, which can have a value of 0 percent.
  • The sum of all of the egress shared buffer partitions must equal 100 percent. Each partition must be configured with a value of at least 5 percent.
  • Lossless and lossless headroom shared buffers serve traffic on which you enable PFC, and do not serve traffic subject to Ethernet PAUSE.
  • The switch uses the dedicated buffer pool first and the shared buffer pool only after the dedicated buffer pool for the port or queue is exhausted.
  • Too little dedicated buffer space results in too much competition for shared buffer space.
  • Too much dedicated buffer space results in poorer burst absorption because there is less shared buffer space available.
  • Always check the syslog messages after you commit a new buffer configuration.
  • The optimal buffer configuration for your network depends on the types of traffic on the network. If your network carries less traffic of a certain type (for example, lossless traffic), then reduce the size of the buffers allocated to that type of traffic (for example, reduce the sizes of the lossless and lossless headroom buffers).

Published: 2013-03-08