About

This article discusses the Multiqueue Priority Queue Discipline (mqprio qdisc) hardware offloading implemented by the standard kernel DPAA driver, how to set it up with the tc command and how to monitor it.

tc Command Analysis

The driver’s documentation mentions the following command:

It involves explicitly:

• the choice of queue discipline,

• definition of network traffic classes,

• mapping between skb priority and traffic classes,

and implicitly:

• packets skb priority itself,

• DPAA Frame Queues,

• DPAA Work Queues,

• device’s Dedicated Channel.

Each of these elements and the relation between them will be explained below, by analyzing the command’s parts.

dev ‹int›

The tc command defines how the egress traffic is to be prioritized in case of specific interface’s congestion, when the operating system produces more packets to send than link’s capacity. As such it operates on a network device ‹int›. For WHLE boards it can be any of eth0, eth1, eth2, eth3, eth4, eth5. When there is no congestion condition the traffic control loses its meaning - any packet arriving on the interface from the operating system gets transferred immediately.

mqprio

The “queue discipline“, or “qdisc“ for short, is a method of handling the congestion condition. The tc command provides multiple queue disciplines to chose from, identified by a short names like fq, choke, sfb, or mqprio (see man:tc). Using mqprio tells tc to pick the “Multiqueue Priority Qdisc (Offloaded Hardware QOS)“ (man:tc-mqprio). While any of the tc provided qdiscs can be used on WHLE-LS1046A/26A board (provided that the kernel was compiled with a proper option enabling it), the mqprio qdisc is special in the sense that the logic of deciding which packet to serve next is handled by LS1046A/26A hardware as an integral part of DPAA architecture, freeing CPU of cycles required to handle traffic.

The multiqueue priority qdisc is based on the existence of multiple hardware packet queues, all associated with a single network interface. In case of the discussed command a total of 1024 DPAA Frame Queues are initialized. The queues are divided into classes with different priorities. Packets from a single network flow are enqueued on the same queue, which helps in preserving the ordering. All queues within the same class are treated equally. The reason for having more than one queue per class is to smooth out the sharing of a link between many flows.

num_tc 4

Use 4 queue classes, identified with numbers 0, 1, 2, 3. The queue’s class (Packet Queue’s Traffic Class in DPAA’s nomenclature) maps directly to the Work Queue in ‹int›’s Direct Connected Channel the queue is put on when having at least one packet (see Upstream DPAA driver and Channels):

DPAA arranges Work Queues into 4 groups, ordered by their increasing DPAA priority:

They are governed by a strict priority rule: a group with priority number n must be emptied of all packets before any packet from the group with number k lower than n can be serviced (k, n in {1,2,3,4}). Because WQ0, WQ1, WQ2, WQ6 corresponding to different traffic classes all belong to different groups, the strict priority rule effectively applies to the traffic classes 0, 1, 2, 3 as well, with 3 having the highest priority.

Each class has exactly 256 queues, resulting in total of 1024 queues in this case. This number cannot be changed with the tc’s queues argument - it’s silently ignored by DPAA’s driver, including the provided offsets. No more than 4 classes can be used. When less than 4 classes are used then the queues are trimmed from the higher priority end. For example, using num_tc 3 would result in 768 queues (3 * 256) belonging to traffic classes 0, 1, 2, using work queues WQ6, WQ2, WQ1.

map 0 0 0 0 1 1 1 1 2 2 2 2 3 3 3 3

This argument maps “skb priority“ 0 .. 15 to the traffic class 0 .. 3 using the value’s position in the sequence as argument.

The “skb priority“, often written as “skb->priority“ in documentation, is the field in the “socket buffer“ structure associated with an IP packet, used by kernel through the whole packet’s processing pipeline. It’s related to the IP’s TOS field, although it can be changed with the use of cgroups or iptables. The control over the skb priority of packets is key to effective use LS1046A’s hardware prioritization feature and is discussed at length in Direct Connection (cgroups) and Ssh Prioritization (iptables).

hw 1

Tells tc to actually use hardware offloading implemented by DPAA architecture instead of emulating this queue discipline in kernel.

Example

Queues Definition

The created queue discipline can be displayed with

The qdisc pfifo_fast 0: parent ... are sub-qdiscs which correspond directly to the DPAA frame queues listed as queues.

The full relation between the entities described so far is as follows:

Keep in mind that the “DPAA Frame Queue“ indexes displayed in tc’s output are not the same as Frame Queue IDs for Tx which can be displayed for eth1 as Tx: 3337 - 4360 with

The Frame Queue IDs are low-level DPAA identifiers which must be globally unique across all network interfaces. The (0:255) (256:511) (512:767) (768:1023) ids are tc-specific and describe only the queues assigned to the interface provided in the argument, in this case eth1.

Queues Monitoring

It’s sometimes very useful to display the usage of all the defined Frame Queues. This can be done with the -statistics option:

The rather impractically long output can be reduced to just the used queues statistics with

or even just the list of used queues with:

This particular output indicates that, so far, 3 classes were used to transfer data

• class 2: queues 2e4, 2df,

• class 1: queues 1b1, 166,

• class 0: queues 100, 76.

If some traffic was expected to be classified into the highest priority class 3 then this list would identify a problem on either skb priority assignment level or the classification level itself.