About

This article discusses the Multiqueue Priority Queue Discipline (mqprio qdisc) hardware offloading implemented by the standard kernel DPAA driver, how to set it up with the tc command and how to monitor it.

Setting tc

The driver’s documentation mentions the following command:

tc qdisc add dev <int> root handle 1: \
       mqprio num_tc 4 map 0 0 0 0 1 1 1 1 2 2 2 2 3 3 3 3 hw 1

Set up the queues discipline for the eth1 interface.

whle_ls1046a

root@whle-ls1046a:~# tc qdisc del dev eth1 root handle 1:
root@whle-ls1046a:~# tc qdisc add dev eth1 \
    root handle 1: mqprio num_tc 4 map 0 0 0 0 1 1 1 1 2 2 2 2 3 3 3 3 hw 1

The first command deletes any qdisc that may have been assigned to eth1 already. It may return an error when there is none, that’s not a problem.

This command encapsulates

• traffic classes,

• packets skb priority,

• mapping between skb priority and traffic classes,

• DPAA Frame Queues,

• DPAA Work Queues,

• device’s channel

The second command initiates 1024 DPAA queues in 4 different classes, each having a different DPAA priority (to distinguish it from skb priority that iptables is concerned with).

Below is the detailed description of the command’s fragments and what setup they result in.

mqprio

Use the “Multiqueue Priority Qdisc (Offloaded Hardware QOS)“ man:tc-mqprio.

num_tc 4
Use 4 queue classes, identified with numbers 0, 1, 2, 3. The queue’s class (Packet Queue’s Traffic Class in DPAA’s nomenclature) maps directly to the Work Queue in eth1’s Direct Connected Channel the queue is put on when having at least one packet (see Upstream DPAA driver and Channels):

DPAA arranges Work Queues into 4 groups, ordered by their increasing DPAA priority:

They are governed by a strict priority rule: a group with priority number n must be emptied of all packets before any packet from the group with number k lower than n can be serviced (k, n in {1,2,3,4}). Because WQ0, WQ1, WQ2, WQ6 corresponding to different traffic classes all belong to different groups, the strict priority rule effectively applies to the traffic classes 0, 1, 2, 3 as well, with 3 having the highest priority.

Each class has exactly 256 queues, resulting in total of 1024 queues in this case. This number cannot be changed with the tc’s queues argument - it’s silently ignored by DPAA’s driver, including the provided offsets. No more than 4 classes can be used. When less than 4 classes are used then the queues are trimmed from the higher priority end. For example, using num_tc 3 would result in 768 queues (3 * 256) belonging to traffic classes 0, 1, 2, using work queues WQ6, WQ2, WQ1.

map 0 0 0 0 1 1 1 1 2 2 2 2 3 3 3 3

This argument maps skb priority 0 .. 15 to the traffic class 0 .. 3 using the value’s position in the sequence as argument.

hw 1

Tells tc to actually use hardware offloading implemented by DPAA architecture instead of emulating this queue discipline in kernel.

The created queue discipline can be displayed with

root@whle-ls1046a:~# tc qdisc show dev eth1

qdisc mqprio 1: root tc 4 map 0 0 0 0 1 1 1 1 2 2 2 2 3 3 3 3 
             queues:(0:255) (256:511) (512:767) (768:1023) 
             mode:dcb
             shaper:dcb
qdisc pfifo_fast 0: parent 1:400 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: parent 1:3ff bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: parent 1:3fe bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: parent 1:3fd bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: parent 1:3fc bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
...
qdisc pfifo_fast 0: parent 1:3 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: parent 1:2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: parent 1:1 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1

The qdisc pfifo_fast 0: parent ... are sub-qdiscs which correspond directly to the DPAA frame queues listed as queues.

The full relation between the entities described so far is as follows:

Keep in mind that the “DPAA Frame Queue“ indexes displayed in tc’s output are not the same as Frame Queue IDs for Tx which can be displayed for eth1 as Tx: 3337 - 4360 with

root@whle-ls1046a:~# cat /sys/class/net/eth1/fqids

Rx error: 2181
Rx default: 2182
Rx PCD: 2304 - 2431
Tx confirmation (mq): 2183 - 2303
Tx confirmation (mq): 2432 - 3334
Tx error: 3335
Tx default confirmation: 3336
Tx: 3337 - 4360

The Frame Queue IDs are low-level DPAA identifiers which must be globally unique across all network interfaces. The (0:255) (256:511) (512:767) (768:1023) ids are tc-specific and describe only the queues assigned to the interface provided in the argument, in this case eth1.

Although it’s not enforced by the configuration, it can be established empirically that packets from iperf3’s traffic fall into classes 0 and 1. Assuming that the iptables configuration properly assigns ssh packets the skb priority 15 before sending them to eth1 for transfer they should all fall into traffic class 3 and be enqueued on the highest priority Work Queue WQ0, to be serviced before all iperf3 packets. This should result in iperf3’s traffic being stopped completely for the duration of scp’s transfer.

Performing the test

Start the iperf3 flow to saturate the link.

PC

user@PC:~$ iperf3 --client 192.168.10.1 --time 0 --reverse

Connecting to host 192.168.10.1, port 5201
Reverse mode, remote host 192.168.10.1 is sending
[  5] local 192.168.3.1 port 41978 connected to 192.168.10.1 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   112 MBytes   941 Mbits/sec                  
[  5]   1.00-2.00   sec   112 MBytes   942 Mbits/sec                  
[  5]   2.00-3.00   sec   112 MBytes   941 Mbits/sec                  
...

Perform the scp transfer in another console.

PC

root@PC:~# time ip netns exec isolated_ns scp /home/user/files/download.xz user@192.168.3.1:

download.xz                                                           100%  706MB 111.7MB/s   00:06    
real	0m6,773s
user	0m3,766s
sys	0m1,534s

The file transfer time is basically the same as if there was no other data transferred on the link. Meanwhile in iperf3’s logs:

...
[  5]  17.00-18.00  sec   112 MBytes   941 Mbits/sec                  
[  5]  18.00-19.00  sec   112 MBytes   942 Mbits/sec                  
[  5]  19.00-20.00  sec   112 MBytes   942 Mbits/sec                  
[  5]  20.00-21.00  sec   112 MBytes   941 Mbits/sec                  
[  5]  21.00-22.00  sec  70.2 MBytes   589 Mbits/sec                  <-- scp transfer start
[  5]  22.00-23.00  sec  0.00 Bytes  0.00 bits/sec                  
[  5]  23.00-24.00  sec  0.00 Bytes  0.00 bits/sec                  
[  5]  24.00-25.00  sec  0.00 Bytes  0.00 bits/sec                  
[  5]  25.00-26.00  sec  0.00 Bytes  0.00 bits/sec                  
[  5]  26.00-27.00  sec  0.00 Bytes  0.00 bits/sec                  
[  5]  27.00-28.00  sec  5.87 MBytes  49.3 Mbits/sec                  <-- scp transfer finish
[  5]  28.00-29.00  sec   112 MBytes   942 Mbits/sec                  
[  5]  29.00-30.00  sec   112 MBytes   942 Mbits/sec                  
[  5]  30.00-31.00  sec   112 MBytes   942 Mbits/sec                  
[  5]  31.00-32.00  sec   112 MBytes   942 Mbits/sec                  
...

The iperf3 flow ceased completely during the scp transfer, showcasing the strict priority rule in action.