About

This article describes how to set up WHLE-LS1046A, using standard upstream DPAA driver, as a router giving strict priority to any ssh packets, making the router's non-ssh workload nearly transparent to any ssh connections going through it.

The article aims to showcase the practical use of DPAA hardware-offloaded Multiqueue Priority Discipline (mqprio qdisc) in conjunction with iptables, so the setup focuses only on the situation of network interface's congestion. Controlling access to other resources, like CPU, required to make the router's workload truly transparent to ssh connections, is outside of the scope.

Connection diagram

The setup is similar to the one used in Router/Bridge Mode: PC + WHLE Setup: whle_ls1046 board acts as a router between two links connected with the testing PC. The difference is that one of the links is 1 Gb/s instead of both being 10 Gb/s. This allows for saturating the physical link with little load on WHLE’s processing power, thus simplifying the setup and eliminating other possible factors which could influence the outcome of ssh throughput measuring experiments.

The speed of ssh connection will be measured between enxc84d4423262e and ens1f0 interfaces on PC. The isolated_ns denotes network namespace in which the ens1f0 interface had to be enclosed to force PC to send the packets through whle_ls1046 instead of short-circuiting to the local interface.

Three scenarios for ssh connection will be considered:

• no other traffic than ssh,

• ssh connection over a link saturated with iperf3 traffic:

  • without using DPAA’s priority queues,

  • with the usage of DPAA’s priority queues.

Network setup

PC

whle_ls1046a

By default the network interfaces on WHLE are controlled by NetworkManager service and the effects of the ip commands above will be periodically overwritten with its own configuration. It may be necessary to temporarily stop the service

or to configure it to ignore the eth1, eth5 interfaces with a configuration like

Services setup

PC

Keep in mind that starting the iperf3 server within the isolated network namespace isolated_ns makes it reachable only through the 192.168.10.1 address. Attempts to connect the client through a different address will result in a cryptic Bad file descriptor error.

It’s assumed that there is a ssh daemon running on PC already.

Tests

Control case: scp transfer through empty network

To measure the ssh throughput the scp program will be used on some decently big file ~700 MB, assumed to be at /home/user/files/download.xz on PC. It will be sent to /home/user on the same machine.

PC

The root access was needed to execute the ip netns command. Transferring the whole file through the empty network takes around 7 seconds.

The direction of the transfer is actually important in this experiment. The notion of queue prioritization in the DPAA architecture (or any other mqprio architecture for that matter) is only applicable to the egress traffic. Sending the local file /home/user/files/download.xz to the “remote“ location 192.168.3.1 from isolated namespace implies the following order of processing for the majority of ssh traffic:

1. PC’s CPU,

2. PC’s ens1f0 interface (egress),

3. whle_ls1046’s eth5 interface (ingress),

4. whle_ls1046’s CPU,

5. whle_ls1046’s eth1 interface (egress),

6. PC’s enxc84d4423262e interface (ingress),

7. PC’s CPU.

Given that the maximum throughput of 1 Gb/s for the whole connection leaves plenty of space on whle_ls1046’s CPU (let alone PC’s) and that the ens1f0 - eth5 link is 10 Gb/s, the 1 Gb/s enxc84d4423262e - eth1 link becomes the bottleneck, with packets congesting at the eth1 funnel where the DPAA prioritization can come into play. Having the transfer go the other way, eg. with

the funnel would form at the testing machine’s enxc84d4423262e interface.

Test case: scp transfer on saturated link, no prioritization

Start the iperf3 flow to saturate the 1 Gb/s link.

PC

Once again the direction of iperf3’s flow is important: it must match the direction scp’s transfer, or there would be no conflict between them to arbitrate. By default iperf3 sends data from client to server. Using the --reverse flag reverses it, ensuring that the data traverses ens1f0 (egress)→ eth5 (ingress) → eth1 (egress) → enxc84d4423262e (ingress).

Perform the scp transfer in another console.

PC

The time to transfer the file doubled. Meanwhile in iperf3’s logs:

This shows that with the default queuing discipline the 1 Gb/s link is shared evenly between iperf3 and scp, an expected behavior where neither flow has higher priority than the other.

Test case: scp transfer on saturated link, with prioritization

Setting iptables

Configure iptables to assign the highest skb priority to ssh packets.

whle_ls1046a

This should result in the following table:

The configuration makes use of the mangle table which is designed for packet modification. While the packets themselves aren’t modified in this scenario, their socket buffer structure’s used by the kernel is, namely the priority field.

The first command simply flushes the mangle table’s configuration to make sure no other rules apply. The second command assigns the priority 15 to any TCP packet with the destination port being 22. The third command does so with the source port. This effectively covers all standard ssh connections.

The actual priority assignment is done, indirectly, by the --set-class 0:f fragment. From iptables-extensions manual:

CLASSIFY
This module allows you to set the skb->priority value (and thus clas-
sify the packet into a specific CBQ class).

--set-class major:minor

Set the major and minor class value. The values are always in-

terpreted as hexadecimal even if no 0x prefix is given.

Unfortunately the documentation doesn’t provide the actual correspondence between major:minor class specification and the affected skb->priority value. This can be found in iptable’s source (iptables-1.8.7/extensions/libxt_CLASSIFY.c):

and kernel’s source (include/uapi/linux/pkt_sched.h):

From this it can be concluded that as long as major is 0 and minor < 0x10000 then skb->priority is simply the value of minor. To use a different priority 10, for example, one would have to use the --set-class 0:a. The values of skb->priority higher than 0xF aren’t recognized by the mqprio qdisc anyway.

The usage of POSTROUTING chain signifies that the prioritization occurs right before the packet is sent to the network interface. It’s not strictly required to do it at the last moment and the FORWARD chain could be used as well. The OUTPUT chain, however, applies only to the packets generated by whle_ls1046 itself, so the routed packets would remain unaffected, while PREROUTING and INPUT chains aren’t even accepted along with the CLASSIFY target by iptables command.

Setting tc

Set up the queues discipline for the eth1 interface.

whle_ls1046a

The first command deletes any qdisc that may have been assigned to eth1 already. It may return an error when there is none, that’s not a problem.

The second command is the same as the one used in Traffic Control with tc | Example - see that article for detailed description.

Although it’s not enforced by the configuration, it can be established empirically that packets from iperf3’s traffic fall into classes 0 and 1. Assuming that the iptables configuration properly assigns ssh packets the skb priority 15 before sending them to eth1 for transfer they should all fall into traffic class 3 and be enqueued on the highest priority Work Queue WQ0, to be serviced before all iperf3 packets. This should result in iperf3’s traffic being stopped completely for the duration of scp’s transfer.

Performing the test

Start the iperf3 flow to saturate the link.

PC

Perform the scp transfer in another console.

PC

The file transfer time is basically the same as if there was no other data transferred on the link. Meanwhile in iperf3’s logs:

The iperf3 flow ceased completely during the scp transfer, showcasing the strict priority rule in action.