Control groups, or cgroups, are a way in Linux to control processes' hardware resources utilization by defining the resources limits, grouping them in a hierarchical structure and assigning processes to them. Cgroups can be used, in particular, to specify the skb priority of all network packets generated by specific process. This provides a convenient way to prioritize network traffic generated in communication with the WHLE board itself (as opposed to the traffic passing through it when it’s used as a router, a case described in Ssh Prioritization (iptables)).

Connection Diagram

The network used is very straightforward and consists of a single 1 Gb/s link between a testing machine (PC) and a WHLE board (whle_ls1046). Two iperf3 streams sending data from whle_ls1046 to PC will be competing for the link’s throughput. Different traffic classes will be used using the cgroups mechanism for the associated iperf3 processes and the resulting changes in data transfer speed will be observed.

Setup

Network Setup

PC

whle_ls1046

By default the network interfaces on WHLE are controlled by NetworkManager service and the effects of the ip commands above will be periodically overwritten with its own configuration. It may be necessary to temporarily stop the service

or to configure it to ignore the eth1 interface with a configuration like

Cgroups Hierarchy Preparation

The cgroups hierarchy can be defined in many ways. Instead of creating the minimal hierarchy specific for the given scenario a more generic directory tree will be used, allowing for convenient assignment of skb priority from the 0 .. 15 range, thus covering all priority levels recognized by the tc command, to all network packets generated by a process with a given PID, in a straightforward fashion like

for example:

The script is as follows:

cgroups-setup.sh:

• mkdir /sys/fs/cgroup/net_prio
  This command creates the root directory for network priority hierarchy inside the /sys/fs/cgroup which should already be present on the system. The name net_prio is arbitrary. It was chosen to reflect the name of the module used to mount the cgroups filesystem there.

• mount -t cgroup -o net_prio none /sys/fs/cgroup/net_prio
  This command mounts the virtual filesystem used to communicate to the kernel the PIDs priority assignments. The -t cgroup signifies the cgroups V1. Unfortunately the more modern cgroups V2 cannot be used in this case as the net_prio module is not defined for it yet. Upon mounting the system the following listing should appear:

  Of these files only the following are relevant in further discussion:

  • net_prio.ifpriomap
    The default priorities per network interface. More details below.

  • cgroups.procs
    List of all PIDs whose packets priority isn’t modified in any way.

• mkdir /sys/fs/cgroup/net_prio/prio-{0..15}
  Create directories prio-0, prio-1, …, prio-15 inside the /sys/fs/cgroup/net_prio. Each of them will be automatically populated with files:

  Again, only two are of concern here:

  • net_prio.ifpriomap
    The mapping of network interfaces to skb priorities, like

    While the initial discussion of cgroups mentioned assigning skb priority to PIDs, the actual priority assignment’s subject is the (PID, interface) pair. This file covers the second part.

  • cgroups.procs
    List of all PIDs whose packets are assigned the priority according to the map given in net_prio.ifpriomap.

• echo "${if} ${p}" > /sys/fs/cgroup/net_prio/prio-${p}/net_prio.ifpriomap
  This line, executed for each network interface if, results in a uniform mapping in prio-‹p›/net_prio.ifpriomap like

  for example:

  This allows for abstracting over the interface prioritization granularity which isn’t needed.

Save the script in the cgroups-setup.sh file and run it on a WHLE-LS1046A board.

whle_ls1046a

Iperf3 Setup

PC

Two iperf3 streams will be created, with servers launched on PC and clients on whle_ls1046, with the default client → server data flow direction.

The direction of the transfer is important in this experiment. The notion of queue prioritization in the DPAA architecture (or any other mqprio architecture for that matter) is only applicable to the egress traffic. Sending data to the remote location 192.168.3.1 from isolated namespace implies the following order of processing for the majority of iperf3 traffic:

1. whle_ls1046’s CPU,

2. whle_ls1046’s eth1 interface (egress),

3. PC’s enxc84d4423262e interface (ingress),

4. PC’s CPU.

Given that the maximum throughput of 1 Gb/s for the whole connection leaves plenty of space on whle_ls1046’s CPU (let alone PC’s), the enxc84d4423262e - eth1 link becomes the bottleneck, with packets congesting at the eth1 funnel where the DPAA prioritization can come into play. Having the transfer go the other way, eg. with clients run on PC and servers on whle_ls1046, the funnel would form at the testing machine’s enxc84d4423262e interface.

Given the peculiarities of setting up iperf3 process' priority on whle_ls1046 it’s easier to track data transfer speed on PC’s side by launching iperf3 in blocking mode instead of as a daemon.

PC, console 1

PC, console 2

whle_ls1046

Launching iperf3 clients on the WHLE board must follow a specific protocol:

1. Clean any mqprio qdiscs on the eth1 interface.

2. Run iperf3 client.

3. Obtain the client’s PID

4. Assign a specific net_prio priority to the given PID, using cgroups.

5. Define the mqprio qdisc on the eth1 interface.

While the (2) < (3) < (4) ordering is pretty obvious, the rest may not be so. Practice has shown that changing packets priority of a process with an ongoing connection while the mqprio is already set up leads to inconsistent results, with the change sometimes reflected on the wire and sometimes not. In contrast, setting mqprio qdisc while all the traffic is already set up and running results in consistent behavior.

Because of this it’s useful to define some bash procedures that would implement the above ordering. First a launch_iperf_with_priority function will be defined which starts the iperf3 client and assigns it a specific priority.

whle_ls1046

The opposite operation will be realized by the kill_iperf procedure.

whle_ls1046

Example uage:

whle_ls1046

This would create a connection with at server at 192.168.3.1, port 5201, for 10 seconds, with the packets sent having skb priority 0. Then it will be killed without waiting for it to finish.

Building on this a third, final procedure will be defined, which coordinates launching two iperf3 streams with different priorities, for the same time period, and the creation of mqprio qdisc.

whle_ls1046

The tc qdisc ... command is the same as the one used in Traffic Control with tc | Example - see that article for detailed description.

Tests

Same priorities

Assuming that iperf3 servers at ports 5201, 5202 are running on PC, run the following command on WHLE:

whle_ls1046

This would create two iperf3 streams with the same skb priority 4, mapping to the traffic class 1. Meanwhile, on the PC side:

PC, console 1

PC, console 2

The experiment shows that the link’s throughput is shared evenly for traffic in the same class. Similar results would be obtained with calls:

(That would cover all 4 traffic classes defined by tc, with skb priorities different from 0, 4, 8, 12 resulting in the same classes set.)

Different priorities

Run test_iperf with different skb priorities, making sure that they map to different traffic classes, for example:

whle_ls1046

Meanwhile, on the PC side:

PC, console 1

PC, console 2

This shows that the traffic class 1 (skb priority 4) has a strict priority over traffic class 0 (skb priority 0). Similar results would be obtained with any of the calls:

(That would cover all pairs of 4 traffic classes defined by tc, with skb priorities different from 0, 4, 8, 12 resulting in one of the classes pairs from above.)