Control groups, or cgroups, are a way in Linux to control processes' hardware resource resources utilization by defining the resources limits, grouping them in a hierarchical structure and assigning processes to them. Cgroups can be used, in particular, to specify the skb priority of all network packets generated by a specific process. This provides a convenient way to prioritize network traffic generated in communication with the WHLE board itself (as opposed to the traffic passing through it when it’s used as a router, a case described in Ssh Prioritization (iptables)).

Connection Diagram

Setup

Network Setup

PC

root@PC~# ip addr flush enxc84d4423262e
root@PC~# ip address add 192.168.3.1/24 dev enxc84d4423262e
root@PC~# ip link set dev enxc84d4423262e up

whle_ls1046

root@whle-ls1046a:~# ip address flush eth1
root@whle-ls1046a:~# ip addr add 192.168.3.2/24 dev eth1
root@whle-ls1046a:~# ip link set dev eth1 up

By default the network interfaces on WHLE are controlled by NetworkManager service and the effects of the ip commands above will be periodically overwritten with its own configuration. It may be necessary to temporarily stop the service

root@whle-ls1046a:~# systemctl stop NetworkManager

or to configure it to ignore the eth1 interface with a configuration like

root@whle-ls1046a:~# echo '
[main]
plugins=ifupdown,keyfile

[keyfile]
unmanaged-devices=interface-name:eth1
' > /etc/NetworkManager/NetworkManager.conf
root@whle-ls1046a:~# systemctl restart NetworkManager

Cgroups

...

Hierarchy Preparation

The cgroups hierarchy can be defined in many ways. Instead of creating the minimal hierarchy specific for the given scenario a more generic directory tree will be used, allowing for convenient assignment of skb priority from the 0 .. 15 range, thus covering all priority levels recognized by the tc command, to all network packets generated by a process with a given PID, in a straightforward fashion like

...

The script is as follows:

cgroups-setup.sh:

#!/usr/bin/bash
mkdir /sys/fs/cgroup/net_prio
mount -t cgroup -o net_prio none /sys/fs/cgroup/net_prio
mkdir /sys/fs/cgroup/net_prio/prio-{0..15}
for p in {0..15}; do
    for if in $(cd /sys/class/net/; ls); do
        echo "${if} ${p}" > /sys/fs/cgroup/net_prio/prio-${p}/net_prio.ifpriomap
    done
done

• mkdir /sys/fs/cgroup/net_prio
  This command creates the root directory for network priority hierarchy inside the /sys/fs/cgroup which should already be present on the system. The name net_prio is arbitrary. It was chosen to reflect the name of the module used to mount the cgroups filesystem there.

• mount -t cgroup -o net_prio none /sys/fs/cgroup/net_prio
  This command mounts the virtual filesystem used to communicate to the kernel the PIDs priority assignments. The -t cgroup signifies the cgroups V1. Unfortunately the more modern cgroups V2 cannot be used in this case as the net_prio module is not defined for it yet. Upon mounting the system the following listing should appear:

  Code Block
  root@whle-ls1046a:~# ls -1 /sys/fs/cgroup/net_prio cgroup.clone_children cgroup.procs cgroup.sane_behavior net_prio.ifpriomap net_prio.prioidx notify_on_release release_agent tasks

  Of these files only the following are relevant in further discussion:

  • net_prio.ifpriomap
    The default priorities per network interface. More details below.

  • cgroups.procs
    List of all PIDs whose packets priority isn’t modified in any way.

• mkdir /sys/fs/cgroup/net_prio/prio-{0..15}
  Create directories prio-0, prio-1, …, prio-15 inside the /sys/fs/cgroup/net_prio. Each of them will be automatically populated with files:

  Code Block
  root@whle-ls1046a:~# ls -1 /sys/fs/cgroup/net_prio/prio-13 cgroup.clone_children cgroup.procs net_prio.ifpriomap net_prio.prioidx notify_on_release tasks

  Again, only two are of concern here:

  • net_prio.ifpriomap
    The mapping of network interfaces to skb priorities, like

    Code Block
    root@whle-ls1046a:~# cat /sys/fs/cgroup/net_prio/prio-13/net_prio.ifpriomap lo 0 eth0 0 eth1 4 eth2 4 eth3 8 eth4 0 eth5 0

    While the initial discussion of cgroups mentioned assigning skb priority to PIDs, the actual priority assignment assignment’s subject is the (PID, interface) pair. This file covers the second part.

  • cgroups.procs
    List of all PIDs whose packets are assigned the priority according to the map given in net_prio.ifpriomap.

• echo "${if} ${p}" > /sys/fs/cgroup/net_prio/prio-${p}/net_prio.ifpriomap
  This line, executed for each network interface if, results in a uniform mapping in prio-‹p›/net_prio.ifpriomap like

  Code Block
  eth0 ‹p› eth1 ‹p› eth2 ‹p› eth3 ‹p› eth4 ‹p› eth5 ‹p›

  for example:

  Code Block
  root@whle-ls1046a:~# cat /sys/fs/cgroup/net_prio/prio-13/net_prio.ifpriomap lo 0 eth0 13 eth1 13 eth2 13 eth3 13 eth4 13 eth5 13

  This allows for abstracting over the interface prioritization granularity which isn’t needed.

Save the script in the cgroups-setup.sh file and run it on a WHLE-LS1046A board.

whle_ls1046a

root@whle-ls1046a:~# chmod +x cgroups-setup.sh
root@whle-ls1046a:~# ./cgroups-setup.sh

Iperf Setup

launch_iperf_with_priority() {
    local port=$1
    local prio=$2
    local iperf_time=$3
    echo "Launching iperf3, port ${port}, priority ${prio}"
    iperf3 --port "${port}" -c 192.168.3.1 --time "${iperf_time}" > /dev/null &
    local pid=$(pgrep -f "iperf3 --port ${port}")
    echo "${pid}"
    echo "${pid}" > "/sys/fs/cgroup/net_prio/prio-${prio}/cgroup.procs"
}

a

kill_iperf() {
    local port=$1
    pkill -f "iperf3 --port ${port}"
}

a

test_iperf() {
    local port1=$1
    local prio1=$2
    local port2=$3
    local prio2=$4
    local iperf_time=$5
    kill_iperf "${port1}"
    kill_iperf "${port2}"
    tc qdisc del dev eth1 root handle 1:
    launch_iperf_with_priority "${port1}" "${prio1}" "${iperf_time}"
    launch_iperf_with_priority "${port2}" "${prio2}" "${iperf_time}"
    tc qdisc add dev eth1 root handle 1: mqprio num_tc 4 \
       map 0 0 0 0 1 1 1 1 2 2 2 2 3 3 3 3 hw 1
    sleep ${iperf_time}
}