Setup

For the tests three WHLE-LS1046A boards were used, identified as whle_ls1046_0, whle_ls1046_1, whle_ls1046_2. The following sections describe setup common for them all.

System

Conclusive Ubuntu image was used, version ubuntu-jammy-whle-ls1046a-2024-05-15-6ccb862.img, available at https://gitlab.conclusive.pl/devices/ubuntu-build/-/packages/109. For the flashing procedure please refer to Installing Ubuntu image.

Kernel

In tests the kernel version 6.5.6-32773-whle-ls1-g1dbc2136b533 was used, although any recent Conclusive kernels 6.5 or 6.1 should work.

No additional kernel arguments are needed, for the tests the default arguments were used:

Software

Router

Connection diagram

Connection speed from whle_ls1046_2 to whle_ls1046_0 will be measured, with whle_ls1046_1 configured as router.

Open

Drivers setup

The tests were carried out using iperf3 tool, TCP protocol, with servers on whle_ls1046_0 and clients on whle_ls1046_2, with the direction of the main network flow being whle_ls1046_2 → whle_ls1046_1 → whle_ls1046_0. The overall speed of transfer depends on the speed at which whle_ls1046_1 and whle_ls1046_2 are able to receive data, which is bound by CPU frequency and depends on how the traffic flows are shared between cores. To achieve optimal bandwidth it's necessary to (for each of whle_ls1046_1 and whle_ls1046_2) bind the specific iperf3 flow to a single core and to make sure the bindings don't overlap.

Kernel DPAA driver allows some limited traffic control with the help of ethtool (https://docs.kernel.org/networking/device_drivers/ethernet/freescale/dpaa.html). Core-binding is achieved by using Receive Side Scaling feature, enabled on 10G ports with

The RSS is enabled by default and the command above shouldn't be necessary. The sfdn argument specifies the packet header fields defining the flow and used to map it to a specific core.

• s: Source IP address

• f: Destination IP address

• d: Destination port

• n: Source port.

The current set of fields used in RSS for the given network device can be printed with

The exact core the flow will be bound to, being based on the hash value from the above fields, is not known in advance, although it's deterministic.

Unfortunately it's apparently not possible to reduce these four fields - using any of s, f, d, n in the ethtool call enables all the other three.

Network Setup

whle_ls1046_0

whle_ls1046_1

whle_ls1046_2

Tests

Iperf servers

On whle_ls1046_0 launch four instances of iperf3 servers, listening on ports 5201-5204.

whle_ls1046_0

Iperf clients

Launching iperf3 clients is a bit more involved as the reproducibility requires controlling the source port, which is an ephemeral port assigned by the system from a specific range, usually 32768-60999 for Linux. This can achieved by temporarily narrowing the range to just a single port such that the iperf3 client has no choice but to use only one available. Use the following helper script wrapping the iperf3 call in the ephemeral ports range setup:

iperfc.sh:

Run the clients simultaneously:

The addresses and ports are picked in such a way that every iperf3 flow is handled by a different core on whle_ls1046_1 and whle_ls1046_2 (the cores 0, 2, 1, 3, respectively).

Stop all the clients after some time

Sum the values from the lines with "sender" at the end

The total bandwidth achieved is 2.16 + 2.24 + 2.17 + 2.16 = 8.73 Gb/s. The high number of retries suggests the receiving endpoint whle_ls1046_2 being the weakest link.