Setup
For the test a single WHLE board was used, identified as whle_ls1046_1, and a PC equipped with a network card having two 10G ports. A single PC machine was set up to emulate two separate machines serving as endpoints for the WHLE board to route between. Please note that the actual speed test results depend on the CPU power of the PC, which, for the tests to measure only WHLE performance undisturbed, must be able to handle traffic at both ends at level exceeding the ability of a single WHLE board, with a good margin. In the carried out tests the 12-core 4.5 GHz x86_64 machine was more than sufficient for the task.
Before proceeding please make sure you followed the common setup described in WHLE-LS1046 kernel DPAA drivers .
Router
Connection diagram
Connection speed from ens1f1 interface to ens1f0 on PC will be measured, with whle_ls1046_1 configured as a router. The isolated_ns denotes network namespace in which the ens1f0 interface had to be enclosed to force PC to send the packets through whle_ls1046_1 instead of short-circuiting to the local interface.
...
Network Setup
PC
| Code Block |
|---|
root@PC:~# ip netns add isolated_ns root@PC:~# ip link set ens1f0 netns isolated_ns root@PC:~# ip netns exec isolated_ns ip addr flush ens1f0 root@PC:~# ip netns exec isolated_ns ip addr add 192.168.10.1/24 dev ens1f0 root@PC:~# ip netns exec isolated_ns ip link set dev ens1f0 up root@PC:~# ip netns exec isolated_ns ip route add 192.168.30.0/24 via 192.168.10.2 root@PC:~# ip addr flush ens1f1 root@PC:~# ip address add 192.168.30.2/24 dev ens1f1 root@PC:~# ip link set dev ens1f1 up root@PC:~# ip route add 192.168.10.0/24 via 192.168.30.1 |
whle_ls1046_1
| Code Block |
|---|
root@whle-ls1046a:~# ip address flush eth1 root@whle-ls1046a:~# ip address flush eth2 root@whle-ls1046a:~# ip address flush eth3 root@whle-ls1046a:~# ip address flush eth5 root@whle-ls1046a:~# ip address flush eth4 root@whle-ls1046a:~# ip addr add 192.168.10.2/24 dev eth5 root@whle-ls1046a:~# ip addr add 192.168.30.1/24 dev eth4 root@whle-ls1046a:~# ip link set dev eth4 up root@whle-ls1046a:~# ip link set dev eth5 up root@whle-ls1046a:~# echo 1 > /proc/sys/net/ipv4/ip_forward |
Tests
Iperf servers
On PC launch four instances of iperf3 servers, listening on ports 5201-5204. The ip netns exec command requires root access.
PC
| Code Block |
|---|
root@PC:~# ip netns exec isolated_ns iperf3 -s -p 5201 & root@PC:~# ip netns exec isolated_ns iperf3 -s -p 5202 & root@PC:~# ip netns exec isolated_ns iperf3 -s -p 5203 & root@PC:~# ip netns exec isolated_ns iperf3 -s -p 5204 & |
Iperf clients
Launch four instances of iperf3 simultaneously.
PC
| Code Block |
|---|
root@PC:~# (
iperf3 -c 192.168.10.1 --port 5201 --cport 55000 --time 0 --omit 5 --title A &
iperf3 -c 192.168.10.1 --port 5202 --cport 55002 --time 0 --omit 5 --title B &
iperf3 -c 192.168.10.1 --port 5203 --cport 55006 --time 0 --omit 5 --title C &
iperf3 -c 192.168.10.1 --port 5204 --cport 55001 --time 0 --omit 5 --title D &
)
A: Connecting to host 192.168.10.1, port 5201
B: Connecting to host 192.168.10.1, port 5202
C: Connecting to host 192.168.10.1, port 5203
D: Connecting to host 192.168.10.1, port 5204
C: [ 5] local 192.168.30.2 port 55006 connected to 192.168.10.1 port 5203
B: [ 5] local 192.168.30.2 port 55002 connected to 192.168.10.1 port 5202
D: [ 5] local 192.168.30.2 port 55001 connected to 192.168.10.1 port 5204
A: [ 5] local 192.168.30.2 port 55000 connected to 192.168.10.1 port 5201
B: [ ID] Interval Transfer Bitrate Retr Cwnd
C: [ ID] Interval Transfer Bitrate Retr Cwnd
B: [ 5] 0.00-1.00 sec 268 MBytes 2.25 Gbits/sec 56 337 KBytes (omitted)
C: [ 5] 0.00-1.00 sec 285 MBytes 2.39 Gbits/sec 152 433 KBytes (omitted)
A: [ ID] Interval Transfer Bitrate Retr Cwnd
D: [ ID] Interval Transfer Bitrate Retr Cwnd
D: [ 5] 0.00-1.00 sec 258 MBytes 2.16 Gbits/sec 195 370 KBytes (omitted)
A: [ 5] 0.00-1.00 sec 285 MBytes 2.39 Gbits/sec 162 584 KBytes (omitted)
B: [ 5] 1.00-2.00 sec 294 MBytes 2.46 Gbits/sec 15 741 KBytes (omitted)
... |
...
Stop all the clients after some time.
PC
| Code Block |
|---|
root@PC:~# kill $(ps a | grep 'iperf3 -[c]' | awk '{ print $1; }') |
...
The total bandwidth achieved is 2.29 + 2.37 + 2.33 + 2.35 = 9.34 Gb/s. This is the upper limit for the TCP protocol on a 10 Gb/s physical link, proving that WHLE-LS1046A board is able to handle routing at its network interface's limit using standard kernel drivers.
WHLE work analysis
Consider the snapshot from the top command ran on whle_ls1046_1 during the performance test:
...
The si column shows the CPU time spent in software interrupts, in this case the network interrupts almost exclusively. Nearly zero time spent by the system or user shows that the routing task is carried out in the interrupts alone. The even spreading of load spread evenly at ~73% between all cores stems from picking the right parameters (ip source address, ip dest address, tcp source port, tcp dest port) defining four data flows assigned by driver's RSS to four separate CPUs. The idle time id at ~25% shows that WHLE operates at 75% capacity, providing a decent margin to account for more realistic routing tasks, with bigger routing tables and less than perfectly CPU-even traffic.
L2 Bridge
Connection diagram
...
Network Setup
PC
| Code Block |
|---|
root@PC:~# ip netns add isolated_ns root@PC:~# ip link set ens1f0 netns isolated_ns root@PC:~# ip netns exec isolated_ns ip addr flush ens1f0 root@PC:~# ip netns exec isolated_ns ip addr add 192.168.30.1/24 dev ens1f0 root@PC:~# ip addr flush ens1f1 root@PC:~# ip address add 192.168.30.2/24 dev ens1f1 |
whle_ls1046_1
| Code Block |
|---|
root@whle-ls1046a:~# ip address flush eth4 root@whle-ls1046a:~# ip address flush eth5 root@whle-ls1046a:~# ip link set dev eth4 down root@whle-ls1046a:~# ip link set dev eth5 down root@whle-ls1046a:~# brctl addbr br0 root@whle-ls1046a:~# brctl addif br0 eth4 root@whle-ls1046a:~# brctl addif br0 eth5 root@whle-ls1046a:~# ip link set dev br0 up root@whle-ls1046a:~# ip link set dev eth4 up root@whle-ls1046a:~# ip link set dev eth5 up |
Tests
Iperf servers
On PC launch four instances of iperf3 servers, listening on ports 5201 to 5204. The ip netns exec command requires root access.
PC
| Code Block |
|---|
root@PC:~# ip netns exec isolated_ns iperf3 -s -p 5201 & root@PC:~# ip netns exec isolated_ns iperf3 -s -p 5202 & root@PC:~# ip netns exec isolated_ns iperf3 -s -p 5203 & root@PC:~# ip netns exec isolated_ns iperf3 -s -p 5204 & |
Iperf clients
Run four clients simultaneously:
PC
| Code Block |
|---|
root@PC:~# (
iperf3 -c 192.168.30.1 --port 5201 --cport 55000 --time 0 --title A &
iperf3 -c 192.168.30.1 --port 5202 --cport 55002 --time 0 --title B &
iperf3 -c 192.168.30.1 --port 5203 --cport 55004 --time 0 --title C &
iperf3 -c 192.168.30.1 --port 5204 --cport 55003 --time 0 --title D &
)
A: Connecting to host 192.168.30.1, port 5201
B: Connecting to host 192.168.30.1, port 5202
C: Connecting to host 192.168.30.1, port 5203
D: Connecting to host 192.168.30.1, port 5204
B: [ 5] local 192.168.30.2 port 55002 connected to 192.168.30.1 port 5202
D: [ 5] local 192.168.30.2 port 55003 connected to 192.168.30.1 port 5204
A: [ 5] local 192.168.30.2 port 55000 connected to 192.168.30.1 port 5201
C: [ 5] local 192.168.30.2 port 55004 connected to 192.168.30.1 port 5203
B: [ ID] Interval Transfer Bitrate Retr Cwnd
B: [ 5] 0.00-1.00 sec 243 MBytes 2.04 Gbits/sec 148 386 KBytes
C: [ ID] Interval Transfer Bitrate Retr Cwnd
C: [ 5] 0.00-1.00 sec 382 MBytes 3.21 Gbits/sec 243 331 KBytes
D: [ ID] Interval Transfer Bitrate Retr Cwnd
D: [ 5] 0.00-1.00 sec 251 MBytes 2.11 Gbits/sec 214 250 KBytes
A: [ ID] Interval Transfer Bitrate Retr Cwnd
A: [ 5] 0.00-1.00 sec 249 MBytes 2.09 Gbits/sec 83 370 KBytes
B: [ 5] 1.00-2.00 sec 210 MBytes 1.76 Gbits/sec 404 454 KBytes
A: [ 5] 1.00-2.00 sec 470 MBytes 3.95 Gbits/sec 173 551 KBytes
C: [ 5] 1.00-2.00 sec 224 MBytes 1.88 Gbits/sec 5 539 KBytes
D: [ 5] 1.00-2.00 sec 218 MBytes 1.83 Gbits/sec 23 362 KBytes
B: [ 5] 2.00-3.00 sec 229 MBytes 1.92 Gbits/sec 422 609 KBytes
... |
...
The total bandwidth achieved is 2.45 + 2.28 + 2.25 + 2.37 = 9.35 Gb/s. This is the upper limit for the TCP protocol on a 10 Gb/s physical link, proving that WHLE-LS1046A board is able to handle bridging at network interface's limit using standard kernel drivers.
WHLE work analysis
Consider the snapshot from the top command ran on whle_ls1046_1 during the performance test:
...
Just like in the case of router (https://conclusive.atlassian.net/wiki/spaces/CW/pages/edit-v2/398721025#WHLE-work-analysis) the only meaningful columns are id (idle) and si (software interrupt). Unlike with the router, however, the CPU load in the bridge mode has a high variance and thus a single top command snapshot can be misleading. It’s useful to record the numbers for a minute or so:
whle_ls1046_1
| Code Block |
|---|
top -d 0.5 -b \
| grep -e ^%Cpu \
| sed -e 's/[,:]/ /g' \
| awk '{print $1 "\t" $8 "\t" $14}' \
| tee cpu-load-id-si-per-5-ds.log |
...