A Simple BBR Test on Linode

Overview:

TCP BBR solves two problems:

• Fully utilizing bandwidth on network links with some packet loss. Suitable for high-latency, high-bandwidth network links.
• Reducing buffer occupancy on network links, thereby lowering latency. Suitable for users with slow access networks.

I remember when BBR was first merged into the Linux 4.9 kernel in 2017, this congestion control algorithm proposed by Google was extremely popular in the tech community. At the time, I was doing cross-border file transfers on Linode, where packet loss was commonplace, so I wanted to test the results firsthand.

Test purpose: This test primarily focuses on packet loss rates. For more convincing tests, please refer to Lawrence Brakmo’s BBR Report. The Other Side of BBR

Test Setup:

• ADDR01: aaa.aaa.aaa.aaa

1
2

$ uname -r
4.8.6-x86_64-linode78

• ADDR02: bbb.bbb.bbb.bbb

1
2

# uname -r
4.8.6-x86_64-linode78

These two Linode machines were in different data centers — one on the US West Coast and one in Europe — simulating a cross-border network environment. At the time, CentOS 7 still used the 4.4 kernel by default, while BBR requires Linux 4.9+, so upgrading the kernel was necessary before enabling BBR.

Test Method:

• Simulated packet loss scenarios from 1% to 30%, testing different kernels with and without BBR enabled.

Linux’s traffic control tool tc can simulate various network conditions through netem, including packet loss, latency, jitter, etc. This is the standard tool for network testing. rsync was chosen as the testing tool because the real-world use case is cross-server file synchronization, which is closest to actual usage.

tc commands used:

1
2
3
4
5
6
7
8

# Clear tc rules:
tc qdisc del root dev eth0
# Simulate 1% packet loss:
tc qdisc add dev eth0 root netem loss 1%
# Simulate 10% packet loss:
tc qdisc add dev eth0 root netem loss 10%
# Simulate 30% packet loss:
tc qdisc add dev eth0 root netem loss 30%

• Test transferring data from ADDR02 to ADDR01. ADDR01’s kernel remained unchanged, while ADDR02’s kernel was adjusted and rebooted for each test.

Test Process:

• Steps omitted. test.gz is approximately 160MB, a compressed package normally used for backing up code. The general process was as follows:

Without BBR enabled, transferring data from ADDR02 to ADDR01:

1
2
3
4
5
6

$ rsync -ave "ssh -l mickey"  --progress test.gz [email protected]:/home/mickey/test.gz
sending incremental file list
test.gz
   166042909 100%    3.27MB/s    0:00:48 (xfer#1, to-check=0/1)
sent 166063274 bytes  received 31 bytes  3288382.28 bytes/sec
total size is 166042909  speedup is 1.00

During the transfer, the difference was very noticeable. Without BBR, the speed fluctuated — at 3.27MB/s, you could clearly see the progress bar moving slowly. Waiting for the 160MB to finish felt like a long time.

Test Data Comparison:

Test Summary and Observations (based on the above results):

• Linode’s custom-compiled kernel has noticeable targeted optimizations with significant effects.
• With BBR enabled, transfers could still complete even with 30% packet loss, demonstrating BBR’s effectiveness.
• The 4.10 kernel used generic instead of lowlatency.
• Originally planned to test 50% packet loss as well. However, after setting 50%, remote access to ADDR01 became nearly impossible, so the test was abandoned. At 50% packet loss, even SSH connections started lagging, making remote operations basically impossible, so this test had to be scrapped.

Appendix:

• CentOS 7 official kernel upgrade method:

This was the standard method for upgrading CentOS 7 to the 4.x kernel at the time, getting the latest ml (mainline) kernel through the elrepo repository:

• Upgrade kernel:

1
2
3

rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-2.el7.elrepo.noarch.rpm
yum --enablerepo=elrepo-kernel install kernel-ml -y

• Update boot:

1
2
3

egrep ^menuentry /etc/grub2.cfg | cut -f 2 -d \'
grub2-set-default 0  #default 0 means set the first kernel as default, choose the latest kernel
reboot

• Enable BBR:

1
2
3
4

modprobe tcp_bbr
echo "net.core.default_qdisc=fq" >> /etc/sysctl.conf
echo "net.ipv4.tcp_congestion_control=bbr" >> /etc/sysctl.conf
sysctl -p