单Iperf线程无法占满CPU？100G带宽测试瓶颈求助

云上老周

2026-8-24

问题描述

我在Ubuntu 18.04系统上使用iperf2进行单CPU带宽测试，拓扑为：带CX5 100G网卡的客户端 → 100G交换机 → 带CX5 100G网卡的服务器。实际跨机测试仅跑出30Gbps带宽，但本地回环测试能达到60Gbps且CPU占用100%，请问该如何解决？

跨机测试输出

Binding to local address 10.0.0.2
Write buffer size:  128 KByte
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  3] local 10.0.0.2 port 42749 connected with 10.0.0.1 port 5001
[ ID] Interval        Transfer    Bandwidth       Write/Err  Rtry    Cwnd/RTT
[  3] 0.00-1.00 sec  3.71 GBytes  31.9 Gbits/sec  30379/0          0     6025K/354 us
[ ID] Interval        Transfer    Bandwidth       Write/Err  Rtry    Cwnd/RTT
[  3] 1.00-2.00 sec  3.59 GBytes  30.8 Gbits/sec  29415/0          0     6684K/271 us
[ ID] Interval        Transfer    Bandwidth       Write/Err  Rtry    Cwnd/RTT
[  3] 2.00-3.00 sec  3.63 GBytes  31.2 Gbits/sec  29712/0          0     7407K/399 us
[ ID] Interval        Transfer    Bandwidth       Write/Err  Rtry    Cwnd/RTT
[  3] 3.00-4.00 sec  3.71 GBytes  31.8 Gbits/sec  30359/0          0     8285K/399 us
[ ID] Interval        Transfer    Bandwidth       Write/Err  Rtry    Cwnd/RTT
[  3] 4.00-5.00 sec  3.73 GBytes  32.0 Gbits/sec  30520/0          0     8285K/328 us
[ ID] Interval        Transfer    Bandwidth       Write/Err  Rtry    Cwnd/RTT
[  3] 5.00-6.00 sec  3.77 GBytes  32.4 Gbits/sec  30911/0          0     8285K/378 us
[ ID] Interval        Transfer    Bandwidth       Write/Err  Rtry    Cwnd/RTT
[  3] 6.00-7.00 sec  3.77 GBytes  32.4 Gbits/sec  30920/0          0     8285K/384 us
[ ID] Interval        Transfer    Bandwidth       Write/Err  Rtry    Cwnd/RTT
[  3] 7.00-8.00 sec  3.74 GBytes  32.1 Gbits/sec  30646/0          0     8285K/325 us
[ ID] Interval        Transfer    Bandwidth       Write/Err  Rtry    Cwnd/RTT
[  3] 8.00-9.00 sec  3.77 GBytes  32.4 Gbits/sec  30917/0          0     8285K/334 us
[ ID] Interval        Transfer    Bandwidth       Write/Err  Rtry    Cwnd/RTT
[  3] 9.00-10.00 sec  3.72 GBytes  32.0 Gbits/sec  30503/0          0     8285K/278 us
[ ID] Interval        Transfer    Bandwidth       Write/Err  Rtry    Cwnd/RTT
[  3] 0.00-10.00 sec  37.1 GBytes  31.9 Gbits/sec  304286/0          0     8285K/278 us

本地回环测试输出

Binding to local address 10.0.0.2
Write buffer size:  128 KByte
TCP window size: 2.50 MByte (default)
------------------------------------------------------------
[  3] local 10.0.0.2 port 37687 connected with 10.0.0.2 port 5001
[ ID] Interval        Transfer    Bandwidth       Write/Err  Rtry    Cwnd/RTT
[  3] 0.00-1.00 sec  3.85 GBytes  33.1 Gbits/sec  31562/0          0     1087K/41 us
[ ID] Interval        Transfer    Bandwidth       Write/Err  Rtry    Cwnd/RTT
[  3] 1.00-2.00 sec  5.82 GBytes  50.0 Gbits/sec  47638/0          0     1087K/20 us
[ ID] Interval        Transfer    Bandwidth       Write/Err  Rtry    Cwnd/RTT
[  3] 2.00-3.00 sec  7.65 GBytes  65.7 Gbits/sec  62644/0          0     1087K/19 us
[ ID] Interval        Transfer    Bandwidth       Write/Err  Rtry    Cwnd/RTT
[  3] 3.00-4.00 sec  7.46 GBytes  64.1 Gbits/sec  61140/0          0     1342K/19 us
[ ID] Interval        Transfer    Bandwidth       Write/Err  Rtry    Cwnd/RTT
[  3] 4.00-5.00 sec  7.66 GBytes  65.8 Gbits/sec  62716/0          0     1342K/19 us
[ ID] Interval        Transfer    Bandwidth       Write/Err  Rtry    Cwnd/RTT
[  3] 5.00-6.00 sec  7.60 GBytes  65.3 Gbits/sec  62231/0          0     1342K/21 us
[ ID] Interval        Transfer    Bandwidth       Write/Err  Rtry    Cwnd/RTT
[  3] 6.00-7.00 sec  7.67 GBytes  65.9 Gbits/sec  62833/0          0     1342K/17 us
[ ID] Interval        Transfer    Bandwidth       Write/Err  Rtry    Cwnd/RTT
[  3] 7.00-8.00 sec  7.71 GBytes  66.2 Gbits/sec  63134/0          0     1342K/19 us
[ ID] Interval        Transfer    Bandwidth       Write/Err  Rtry    Cwnd/RTT
[  3] 8.00-9.00 sec  7.73 GBytes  66.4 Gbits/sec  63291/0          0     1342K/19 us
[ ID] Interval        Transfer    Bandwidth       Write/Err  Rtry    Cwnd/RTT
[  3] 9.00-10.00 sec  7.73 GBytes  66.4 Gbits/sec  63311/0          0     2110K/44 us
[ ID] Interval        Transfer    Bandwidth       Write/Err  Rtry    Cwnd/RTT
[  3] 0.00-10.00 sec  70.9 GBytes  60.9 Gbits/sec  580508/0          0     2110K/44 us

解决方法

1. 调整TCP窗口大小

从测试输出可见，跨机测试的TCP窗口仅85KB，远小于回环测试的2.5MB，这是带宽受限的核心原因——过小的TCP窗口会导致发送端频繁等待ACK，无法充分利用链路带宽。

iperf参数调整：测试时手动指定更大的TCP窗口，客户端和服务器端需同步设置：
```
# 服务器端
iperf -s -w 4M
# 客户端
iperf -c <服务器IP> -t 10 -w 4M
```
可根据带宽延迟积（BDP）计算最优窗口：BDP = 带宽(Gbps) × 延迟(ms) × 125（1Gbps=125MB/s），按你的链路延迟约300us=0.3ms计算，30Gbps的BDP为30×0.3×125=1125KB，设置4M窗口完全足够。

系统层面优化：确保TCP窗口缩放开启，并调整系统默认内存参数：

sysctl -w net.ipv4.tcp_window_scaling=1
sysctl -w net.ipv4.tcp_mem="4096 87380 16777216"
sysctl -w net.ipv4.tcp_wmem="4096 16384 16777216"
sysctl -w net.ipv4.tcp_rmem="4096 87380 16777216"

若要永久生效，将上述参数写入/etc/sysctl.conf后执行sysctl -p。

2. 绑定中断与iperf进程到同一CPU核心

单CPU测试时，网卡中断若未绑定到测试核心，会导致CPU在进程和中断处理间频繁切换，浪费资源。

查看网卡中断号：
```
cat /proc/interrupts | grep mlx5
```
找到CX5网卡对应的irq号（如mlx5_comp开头的条目）。
绑定中断到目标核心：
假设绑定到核心0，执行：
```
echo 1 > /proc/irq/<irq号>/smp_affinity_list
```
（核心编号对应掩码：核心0为1，核心1为2，核心2为4，以此类推）

绑定iperf进程到同一核心：

# 客户端
taskset -c 0 iperf -c <服务器IP> -t 10 -w 4M
# 服务器端
taskset -c 0 iperf -s -w 4M

3. 启用网卡硬件卸载功能

CX5网卡支持多种硬件卸载，可将数据包封装/解封装工作从CPU转移到网卡，减轻CPU负载。

检查当前卸载状态：
```
ethtool -k <网卡名>
```
查看tso、gso、rx-udp-gro-forwarding、tx-udp-segmentation等选项是否为on。

开启硬件卸载：

ethtool -K <网卡名> tso on gso on rx on tx on rx-udp-gro-forwarding on tx-udp-segmentation on

4. 排查交换机配置

确认交换机端口设置为100G全双工模式，无带宽限制。
关闭端口流量控制（Pause Frame）：客户端和服务器端执行ethtool -A <网卡名> rx off tx off，同时在交换机端关闭对应端口的流量控制，避免因帧暂停导致带宽下降。
检查交换机是否有QoS策略限制该端口带宽，确保策略未生效。

5. 更新网卡驱动与固件

Ubuntu 18.04默认的Mellanox驱动版本较旧，可能存在性能瓶颈。

安装最新OFED驱动：
添加Mellanox官方源后执行：
```
apt update && apt install mlnx-ofed-kernel-dkms
```
更新网卡固件：
```
mstfwmanager -u
```
执行前需安装mstflint工具：apt install mstflint

内容的提问来源于stack exchange，提问作者pomelo

单Iperf线程无法占满CPU？100G带宽测试瓶颈求助

跨机测试输出 #

本地回环测试输出 #

1. 调整TCP窗口大小 #

2. 绑定中断与iperf进程到同一CPU核心 #

3. 启用网卡硬件卸载功能 #

4. 排查交换机配置 #

5. 更新网卡驱动与固件 #