ip_reassembly吞吐量偏低问题咨询及测试详情

问题概述 #

测试DPDK的ip_reassembly示例时遇到吞吐量瓶颈：

• PC1（搭载10Gb/s ixgbe网卡）运行ip_reassembly
• PC2（1Gb/s网卡，MTU=1500）通过iperf发送1600字节UDP数据包（触发IP分片），最高仅能跑到900Mbps
• PC2累计发送2211842个数据报，ip_reassembly仅成功重组872727个；将发送速率降至800Mbps时无丢包现象

测试环境与执行细节 #

硬件配置 #

• PC1：10Gb/s网卡（PCI设备0000:04:00.0/0000:04:00.1，驱动为net_ixgbe）
• PC2：1Gb/s网卡，MTU设置为1500

启动命令与日志 #

PC1启动ip_reassembly：

./build/ip_reassembly -l 2 -- -p 0x1 &

启动输出：

EAL: Detected 4 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: No available hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:00:1f.6 on NUMA socket -1
EAL:   Invalid NUMA socket, default to 0
EAL:   probe driver: 8086:15b7 net_e1000_em
EAL: PCI device 0000:04:00.0 on NUMA socket -1
EAL:   Invalid NUMA socket, default to 0
EAL:   probe driver: 8086:10fb net_ixgbe
EAL: PCI device 0000:04:00.1 on NUMA socket -1
EAL:   Invalid NUMA socket, default to 0
EAL:   probe driver: 8086:10fb net_ixgbe
IP_RSMBL: Creating LPM table on socket 0
IP_RSMBL: Creating LPM6 table on socket 0
USER1: rte_ip_frag_table_create: allocated of 25165952 bytes at socket 0
Initializing port 0 ... Port 0 modified RSS hash function based on hardware support,requested:0xa38c configured:0x8104
 Address:00:1B:21:C1:E9:C6
txq=2,0,0 

IP_RSMBL: Socket 0: adding route 100.10.0.0/16 (port 0)
IP_RSMBL: Socket 0: adding route 100.20.0.0/16 (port 1)
IP_RSMBL: Socket 0: adding route 100.30.0.0/16 (port 2)
IP_RSMBL: Socket 0: adding route 100.40.0.0/16 (port 3)
IP_RSMBL: Socket 0: adding route 100.50.0.0/16 (port 4)
IP_RSMBL: Socket 0: adding route 100.60.0.0/16 (port 5)
IP_RSMBL: Socket 0: adding route 100.70.0.0/16 (port 6)
IP_RSMBL: Socket 0: adding route 100.80.0.0/16 (port 7)
IP_RSMBL: Socket 0: adding route 0101:0101:0101:0101:0101:0101:0101:0101/48 (port 0)
IP_RSMBL: Socket 0: adding route 0201:0101:0101:0101:0101:0101:0101:0101/48 (port 1)
IP_RSMBL: Socket 0: adding route 0301:0101:0101:0101:0101:0101:0101:0101/48 (port 2)
IP_RSMBL: Socket 0: adding route 0401:0101:0101:0101:0101:0101:0101:0101/48 (port 3)
IP_RSMBL: Socket 0: adding route 0501:0101:0101:0101:0101:0101:0101:0101/48 (port 4)
IP_RSMBL: Socket 0: adding route 0601:0101:0101:0101:0101:0101:0101:0101/48 (port 5)
IP_RSMBL: Socket 0: adding route 0701:0101:0101:0101:0101:0101:0101:0101/48 (port 6)
IP_RSMBL: Socket 0: adding route 0801:0101:0101:0101:0101:0101:0101:0101/48 (port 7)

Checking link status
done
Port0 Link Up. Speed 10000 Mbps - full-duplex
IP_RSMBL: entering main loop on lcore 2
IP_RSMBL:  -- lcoreid=2 portid=0

PC2执行iperf命令：

iperf -c 192.168.10.157 -i 1 -u -t 30 -p 2152 -b 900M -l 1600

iperf输出：

------------------------------------------------------------
Client connecting to 192.168.10.157, UDP port 2152
Sending 1600 byte datagrams, IPG target: 13.56 us (kalman adjust)
UDP buffer size:  958 MByte (default)
------------------------------------------------------------
[  3] local 192.168.10.100 port 37771 connected with 192.168.10.157 port 2152
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 1.0 sec   113 MBytes   944 Mbits/sec
[  3]  1.0- 2.0 sec   112 MBytes   944 Mbits/sec
[  3]  2.0- 3.0 sec   112 MBytes   944 Mbits/sec
[  3]  3.0- 4.0 sec   113 MBytes   944 Mbits/sec
[  3]  4.0- 5.0 sec   112 MBytes   944 Mbits/sec
[  3]  5.0- 6.0 sec   112 MBytes   944 Mbits/sec
[  3]  6.0- 7.0 sec   113 MBytes   944 Mbits/sec
[  3]  7.0- 8.0 sec   112 MBytes   944 Mbits/sec
[  3]  8.0- 9.0 sec   113 MBytes   944 Mbits/sec
[  3]  9.0-10.0 sec   112 MBytes   944 Mbits/sec
[  3] 10.0-11.0 sec   112 MBytes   944 Mbits/sec
[  3] 11.0-12.0 sec   112 MBytes   944 Mbits/sec
[  3] 12.0-13.0 sec   112 MBytes   944 Mbits/sec
[  3] 13.0-14.0 sec   112 MBytes   944 Mbits/sec
[  3] 14.0-15.0 sec   113 MBytes   944 Mbits/sec
[  3] 15.0-16.0 sec   112 MBytes   944 Mbits/sec
[  3] 16.0-17.0 sec   113 MBytes   944 Mbits/sec
[  3] 17.0-18.0 sec   112 MBytes   944 Mbits/sec
[  3] 18.0-19.0 sec   112 MBytes   944 Mbits/sec
[  3] 19.0-20.0 sec   112 MBytes   944 Mbits/sec
[  3] 20.0-21.0 sec   113 MBytes   944 Mbits/sec
[  3] 21.0-22.0 sec   112 MBytes   944 Mbits/sec
[  3] 22.0-23.0 sec   112 MBytes   944 Mbits/sec
[  3] 23.0-24.0 sec   112 MBytes   944 Mbits/sec
[  3] 24.0-25.0 sec   112 MBytes   944 Mbits/sec
[  3] 25.0-26.0 sec   113 MBytes   944 Mbits/sec
[  3] 26.0-27.0 sec   112 MBytes   944 Mbits/sec
[  3] 27.0-28.0 sec   112 MBytes   944 Mbits/sec
[  3] 28.0-29.0 sec   113 MBytes   944 Mbits/sec
[  3] 29.0-30.0 sec   113 MBytes   944 Mbits/sec
[  3] WARNING: did not receive ack of last datagram after 10 tries.
[  3]  0.0-30.0 sec  3.30 GBytes   944 Mbits/sec
[  3] Sent 2211842 datagrams

分片重组统计信息 #

发送kill -SIGUSR1 <pid>获取ip_reassembly统计：

# ps
  PID TTY          TIME CMD
  335 pts/9    00:02:35 ip_reassembly
  535 pts/9    00:00:00 ps
25304 pts/9    00:00:00 su
25306 pts/9    00:00:00 zsh
# kill -SIGUSR1 335
 -- lcoreid=2 portid=0 frag tbl stat:
max entries:    4096;
entries in use: 4088;
finds/inserts:  4344078;                                                                                                                        
entries added:  883521;
entries deleted by timeout:     837;
entries reused by timeout:      0;
total add failures:     2581961;
add no-space failures:  2581961;
add hash-collisions failures:   0;
TX bursts:      0
TX packets _queued:     0
TX packets dropped:     0
TX packets send:    0
RX gtpu packets:        872727

根因分析 #

从统计数据中add no-space failures: 2581961可以明确：分片表的默认容量（4096条目）不足以支撑900Mbps的分片流量，导致大量新分片无法插入表中，最终无法完成重组。

当流量速率降至800Mbps时，分片生成速率降低，分片表有足够时间回收超时条目，因此无丢包。

解决建议 #

1. 扩容分片表：修改ip_reassembly示例中rte_ip_frag_table_create的max_entries参数，建议调整为16384或更高（需确保系统有足够的大页内存分配）。
2. 优化超时配置：缩短分片的超时时间（默认通常为2秒），让分片表中的条目更快被回收复用，减少表满的概率。
3. 多核并行处理：当前仅使用lcore 2处理流量，可修改启动命令（如-l 2,3）并调整代码，将不同队列的流量分配到多个lcore处理，提升整体处理能力。
4. 优化RSS哈希：当前RSS哈希配置（requested:0xa38c configured:0x8104）可能未对分片数据包做均匀分发，可调整网卡RSS配置，确保同一IP流的分片被分发到同一队列，同时让不同流的分片均匀分布到多个队列，避免单队列过载。

内容的提问来源于stack exchange，提问作者happy