Linux iptables环境下NAT打洞失败问题排查

我正在测试一款自定义NAT打洞库，用Docker搭建了包含3个子网的测试模拟器：

• 两个NAT子网（172.30.0.0/16和172.31.0.0/16），各部署一台Debian机器作为NAT实例，运行指定的iptables配置脚本；
• 一个“公网”子网（172.29.0.0/16），两台NAT实例分别拥有IP：172.29.0.3和172.29.0.4。

目标：让172.30.0.10节点与172.31.0.10节点通信。

NAT实例的iptables配置脚本 #

#!/bin/bash

# Required env:
# NATED_SUBNET the source subnet
# OUTPUT_IP the ip of the output ("public" network) interface
# INPUT_IP the ip of the input (NATed network) interface
# OUTPUT_GATEWAY_IP the ip of the gateway in the output network

set -e -o xtrace

OUTPUT_IFNAME=$(ip -br addr show | grep $OUTPUT_IP | cut -d "@" -f 1)

# enable access to public internet
ip route replace default via $OUTPUT_GATEWAY_IP dev $OUTPUT_IFNAME

# setup syslogs
iptables -I FORWARD -s $NATED_SUBNET -d 0/0 -j NFLOG --nflog-prefix "[nated|FORWARD|OUT]:" --nflog-group 1
iptables -I FORWARD -s 0/0 -d $NATED_SUBNET -j NFLOG --nflog-prefix "[nated|FORWARD|IN]:" --nflog-group 1
iptables -N LOG_DROP 
iptables -A LOG_DROP -j NFLOG --nflog-prefix "[nated|DROP]:" --nflog-group 1
iptables -A LOG_DROP -j DROP

iptables -t nat -A POSTROUTING -s $NATED_SUBNET -o $OUTPUT_IFNAME -j MASQUERADE

iptables -A INPUT -j LOG_DROP

运行日志 #

docker-router2-1  | Feb  5 18:21:11 69e0e4683741 [nated|FORWARD|OUT]: IN=eth0 OUT=eth1 MAC=02:42:ac:1f:00:03:02:42:ac:1f:00:0a:08:00 SRC=172.31.0.10 DST=172.29.0.3 LEN=60 TOS=00 PREC=0x00 TTL=63 ID=35566 DF PROTO=TCP SPT=8080 DPT=56943 SEQ=3939548479 ACK=0 WINDOW=64240 SYN URGP=0 MARK=0 
docker-router1-1  | Feb  5 18:21:11 86ed5150c6b9 [nated|DROP]: IN=eth1 OUT= MAC=02:42:ac:1d:00:03:02:42:ac:1d:00:04:08:00 SRC=172.29.0.4 DST=172.29.0.3 LEN=60 TOS=00 PREC=0x00 TTL=63 ID=35566 DF PROTO=TCP SPT=8080 DPT=56943 SEQ=3939548479 ACK=0 WINDOW=64240 SYN URGP=0 MARK=0 
docker-router1-1  | Feb  5 18:21:12 86ed5150c6b9 [nated|FORWARD|OUT]: IN=eth0 OUT=eth1 MAC=02:42:ac:1e:00:03:02:42:ac:1e:00:0a:08:00 SRC=172.30.0.10 DST=172.29.0.4 LEN=60 TOS=00 PREC=0x00 TTL=63 ID=7623 DF PROTO=TCP SPT=56943 DPT=8080 SEQ=491450156 ACK=0 WINDOW=64240 SYN URGP=0 MARK=0 
docker-router1-1  | Feb  5 18:21:12 86ed5150c6b9 [nated|DROP]: IN=eth1 OUT= MAC=02:42:ac:1d:00:03:02:42:ac:1d:00:04:08:00 SRC=172.31.0.10 DST=172.29.0.3 LEN=60 TOS=00 PREC=0x00 TTL=63 ID=0 DF PROTO=TCP SPT=8080 DPT=56943 SEQ=3955213225 ACK=491450157 WINDOW=65160 ACK SYN URGP=0 MARK=0 
docker-router2-1  | Feb  5 18:21:12 69e0e4683741 [nated|FORWARD|IN]: IN=eth1 OUT=eth0 MAC=02:42:ac:1d:00:04:02:42:ac:1d:00:03:08:00 SRC=172.29.0.3 DST=172.31.0.10 LEN=60 TOS=00 PREC=0x00 TTL=62 ID=7623 DF PROTO=TCP SPT=56943 DPT=8080 SEQ=491450156 ACK=0 WINDOW=64240 SYN URGP=0 MARK=0 
docker-router2-1  | Feb  5 18:21:12 69e0e4683741 [nated|FORWARD|OUT]: IN=eth0 OUT=eth1 MAC=02:42:ac:1f:00:03:02:42:ac:1f:00:0a:08:00 SRC=172.31.0.10 DST=172.29.0.3 LEN=60 TOS=00 PREC=0x00 TTL=63 ID=0 DF PROTO=TCP SPT=8080 DPT=56943 SEQ=3955213225 ACK=491450157 WINDOW=65160 ACK SYN URGP=0 MARK=0 

注：日志顺序因Docker Compose收集不同主机日志存在偏差。

日志观察结论 #

• docker-router2-1发出的首个打洞SYN包被docker-router1-1正常丢弃；
• docker-router1-1后的客户端发起的连接借助打洞成功通过docker-router2-1；
• docker-router2-1后的服务器返回SYN-ACK包，符合预期；
• 但docker-router2-1未正确替换SYN-ACK包的源IP——docker-router1-1收到的包源IP为172.31.0.10而非172.29.0.4，导致被丢弃。

conntrack输出 #

simulator-router2-1 | tcp 6 118 SYN_SENT2 src=172.31.0.10 dst=172.29.0.3 sport=8080 dport=42773 src=172.29.0.3 dst=172.29.0.4 sport=42773 dport=8080 mark=0 use=1
simulator-router1-1 | tcp 6 110 SYN_SENT src=172.30.0.10 dst=172.29.0.4 sport=42773 dport=8080 [UNREPLIED] src=172.29.0.4 dst=172.29.0.3 sport=8080 dport=42773 mark=0 use=1

当前配置存在什么问题？

内容的提问来源于stack exchange，提问作者Remi D