基于OpenVPN实现Docker Swarm Overlay网络VXLAN路由问题咨询

Alright, let's get your mixed on-prem/AWS Docker Swarm overlay network fully functional. You already have service discovery working (container name pings go through), so we just need to iron out the VXLAN routing kinks to ensure end-to-end connectivity for all swarm operations. Here's a step-by-step breakdown tailored to your setup:

1. Ensure Docker Swarm Uses the Correct Advertise Addresses #

Docker Swarm needs to know which network interface to use for cluster control plane and overlay traffic. If your nodes joined the swarm using the wrong interface, this will break VXLAN packet routing.

• For your laptop node (node1):
  Update the swarm to use the OpenVPN tun0 IP that's reachable via the NAT gateway:

  docker swarm update --advertise-addr 172.16.0.6
  

• For AWS nodes (node2 & node3):
  Use their eth2 private IPs, which are reachable within AWS and the NAT gateway:

  # Node2
  docker swarm update --advertise-addr 10.0.30.62
  # Node3
  docker swarm update --advertise-addr 10.0.140.122
  

2. Verify VXLAN Port (UDP 4789) Connectivity #

Docker overlay networks rely on UDP port 4789 for VXLAN tunneling. All nodes must be able to reach each other on this port through your OpenVPN/NAT path:

• Test from node1 to node2:

  nc -uvz 10.0.30.62 4789
  

• Test from node2 to node1:

  nc -uvz 172.16.0.6 4789
  

If these fail, fix the following:

• OpenVPN Server: Add firewall rules allowing UDP 4789 traffic through the tun0 interface
• AWS Security Groups: Allow UDP 4789 between eth2 interfaces of EC2 nodes and the NAT gateway
• NAT Gateway: Ensure port forwarding/routing rules pass UDP 4789 between the 172.16.0.0/24 and 10.0.0.0/16 subnets

3. Add Static Routes for Overlay Subnet #

Your overlay subnet is 192.169.1.0/24—each node needs explicit routes to send traffic for this subnet to Docker's overlay bridge interface.

First, find your overlay network's bridge interface on each node:

docker network inspect <your-overlay-network-name> | grep -E "Bridge|Name"
# Look for a value like "br-abc123"

Add the static route on each node:

# Node1, Node2, Node3 (replace <bridge-interface> with your actual interface)
ip route add 192.169.1.0/24 dev <bridge-interface>

Critical NAT Gateway Routes #

Your NAT gateway needs to map individual container IPs to their respective nodes:

# On NAT gateway (172.16.0.1)
ip route add 192.169.1.10/32 via 172.16.0.6  # Route n1 traffic to node1
ip route add 192.169.1.9/32 via 10.0.30.62   # Route n2 traffic to node2
ip route add 192.169.1.12/32 via 10.0.140.122 # Route n3 traffic to node3

Also, enable IP forwarding on the NAT gateway to allow traffic transit:

echo 1 > /proc/sys/net/ipv4/ip_forward

4. Validate Overlay Network Configuration #

Double-check that your overlay network is properly configured for swarm mode:

docker network inspect <your-overlay-network-name>

Confirm these key values:

• Scope: swarm
• Driver: overlay
• IPAM.Config.Subnet: 192.169.1.0/24
• Attachable: true (if you need to manually attach containers outside swarm services)

5. Test Full Connectivity #

Run these tests to confirm everything works:

1. Inter-container IP ping:

   docker exec n1 ping 192.169.1.9
   docker exec n2 ping 192.169.1.12
   docker exec n3 ping 192.169.1.10
   

2. Swarm service deployment test:

   docker service create --name test-nginx --network <your-overlay-network-name> --replicas 3 nginx
   

   Then access the service from any container using curl test-nginx to verify service discovery and load balancing work.

6. Persist Routes Across Reboots #

To keep routes after node restarts:

• For Ubuntu/Debian: Add routes to /etc/netplan/00-installer-config.yaml
• For RHEL/CentOS: Add routes to /etc/sysconfig/network-scripts/ifcfg-<interface>
• For the NAT gateway: Add the static routes to a startup script like /etc/rc.local

内容的提问来源于stack exchange，提问作者dwjbosman