Consul K8s客户端UDP探测失败问题排查求助

我们已在虚拟机（VM）上部署Consul Server集群，现尝试在另一台虚拟机的Kubernetes（K8s）环境中部署Consul Client。Consul Agent可成功加入集群，但出现UDP探测失败问题：日志显示可通过TCP连接Consul Server节点，但UDP探测失败，提示网络可能配置错误。使用Netcat检测确认UDP端口已开放，Dockerfile已暴露对应端口，K8s层面仅配置了如下PodSpec，未进行额外端口转发或暴露操作。

日志数据 #

==> Log data will now stream in as it occurs:

2023-03-15T12:31:39.897Z [WARN]  agent: using enable-script-checks without ACLs and without allow_write_http_from is DANGEROUS, use enable-local-script-checks instead, see https://www.hashicorp.com/blog/protecting-consul-from-rce-risk-in-specific-configurations/
2023-03-15T12:31:39.903Z [WARN]  agent.auto_config: using enable-script-checks without ACLs and without allow_write_http_from is DANGEROUS, use enable-local-script-checks instead, see https://www.hashicorp.com/blog/protecting-consul-from-rce-risk-in-specific-configurations/
2023-03-15T12:31:39.904Z [INFO]  agent.client.serf.lan: serf: EventMemberJoin: consul POD_IP
2023-03-15T12:31:39.904Z [INFO]  agent.router: Initializing LAN area manager
2023-03-15T12:31:39.904Z [INFO]  agent: Started DNS server: address=127.0.0.1:82210 network=udp
2023-03-15T12:31:39.904Z [INFO]  agent: Started DNS server: address=127.0.0.1:82210 network=tcp
2023-03-15T12:31:39.904Z [INFO]  agent: Starting server: address=127.0.0.1:82208 network=tcp protocol=http
2023-03-15T12:31:39.904Z [INFO]  agent: Started gRPC listeners: port_name=grpc address=127.0.0.1:82209 network=tcp
2023-03-15T12:31:39.904Z [WARN]  agent.router.manager: No servers available
2023-03-15T12:31:39.904Z [ERROR] agent.anti_entropy: failed to sync remote state: error="No known Consul servers"
2023-03-15T12:31:39.904Z [INFO]  agent: started state syncer
2023-03-15T12:31:39.904Z [INFO]  agent: Consul agent running!
2023-03-15T12:31:39.904Z [INFO]  agent: Retry join is supported for the following discovery methods: cluster=LAN discovery_methods="aliyun aws azure digitalocean gce hcp k8s linode mdns os packet scaleway softlayer tencentcloud triton vsphere"
2023-03-15T12:31:39.904Z [INFO]  agent: Joining cluster...: cluster=LAN
2023-03-15T12:31:39.904Z [INFO]  agent: (LAN) joining: lan_addresses=["consul_server_2", "consul_server_1", "consul_server_3"]
2023-03-15T12:31:39.907Z [INFO]  agent.client.serf.lan: serf: EventMemberJoin: hostA consul_server_2
2023-03-15T12:31:39.907Z [INFO]  agent.client.serf.lan: serf: EventMemberJoin: hostB consul_server_3
2023-03-15T12:31:39.907Z [INFO]  agent.client.serf.lan: serf: EventMemberJoin: hostC consul_server_1
2023-03-15T12:31:39.908Z [INFO]  agent.client: adding server: server="hostA (Addr: tcp/consul_server_2:82205) (DC: dc1)"
2023-03-15T12:31:39.908Z [INFO]  agent.client: adding server: server="hostB (Addr: tcp/consul_server_3:82205) (DC: dc1)"
2023-03-15T12:31:39.908Z [INFO]  agent.client: adding server: server="hostC (Addr: tcp/consul_server_1:82205) (DC: dc1)"
2023-03-15T12:31:39.910Z [INFO]  agent: (LAN) joined: number_of_nodes=3
2023-03-15T12:31:39.910Z [INFO]  agent: Join cluster completed. Synced with initial agents: cluster=LAN num_agents=3
2023-03-15T12:31:41.905Z [WARN]  agent.client.memberlist.lan: memberlist: Was able to connect to hostC over TCP but UDP probes failed, network may be misconfigured
2023-03-15T12:31:42.778Z [INFO]  agent: Synced node info
2023-03-15T12:31:42.905Z [WARN]  agent.client.memberlist.lan: memberlist: Was able to connect to hostB over TCP but UDP probes failed, network may be misconfigured
2023-03-15T12:31:43.905Z [WARN]  agent.client.memberlist.lan: memberlist: Was able to connect to hostA over TCP but UDP probes failed, network may be misconfigured
2023-03-15T12:31:44.906Z [WARN]  agent.client.memberlist.lan: memberlist: Was able to connect to hostA over TCP but UDP probes failed, network may be misconfigured
2023-03-15T12:31:45.906Z [WARN]  agent.client.memberlist.lan: memberlist: Was able to connect to hostC over TCP but UDP probes failed, network may be misconfigured
2023-03-15T12:31:46.906Z [WARN]  agent.client.memberlist.lan: memberlist: Was able to connect to hostB over TCP but UDP probes failed, network may be misconfigured
2023-03-15T12:31:47.907Z [WARN]  agent.client.memberlist.lan: memberlist: Was able to connect to hostC over TCP but UDP probes failed, network may be misconfigured
2023-03-15T12:31:48.908Z [WARN]  agent.client.memberlist.lan: memberlist: Was able to connect to hostA over TCP but UDP probes failed, network may be misconfigured
2023-03-15T12:31:49.909Z [WARN]  agent.client.memberlist.lan: memberlist: Was able to connect to hostB over TCP but UDP probes failed, network may be misconfigured
2023-03-15T12:31:50.910Z [WARN]  agent.client.memberlist.lan: memberlist: Was able to connect to hostA over TCP but UDP probes failed, network may be misconfigured
2023-03-15T12:31:51.910Z [WARN]  agent.client.memberlist.lan: memberlist: Was able to connect to hostB over TCP but UDP probes failed, network may be misconfigured
2023-03-15T12:31:52.911Z [WARN]  agent.client.memberlist.lan: memberlist: Was able to connect to hostC over TCP but UDP probes failed, network may be misconfigured
2023-03-15T12:31:53.911Z [WARN]  agent.client.memberlist.lan: memberlist: Was able to connect to hostB over TCP but UDP probes failed, network may be misconfigured
2023-03-15T12:31:54.912Z [WARN]  agent.client.memberlist.lan: memberlist: Was able to connect to hostA over TCP but UDP probes failed, network may be misconfigured

PodSpec配置 #

apiVersion: v1
kind: Pod
metadata:
  name: consul
spec:
  containers:
  - name: consul
    image: consul_test:0.4
    ports:
apiVersion: v1
kind: Pod
metadata:
  name: consul
spec:
  containers:
  - name: consul
    image: consul_test:latest
    ports:
      - containerPort: 82205
        hostPort: 82205
        protocol: TCP
      - containerPort: 82206
        hostPort: 82206
        protocol: TCP
      - containerPort: 82206
        hostPort: 82206
        protocol: UDP  
      - containerPort: 82207
        hostPort: 82207
        protocol: TCP
      - containerPort: 82207
        hostPort: 82207
        protocol: UDP
      - containerPort: 82208
        hostPort: 82208
        protocol: TCP
      - containerPort: 82209
        hostPort: 82209
        protocol: TCP
      - containerPort: 82210
        hostPort: 82210
        protocol: TCP
      - containerPort: 82210
        hostPort: 82210
        protocol: UDP

Consul Agent配置 #

{
    "addresses": {
        "dns": "127.0.0.1",
        "grpc": "127.0.0.1",
        "http": "127.0.0.1",
        "https": "127.0.0.1"
    },
    "advertise_addr": "{{ GetInterfaceIP \"eth0\" }}",
    "advertise_addr_wan": "{{ GetInterfaceIP \"eth0\" }}",
    "bind_addr": "{{ GetInterfaceIP \"eth0\" }}",
    "client_addr": "127.0.0.1",
    "data_dir": "/var/lib/consul",
    "datacenter": "dc1",
    "disable_update_check": true,
    "domain": "consul",
    "enable_local_script_checks": true,
    "enable_script_checks": true,
    "enable_syslog": false,
    "encrypt": "string",
    "encrypt_verify_incoming": true,
    "encrypt_verify_outgoing": true,
    "log_level": "INFO",
    "performance": {
        "leave_drain_time": "5s",
        "raft_multiplier": 1,
        "rpc_hold_timeout": "7s"
    },
    "ports": {

"dns": 82210,
"grpc": 82209,
"http": 82208,
"https": -1,
"grpc_tls": -1,
"serf_lan": 82206,
"serf_wan": 82207,
"server": 82205

    },
    "raft_protocol": 3,
    "retry_interval": "30s",
    "retry_join": [
        "server1",
        "server2",
        "server3"
    ],
    "retry_max": 0,
    "server": false,
    "syslog_facility": "local0",
    "translate_wan_addrs": false,
    "ui_config": {
        "enabled": false
    }
}

内容的提问来源于stack exchange，提问作者Oxycash