Google Compute Engine实例配置Cloud NAT后仍无法访问公网的故障排查请求

大家好，我碰到了一个头疼的问题：我们打算给一台Compute Engine实例做维护（通过apt update/upgrade更新依赖），但这台实例完全没法访问公网——哪怕我们已经在对应的VPC里配置了Cloud NAT。

问题现象 #

执行sudo apt update时，出现连接超时错误：

Err:1 http://security.ubuntu.com/ubuntu jammy-jellyfish-security InRelease
Could not connect to security.ubuntu.com:80 (185.125.190.39), connection timed out Could not connect to security.ubuntu.com:80 (91.189.91.39), connection timed out Could not connect to security.ubuntu.com:80 (185.125.190.36), connection timed out
...

尝试ping公网IP也完全不通：

ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
^C
--- 8.8.8.8 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4096ms

我们自己排查了很久都没找到原因，下面是收集到的所有环境信息，麻烦大家帮忙看看问题出在哪！

实例基本信息 #

Cloud NAT与Cloud Router配置 #

Cloud NAT网关 #

Cloud Router #

防火墙规则 #

默认防火墙规则 #

测试用新增防火墙规则（未解决问题） #

实例内部网络配置 #

/etc/hosts内容 #

127.0.0.1 localhost
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
169.254.169.254 metadata.google.internal metadata

/etc/resolv.conf内容 #

nameserver 127.0.0.53
options edns0
search us-central1-c.c.project-name.internal c.project-name.internal google.internal

ifconfig输出 #

ens4: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1460
inet 10.128.0.2  netmask 255.255.255.255  broadcast 0.0.0.0
inet6 fe80::4001:aff:fe80:2  prefixlen 64  scopeid 0x20<link>
ether 42:01:0a:80:00:02  txqueuelen 1000  (Ethernet)
RX packets 9167  bytes 4126195 (4.1 MB)
RX errors 0  dropped 0  overruns 0  frame 0
TX packets 10106  bytes 1160860 (1.1 MB)
TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
inet 127.0.0.1  netmask 255.0.0.0
inet6 ::1  prefixlen 128  scopeid 0x10<host>
loop  txqueuelen 1000  (Local Loopback)
RX packets 640  bytes 61620 (61.6 KB)
RX errors 0  dropped 0  overruns 0  frame 0
TX packets 640  bytes 61620 (61.6 KB)
TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ip rule list输出 #

0:      from all lookup local
32766:  from all lookup main
32767:  from all lookup default

ip route show table all输出 #

default via 10.128.0.1 dev ens4 proto dhcp src 10.128.0.2 metric 100
10.128.0.1 dev ens4 proto dhcp scope link src 10.128.0.2 metric 100
local 10.128.0.2 dev ens4 table local proto kernel scope host src 10.128.0.2
broadcast 127.0.0.0 dev lo table local proto kernel scope link src 127.0.0.1
local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1
local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1
::1 dev lo proto kernel metric 256 pref medium
fe80::/64 dev ens4 proto kernel metric 256 pref medium
local ::1 dev lo table local proto kernel metric 0 pref medium
local fe80::4001:aff:fe80:2 dev ens4 table local proto kernel metric 0 pref medium
ff00::/8 dev ens4 table local metric 256 pref medium

dig google.com输出 #

; <<>> DiG 9.11.3-1ubuntu1.11-Ubuntu <<>> google.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 34116
;; flags: qr rd ra; QUERY: 1, ANSWER: 6, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494

;; QUESTION SECTION:
;google.com.                    IN      A

;; ANSWER SECTION:
google.com.             300     IN      A       142.251.161.101
google.com.             300     IN      A       142.251.161.113
google.com.             300     IN      A       142.251.161.102
google.com.             300     IN      A       142.251.161.138
google.com.             300     IN      A       142.251.161.100
google.com.             300     IN      A       142.251.161.139

;; Query time: 19 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Sun May 07 21:28:29 UTC 2023
;; MSG SIZE  rcvd: 135

如果需要更多细节或者日志信息，我随时可以补充，谢谢大家的帮忙！

备注：内容来源于stack exchange，提问作者Philippe Hebert