创建GKE集群时节点启动失败，出现connection refused错误

问题现象 #

集群创建完成后节点无法正常启动，节点串口1日志显示无法下载GKE初始化组件：

[   24.630930] configure.sh[1010]: Getting the service account access token configured for VM. 
[   24.673381] configure.sh[1010]: Service account access token is received. Downloading https://storage.googleapis.com/gke-release/npd-custom-plugins/v1.0.4/npd-custom-plugins-v1.0.4.tar.gz using this token. 
[   24.969664] configure.sh[1116]: curl: (7) Failed to connect to storage.googleapis.com port 443 after 292 ms: Connection refused 
[   24.969859] configure.sh[1116]: Warning: Problem : connection refused. Will retry in 10 seconds. 6 retries.
[   66.167634] configure.sh[1116]: Warning: left. 
[   76.459028] configure.sh[1116]: curl: (7) Failed to connect to storage.googleapis.com port 443 after 281 ms: Connection refused 
[   76.459198] configure.sh[1116]: Warning: Problem : connection refused. Will retry in 10 seconds. 1 retries 
[   76.459252] configure.sh[1116]: Warning: left. 
[   86.759856] configure.sh[1116]: curl: (7) Failed to connect to storage.googleapis.com port 443 after 290 ms: Connection refused 
[   86.760211] configure.sh[1010]: == Failed to download https://storage.googleapis.com/gke-release/npd-custom-plugins/v1.0.4/npd-custom-plugins-v1.0.4.tar.gz. Retrying.

已配置的服务账号权限 #

• service-abcde@container-engine-robot.iam.gserviceaccount.com：
  • Compute Network User
  • Compute Network Admin
  • Compute Security Admin
  • Kubernetes Engine Host Service Agent User
  • Service Networking Service Agent
• abcde@cloudservices.gserviceaccount.com：
  • Compute Network User

集群创建命令 #

gcloud container clusters create poc-65 --project project-xyz \
  --zone=us-west1-a --enable-ip-alias \
  --network projects/project-xyz/global/networks/network-xyz \
  --subnetwork projects/project-xyz/regions/us-west1/subnetworks/subnet-xyz \
  --services-secondary-range-name uat-xyz-service \
  --cluster-secondary-range-name uat-xyz-pod \
  --no-enable-master-authorized-networks \
  --master-ipv4-cidr 172.16.2.32/28 \
  --enable-private-nodes --num-nodes 3 \
  --default-max-pods-per-node 20

解决方案 #

核心问题是私有节点无法访问storage.googleapis.com，这是GKE节点初始化的必要外部依赖，按以下步骤排查修复：

1. 检查云NAT配置 #

因为启用了--enable-private-nodes，私有节点无公网IP，必须通过云NAT访问外部资源：

• 确认集群所在VPC已配置云NAT网关，且网关关联了节点所在子网。
• 确保NAT网关允许访问storage.googleapis.com的IP段（或直接允许所有外部IP）。

2. 验证防火墙规则 #

检查VPC防火墙是否允许节点出站访问443端口：

• 确认存在允许节点子网IP段向0.0.0.0/0发起HTTPS（443端口）出站流量的规则。
• GKE默认会创建必要规则，若自定义规则覆盖了默认配置，需调整规则优先级或补充HTTPS出站许可。

3. 补充服务账号权限 #

给service-abcde@container-engine-robot.iam.gserviceaccount.com添加Storage Object Viewer角色，确保节点能下载GCS中的GKE组件包。

4. 检查DNS配置 #

验证私有节点能否正确解析storage.googleapis.com：

• 确认VPC使用Google Cloud DNS，或自定义DNS服务器可正常解析该域名。
• 可通过堡垒机或SSH转发登录节点，执行nslookup storage.googleapis.com验证解析结果。

内容的提问来源于stack exchange，提问作者Lu Xin