启用Stackdriver监控导致metadata-agent Pod崩溃问题求助

问题现象 #

在GKE 1.11.6-gke.3集群中，通过Cloud Console事后启用Stackdriver监控后，metadata-agent Pod出现异常频繁的重启现象，部分Pod重启次数甚至高达数百次：

➜ kubectl get pods --namespace=kube-system | grep metadata-agent
NAME READY STATUS RESTARTS AGE
metadata-agent-cluster-level-579ffb7c5f-vm8q8 1/1 Running 908 3d
metadata-agent-gdnb6 1/1 Running 908 3d
metadata-agent-q7vct 1/1 Running 885 3d
metadata-agent-rcfl8 1/1 Running 907 3d
metadata-agent-vvtss 1/1 Running 908 3d
metadata-agent-zvz6f 1/1 Running 816 3d

查看metadata-agent的日志，发现核心报错是持续的403 Forbidden，无法从metadata server获取授权令牌：

➜ kubectl logs pods/metadata-agent-gdnb6 --namespace=kube-system
I0130 10:32:38 7eff97c7f740 updater.cc:40 Not starting DockerUpdater
I0130 10:32:38 7eff97c7f740 kubernetes.cc:1324 Watching for node-level metadata
I0130 10:32:38 7eff94e58700 kubernetes.cc:1163 Watch thread (pods) started for node gke-rain-rain-node-pool-16891a38-p99s
I0130 10:32:38 7eff8effd700 kubernetes.cc:1203 Watch thread (node) started for node gke-rain-rain-node-pool-16891a38-p99s
I0130 10:32:38 7eff7ffff700 reporter.cc:46 Metadata reporter started
I0130 10:32:41 7eff7ffff700 environment.cc:270 No credentials found at /etc/google/auth/application_default_credentials.json
I0130 10:32:41 7eff7ffff700 environment.cc:146 Got project id from metadata server: 11111111
I0130 10:32:41 7eff7ffff700 oauth2.cc:283 Getting auth token from metadata server
E0130 10:32:41 7eff7ffff700 reporter.cc:64 Metadata request unsuccessful: Server responded with 'Forbidden' (403): Transport endpoint is not connected
E0130 10:33:41 7eff7ffff700 reporter.cc:64 Metadata request unsuccessful: Server responded with 'Forbidden' (403): Transport endpoint is not connected

补充背景：仅在集群创建完成后事后启用Stackdriver监控时出现该问题，集群创建阶段直接启用则无此异常。

问题根源 #

当在集群创建时启用Stackdriver监控，GCP会自动完成全套配置：为metadata-agent关联的服务账号绑定monitoring.metricWriter、logging.logWriter等必要IAM角色，同时正确配置节点的metadata访问规则。但事后启用监控时，部分权限配置步骤会被遗漏，导致metadata-agent无法通过metadata server获取合法授权令牌，触发Pod的重启机制。

解决方案 #

1. 补全服务账号IAM权限 #

首先定位metadata-agent使用的服务账号：

kubectl describe pod metadata-agent-gdnb6 -n kube-system | grep "Service Account"

通常这个服务账号名为metadata-agent，接下来为其添加所需的IAM角色（替换YOUR_PROJECT_ID为你的项目ID）：

gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \
  --member=serviceAccount:metadata-agent@YOUR_PROJECT_ID.iam.gserviceaccount.com \
  --role=roles/monitoring.metricWriter

gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \
  --member=serviceAccount:metadata-agent@YOUR_PROJECT_ID.iam.gserviceaccount.com \
  --role=roles/logging.logWriter

2. 重启监控组件刷新配置 #

强制重启metadata-agent的DaemonSet和Deployment，让组件加载新的权限配置：

kubectl rollout restart daemonset metadata-agent -n kube-system
kubectl rollout restart deployment metadata-agent-cluster-level -n kube-system

3. 验证修复效果 #

等待Pod重启完成后，检查Pod状态和日志：

# 查看Pod重启次数是否停止增长
kubectl get pods -n kube-system | grep metadata-agent

# 查看最新日志是否无403错误
kubectl logs metadata-agent-gdnb6 -n kube-system --tail=20

如果日志中不再出现Forbidden报错，且RESTARTS数值不再持续增加，说明问题已解决。

内容的提问来源于stack exchange，提问作者dolftax