已配置足够权限的服务账号调用Delete Instance API仍报403错误求助

Hey, let's figure out this annoying 403 issue you're dealing with—it's so frustrating when the Policy Troubleshooter says you have the right permissions but the actual API call still fails! Here are the most likely things to check:

1. Make sure your K8s workload is using the correct service account #

This is the most common gotcha in GCP + K8s setups. Your K8s pods (or the node itself, if you're running the call directly on the node) might not be authenticating with the preemptible-killer service account you configured.

• If running directly on the node: Run gcloud auth list to see which account is active. If it's the node's default service account (like {project-number}-compute@developer.gserviceaccount.com), that's why you're getting 403—you're using the wrong identity.
• If running in a pod: Double-check your workload identity setup (if using GKE) or confirm the pod is mounting the correct service account key secret. You can exec into the pod and run echo $GOOGLE_APPLICATION_CREDENTIALS to verify the key path, or use curl -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/email to see which account the pod is using.

2. Test the service account directly on the node #

Let's bypass K8s to isolate the problem:

1. Activate the preemptible-killer service account on the node:

   gcloud auth activate-service-account preemptible-killer@{project}.iam.gserviceaccount.com --key-file=/path/to/your/service-account-key.json
   

2. Try deleting the instance directly with this account:

   gcloud compute instances delete {instance} --project={project}
   

If this works, the issue is definitely in how K8s is handling the service account. If it still fails, then we need to dig deeper into IAM policies.

3. Check for organizational deny policies or VPC Service Controls #

The Policy Troubleshooter doesn't always show deny policies that are set at the folder or organization level.

• Head to the IAM page in the GCP Console, switch to the Deny policies tab, and check if there's any rule blocking compute.instances.delete for your service account.
• If your project is in a VPC Service Controls perimeter, make sure Compute Engine API is allowed in the perimeter, and your K8s nodes are in the trusted IP range for that perimeter.

4. Wait for IAM permissions to propagate #

While usually fast, GCP IAM can have a propagation delay (1-5 minutes, sometimes longer for custom roles). If you just added the Editor role or updated the Preemptible Killer custom role, wait 10 minutes and try again—sometimes that's all it takes.

5. Check the instance's individual IAM policy #

Rarely, an instance might have its own IAM bindings that override project-level permissions. Run this to check:

gcloud compute instances get-iam-policy {instance} --project={project}

Look for any deny bindings or restrictive roles that might be blocking your service account from deleting the instance.

内容的提问来源于stack exchange，提问作者donzul