Kubernetes拉取apache/zeppelin镜像时authorization failed报错求助
Since your Spark pods deploy without issues, this problem is isolated to the Zeppelin image or how your cluster interacts with Docker Hub for this specific repository. Let's walk through targeted troubleshooting steps:
1. Verify Direct Image Pull on Cluster Nodes
First, rule out node-level connectivity or Docker Hub issues by pulling the image directly on one of your cluster nodes:
docker pull apache/zeppelin:0.9.0
- If this fails with the same
invalid_tokenerror: The issue likely stems from Docker Hub access on the node. Try logging into Docker Hub with a valid account (docker login) and re-run the pull—anonymous pull limits or temporary token glitches can cause this even for public images. - If this succeeds: The problem is specific to Kubernetes' image pull mechanism in your cluster.
2. Check Kubernetes Image Pull Secrets
Even though other public images work, ensure your cluster isn't using a stale or invalid image pull secret that's interfering with this image:
- List existing secrets:
kubectl get secrets - Check if your Zeppelin deployment/pod uses an
imagePullSecret(inspect your deployment YAML or describe the pod):kubectl describe pod zeppelin-server-6f5646f56d-6slj2
If a secret is configured, verify its validity by decoding the dockerconfigjson data:
kubectl get secret <secret-name> -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d
Ensure the auth token for Docker Hub isn't expired. If you're using an anonymous secret (unlikely), remove it and try pulling again.
3. Test with Image Digest Instead of Tag
Sometimes, image tag manifests can have temporary issues on Docker Hub. Try pulling using the specific digest for apache/zeppelin:0.9.0 (you can find this on Docker Hub's repository page or via docker inspect apache/zeppelin:0.9.0):
kubectl run zeppelin --image=apache/zeppelin@sha256:[your-digest-here] --restart=Never
If this works, the tag's manifest might be corrupted or have authorization issues on Docker Hub's end.
4. Check for Docker Hub Rate Limiting
Docker Hub imposes strict rate limits on anonymous pulls (100 pulls/6 hours per IP). If your cluster is pulling many images anonymously, you might be hitting this limit—even if the error message mentions invalid_token, rate limiting can sometimes manifest this way.
- To confirm, log into Docker Hub on your cluster nodes with a free account (which raises limits to 200 pulls/6 hours) and retry the pull.
5. Validate Cluster Registry Mirror Configuration
If your k3d cluster uses a registry mirror (like a local proxy), the mirror might be caching an invalid token for the Zeppelin image. Try temporarily disabling the mirror and pulling directly from Docker Hub to rule this out.
内容的提问来源于stack exchange，提问作者Jordi
