VMware上Anthos Kubernetes集群无法从registry.k8s.io拉取镜像

问题背景 #

在VMware上运行的Anthos集群（基于Ubuntu 18.04 containerd默认镜像，Anthos版本1.14.1-gke.39）拉取registry.k8s.io/csi-secrets-store/driver-crds:v1.3.3时出现以下错误：

Failed to pull image "registry.k8s.io/csi-secrets-store/driver-crds:v1.3.3": rpc error: code = Unknown desc = failed to pull and unpack image "registry.k8s.io/csi-secrets-store/driver-crds:v1.3.3": failed to resolve reference "registry.k8s.io/csi-secrets-store/driver-crds:v1.3.3": failed to do request: Head "https://europe-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/csi-secrets-store/driver-crds/manifests/v1.3.3": x509: certificate signed by unknown authority

已排除防火墙访问限制，同VLAN内标准Ubuntu 20.04.5节点可正常访问该镜像，推测问题源于Anthos节点镜像的可信证书缺失。

可行解决方法 #

1. 测试环境临时绕过证书验证 #

修改containerd配置，针对registry.k8s.io跳过证书验证：

• 在Anthos节点编辑/etc/containerd/config.toml，添加：

  [plugins."io.containerd.grpc.v1.cri".registry.configs."registry.k8s.io".tls]
    insecure_skip_verify = true
  

• 重启containerd服务：sudo systemctl restart containerd

注意：此方法降低安全性，仅适合测试场景，生产环境禁用。

2. 生产环境批量更新节点根证书 #

通过DaemonSet批量为所有节点补充缺失的根证书：

1. 提取缺失的根证书：在正常Ubuntu 20.04节点执行以下命令，提取最后一个证书块（根CA）保存为missing-ca.crt：

   openssl s_client -connect europe-west2-docker.pkg.dev:443 -showcerts
   

2. 创建ConfigMap存储证书：

   kubectl create configmap missing-ca-configmap --from-file=missing-ca.crt -n kube-system
   

3. 部署DaemonSet批量更新证书：
   创建以下YAML文件并应用：

   apiVersion: apps/v1
   kind: DaemonSet
   metadata:
     name: update-ca-certificates
     namespace: kube-system
   spec:
     selector:
       matchLabels:
         name: update-ca-certificates
     template:
       metadata:
         labels:
           name: update-ca-certificates
       spec:
         hostPID: true
         containers:
         - name: update-ca
           image: ubuntu:18.04
           command: ["sh", "-c"]
           args:
           - |
             cp /tmp/missing-ca.crt /host/usr/local/share/ca-certificates/
             chroot /host update-ca-certificates
             systemctl restart containerd
           volumeMounts:
           - name: host-certs
             mountPath: /host/usr/local/share/ca-certificates/
           - name: host-etc
             mountPath: /host/etc/ssl/certs/
           - name: host-systemd
             mountPath: /host/systemd/
           - name: ca-file
             mountPath: /tmp/
         volumes:
         - name: host-certs
           hostPath:
             path: /usr/local/share/ca-certificates/
         - name: host-etc
           hostPath:
             path: /etc/ssl/certs/
         - name: host-systemd
           hostPath:
             path: /run/systemd/
         - name: ca-file
           configMap:
             name: missing-ca-configmap
   

3. 升级Anthos节点镜像或版本 #

• 检查是否有更新的Ubuntu 18.04 Anthos节点镜像，谷歌可能已修复证书缺失问题；
• 升级Anthos到较新的稳定版本，新版本通常会包含更新的根证书库。

4. 切换到Google Artifact Registry拉取镜像 #

将registry.k8s.io的镜像同步到Google Artifact Registry（GAR），Anthos节点默认信任GAR的证书：

1. 同步镜像：

   gcloud artifacts docker images copy registry.k8s.io/csi-secrets-store/driver-crds:v1.3.3 us-central1-docker.pkg.dev/[你的项目ID]/[仓库名]/driver-crds:v1.3.3 --source-registry-hostname=registry.k8s.io
   

2. 修改集群中所有相关镜像引用为GAR地址。

内容的提问来源于stack exchange，提问作者user2399020