修改containerd的pause镜像地址后不生效，如何排查解决？

我已在/etc/containerd/config.toml中将Kubernetes默认pause镜像下载地址修改为registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.6，配置内容如下：

[root@k8sslave01 kubelet]# cat /etc/containerd/config.toml
#   Copyright 2018-2022 Docker Inc.

#   Licensed under the Apache License, Version 2.0 (the "License");
#   you may not use this file except in compliance with the License.
#   You may obtain a copy of the License at

#       http://www.apache.org/licenses/LICENSE-2.0

#   Unless required by applicable law or agreed to in writing, software
#   distributed under the License is distributed on an "AS IS" BASIS,
#   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
#   See the License for the specific language governing permissions and
#   limitations under the License.

#disabled_plugins = ["cri"]

#root = "/var/lib/containerd"
#state = "/run/containerd"
#subreaper = true
#oom_score = 0
[plugins."io.containerd.grpc.v1.cri".registry]
   config_path = "/etc/containerd/certs.d"

[plugins."io.containerd.grpc.v1.cri"]
  #sandbox_image = "k8s.m.daocloud.io/pause:3.6"
  sandbox_image = "registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.6"

#[grpc]
#  address = "/run/containerd/containerd.sock"
#  uid = 0
#  gid = 0

#[debug]
#  address = "/run/containerd/debug.sock"
#  uid = 0
#  gid = 0
#  level = "info"

重启containerd后发现pause镜像并未从新地址拉取，配置未生效。

containerd版本：containerd containerd.io 1.6.24 61f9fd88f79f081d64d6fa3bb1a0dc71ec870523

containerd运行状态：

[root@k8sslave01 kubelet]# systemctl status containerd
● containerd.service - containerd container runtime
   Loaded: loaded (/usr/lib/systemd/system/containerd.service; disabled; vendor preset: disabled)
   Active: active (running) since Mon 2024-02-19 13:11:30 CST; 10min ago
     Docs: https://containerd.io
  Process: 9709 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
 Main PID: 9711 (containerd)
    Tasks: 164
   Memory: 563.7M
   CGroup: /system.slice/containerd.service
           ├─2070 /usr/bin/containerd-shim-runc-v2 -namespace k8s.io -id 85bb2d9e9ad97fcef48745279e396aaaac8677f02c72efa85a78eabe33764529 -address /run/containerd/containerd.sock
           ├─2088 /usr/bin/containerd-shim-runc-v2 -namespace k8s.io -id 879cd87bf469b8966730fec490269673b46221acdc68685fb1213379616e8d08 -address /run/containerd/containerd.sock
           ├─2131 /usr/bin/containerd-shim-runc-v2 -namespace k8s.io -id d1a1220b677c9df2016dbdbe89a96778bb088af009bf9fe991b75f0260c27dfd -address /run/containerd/containerd.sock
           ├─2164 /usr/bin/containerd-shim-runc-v2 -namespace k8s.io -id 62f0f6b1e966f977d1384c57302840319f4e20f075761cba6f62e5e651bb6952 -address /run/containerd/containerd.sock
           ├─3579 /usr/bin/containerd-shim-runc-v2 -namespace k8s.io -id 949a8f4a46bb1dcce10054af8eaba77906d2fc746dcf3a295b69f41274968bd9 -address /run/containerd/containerd.sock
           ├─9711 /usr/bin/containerd
           ├─kubepods-besteffort-podf5b81dee_42cf_4c80_8927_239a45cbf0fb.slice:cri-containerd:144eb54ca91511a50c4f31a82bb18d1da1267650193f72d5eaac6adec4cac029
           │ └─3707 /usr/bin/promtail -config.file=/etc/promtail/promtail.yaml
           ├─kubepods-besteffort-podf5b81dee_42cf_4c80_8927_239a45cbf0fb.slice:cri-containerd:949a8f4a46bb1dcce10054af8eaba77906d2fc746dcf3a295b69f41274968bd9
           │ └─3599 /pause
           ├─kubepods-besteffort-poda668fff9_1320_40b2_a9b1_f04f721b6b16.slice:cri-containerd:54e9858d309cedfee9975af9e21c71ddfdd1ef479c6efe97369cbd9c5a253e9f
           │ └─2254 /usr/local/bin/kube-proxy --config=/var/lib/kube-proxy/config.conf --hostname-override=k8sslave01
           ├─kubepods-burstable-podff9b5985_0a25_4361_bcfe_f5eddbc4cef5.slice:cri-containerd:765d6d3050cd0cb37acf7f88b68b40adeebf9abae24b77e7329b7e819cdf4431
           │ ├─25676 cilium-agent --config-dir=/tmp/cilium/config-map
           │ └─29950 cilium-health-responder --listen 4240 --pidfile /var/run/cilium/state/health-endpoint.pid
           ├─kubepods-burstable-podff9b5985_0a25_4361_bcfe_f5eddbc4cef5.slice:cri-containerd:62f0f6b1e966f977d1384c57302840319f4e20f075761cba6f62e5e651bb6952
           │ └─2202 /pause
           ├─kubepods-besteffort-podd1ff8e59_2b3c_4baf_bd98_4ce750f4e1ac.slice:cri-containerd:879cd87bf469b8966730fec490269673b46221acdc68685fb1213379616e8d08
           │ └─2141 /pause
           ├─kubepods-besteffort-poda668fff9_1320_40b2_a9b1_f04f721b6b16.slice:cri-containerd:85bb2d9e9ad97fcef48745279e396aaaac8677f02c72efa85a78eabe33764529
           │ └─2116 /pause
           ├─kubepods-besteffort-podd1ff8e59_2b3c_4baf_bd98_4ce750f4e1ac.slice:cri-containerd:bff2d1eba164983b1a6da78ee6af561f0854f933ecc8728146b0a812642e2851
           │ └─12221 /bin/node_exporter --path.procfs=/host/proc --path.sysfs=/host/sys --path.rootfs=/host/root --path.udev.data=/host/root/run/udev/data --web.listen-address=[0.0.0.0]:9101
           ├─kubepods-besteffort-pod19058bec_ec59_4c9a_b98e_066e7259d7ce.slice:cri-containerd:d1a1220b677c9df2016dbdbe89a96778bb088af009bf9fe991b75f0260c27dfd
           │ └─2188 /pause
           └─kubepods-besteffort-pod19058bec_ec59_4c9a_b98e_066e7259d7ce.slice:cri-containerd:bd00a1ad428d5823a48518861959f2e499d3357cdd810dfb6a2efe04fe3b08f7
             └─28261 cilium-operator-generic --config-dir=/tmp/cilium/config-map --debug=false

1. 旧pause镜像已本地缓存 #

containerd会优先使用本地已存在的镜像，不会重复拉取。先检查本地是否有旧的pause镜像：

crictl images | grep pause

如果存在旧镜像，删除后再重启组件：

# 替换为实际的旧镜像ID或名称
crictl rmi <旧pause镜像ID/名称>

2. 未重启kubelet #

修改containerd配置后，仅重启containerd不够，kubelet才是调用containerd创建sandbox的核心组件，需要同步重启：

systemctl restart kubelet

3. 配置未被正确加载 #

用以下命令验证containerd实际加载的配置：

containerd config dump | grep sandbox_image

如果输出不是你设置的阿里云地址，检查：

• /etc/containerd/config.toml权限是否为644，属主属组是否为root:root
• 配置文件是否存在语法错误（比如引号、缩进问题）

验证配置生效 #

创建测试Pod，查看使用的sandbox镜像地址：

# 创建测试Pod
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: test-pause
spec:
  containers:
  - name: nginx
    image: nginx:alpine
EOF

# 查看Pod使用的sandbox镜像
kubectl describe pod test-pause | grep -i "sandbox image"

内容的提问来源于stack exchange，提问作者Dolphin