K3s集群无法从私有GitLab仓库拉取镜像问题求助

在K3s集群中部署Pod时，从私有GitLab仓库拉取镜像失败，返回403 Forbidden错误，但在工作节点上用crictl手动拉取可成功，旧版K8s集群用相同配置能正常拉取，怀疑是K3s版本问题。

Pod配置 #

apiVersion: v1
kind: Pod
metadata:
[...] 
spec:
  containers:
  - image: gitlab.private:31443/cdn/cdndemo/appcdnmanagerui/main
    imagePullPolicy: Always
[...] 
  imagePullSecrets:
    - name: gitlab-dc-cdndemo-2

镜像拉取密钥信息 #

执行命令解析密钥内容：

$ base64 -d <(kubectl -n test-cdndemo get secret gitlab-dc-cdndemo-2 -o json | jq -r '.data.".dockerconfigjson"') | jq

输出结果：

{
  "auths": {
    "https://gitlab.private:31443": {
      "username": "gitlab+deploy-token-22",
      "password": "EGDLqGKJwBtfYYf9cDFg",
      "email": "example@example.com",
      "auth": "Z2l0bGFiK2RlcGxveS10b2tlbi0yMjpFR0RMcUdLSndCdGZZWWY5Y0RGZw=="
    }
  }
}

部署后错误事件 #

Pod部署后出现以下拉取失败事件：

Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  14m                   default-scheduler  Successfully assigned test-cdndemo/appcdnmanagerui-68c8f8c6dd-qcxr5 to node-waw107
  Normal   Pulling    13m (x4 over 14m)     kubelet            Pulling image "gitlab.private:31443/cdn/cdndemo/appcdnmanagerui/main"
  Warning  Failed     13m (x4 over 14m)     kubelet            Failed to pull image "gitlab.private:31443/cdn/cdndemo/appcdnmanagerui/main": rpc error: code = Unknown desc = failed to pull and unpack image "gitlab.private:31443/cdn/cdndemo/appcdnmanagerui/main:latest": failed to resolve reference "gitlab.private:31443/cdn/cdndemo/appcdnmanagerui/main:latest": failed to authorize: failed to fetch anonymous token: unexpected status: 403 Forbidden
  Warning  Failed     13m (x4 over 14m)     kubelet            Error: ErrImagePull
  Warning  Failed     12m (x6 over 14m)     kubelet            Error: ImagePullBackOff
  Normal   BackOff    4m41s (x43 over 14m)  kubelet            Back-off pulling image "gitlab.private:31443/cdn/cdndemo/appcdnmanagerui/main"
                                                                                       gitlab.private:31443/cdn/cdndemo/appcdnmanagerui/main

错误为403 Forbidden而非401 Unauthorized，说明凭证本身有效。

工作节点手动拉取验证 #

在工作节点上用crictl手动指定凭证拉取镜像成功：

$ crictl --debug pull --creds gitlab+deploy-token-22:EGDLqGKJwBtfYYf9cDFg gitlab.private:31443/cdn/cdndemo/appcdnmanagerui/main:latest
DEBU[0000] get image connection
DEBU[0000] PullImageRequest: &PullImageRequest{Image:&ImageSpec{Image:gitlab.private:31443/cdn/cdndemo/appcdnmanagerui/main:latest,Annotations:map[string]string{},},Auth:&AuthConfig{Username:gitlab+deploy-token-22,Password:EGDLqGKJwBtfYYf9cDFg,Auth:,ServerAddress:,IdentityToken:,RegistryToken:,},SandboxConfig:nil,}
DEBU[0006] PullImageResponse: &PullImageResponse{ImageRef:sha256:0c3b5d355c164d02aaa7b6cbe91bbfa12bd35826566472317efac63cb467d260,}
Image is up to date for sha256:0c3b5d355c164d02aaa7b6cbe91bbfa12bd35826566472317efac63cb467d260

$ crictl image
IMAGE                                                   TAG                  IMAGE ID            SIZE
[...] 
gitlab.private:31443/cdn/cdndemo/appcdnmanagerui/main   latest               0c3b5d355c164       105MB

使用base64编码的auth值拉取也成功：

crictl --debug pull --auth Z2l0bGFiK2RlcGxveS10b2tlbi0yMjpFR0RMcUdLSndCdGZZWWY5Y0RGZw== gitlab.private:31443/cdn/cdndemo/appcdnmanagerui/main:latest

当前使用版本 #

# k3s -v
k3s version v1.25.4+k3s1 (0dc63334)
go version go1.19.3
# crictl -v
crictl version v1.25.0-k3s1

疑问 #

旧版K8s集群用相同配置可正常拉取镜像，怀疑是K3s版本bug，有没有人遇到过类似问题并解决？

内容的提问来源于stack exchange，提问作者Rafał W.