使用containerd client+soci snapshotter连接不安全私有仓库报错排查

我尝试通过以下Go代码配置一个不安全的containerd客户端，从不安全私有仓库拉取镜像，并使用soci snapshotter：

package main

import (
    "context"
    "crypto/tls"
    "fmt"
    "os"

    "github.com/awslabs/soci-snapshotter/fs/source"
    "github.com/containerd/containerd"
    "github.com/containerd/containerd/pkg/snapshotters"
    "github.com/containerd/containerd/remotes/docker"
    "github.com/containerd/containerd/remotes/docker/config"
)

func main() {
    var address = "/run/containerd/containerd.sock"
    var ref = "localhost:5000/python:3.9"
    var sociIndexDigest = "sha256:7b09431ef0749bee7491ba28d1adbe6e6e9e008e9be65fe35eed0aca31a01c91"

    client, err := containerd.New(address, containerd.WithDefaultNamespace("default"))
    if err != nil {
        fmt.Println(err.Error())
        os.Exit(1)
    }
    defer client.Close()

    options := docker.ResolverOptions{
        Hosts: config.ConfigureHosts(context.TODO(), config.HostOptions{
            DefaultScheme: "http",
            DefaultTLS: &tls.Config{
                InsecureSkipVerify: true,
            },
        }),
    }

    _, err = client.Pull(context.TODO(), ref,
        containerd.WithResolver(docker.NewResolver(options)),
        containerd.WithPullSnapshotter("soci"),
        containerd.WithPullUnpack,
        containerd.WithImageHandlerWrapper(source.AppendDefaultLabelsHandlerWrapper(sociIndexDigest, snapshotters.AppendInfoHandlerWrapper(ref))))
    if err != nil {
        fmt.Println(err.Error())
        os.Exit(1)
    }
    fmt.Println("Success")
}

但持续出现如下错误（截取日志）：

{"error":"skipping mounting layer sha256:a99509a323905a80628005e4f3bc26ac15ebaf3ffdb08a9646a7f2d110ab38f9 as FUSE mount: no ztoc for layer","key":"default/33/extract-128831236-kfhG sha256:781d5934416a582cf712c35212a8f92940c0223da02c1360d9ebb834d0f2c873","level":"warning","msg":"failed to prepare remote snapshot","parent":"sha256:9bb22d850b6e163c76b5cee00494067210e96c4cf585e2cd9d68898e31f43f69","remote-snapshot-prepared":"false","time":"2024-06-13T14:49:30.132049905Z"}
...
...
{"error":"cannot unpack the layer: cannot fetch layer: unable to fetch descriptor (sha256:a99509a323905a80628005e4f3bc26ac15ebaf3ffdb08a9646a7f2d110ab38f9) from remote store: Get \"https://localhost:5000/v2/python/blobs/sha256:a99509a323905a80628005e4f3bc26ac15ebaf3ffdb08a9646a7f2d110ab38f9\": context canceled","key":"default/33/extract-128831236-kfhG sha256:781d5934416a582cf712c35212a8f92940c0223da02c1360d9ebb834d0f2c873","level":"warning","msg":"failed to prepare snapshot; deferring to container runtime","parent":"sha256:9bb22d850b6e163c76b5cee00494067210e96c4cf585e2cd9d68898e31f43f69","time":"2024-06-13T14:49:30.363771421Z"}
...
...
{"error":"cannot unpack the layer: cannot fetch layer: unable to fetch descriptor (sha256:a99509a323905a80628005e4f3bc26ac15ebaf3ffdb08a9646a7f2d110ab38f9) from remote store: Get \"https://localhost:5000/v2/python/blobs/sha256:a99509a323905a80628005e4f3bc26ac15ebaf3ffdb08a9646a7f2d110ab38f9\": unknown \"unknown\": giving up request after 9 attempt(s): Get \"https://localhost:5000/v2/python/blobs/sha256:a99509a323905a80628005e4f3bc26ac15ebaf3ffdb08a9646a7f2d110ab38f9\": http: server gave HTTP response to HTTPS client","key":"default/34/extract-776892714-Ki3b sha256:781d5934416a582cf712c35212a8f92940c0223da02c1360d9ebb834d0f2c873","level":"warning","msg":"failed to prepare snapshot; deferring to container runtime","parent":"sha256:9bb22d850b6e163c76b5cee00494067210e96c4cf585e2cd9d68898e31f43f69","time":"2024-06-13T14:49:44.089475833Z"}

我的/etc/containerd/config.toml配置如下：

version = 2
[plugins."io.containerd.grpc.v1.cri".containerd]
    disable_snapshot_annotations = false
[proxy_plugins]
    [proxy_plugins.soci]
        type = "snapshot"
        address = "/run/soci-snapshotter-grpc/soci-snapshotter-grpc.sock"

我尝试移除DefaultTLS选项后结果仍相同，但使用nerdctl加--insecure-registry选项时推拉流程正常。请问我遗漏了什么或操作有误？

解决方法 #

1. 核心问题：soci snapshotter未配置不安全仓库 #

你只在containerd客户端里配置了不安全仓库，但soci snapshotter是独立的代理插件，会自行拉取ztoc文件和镜像层，不会复用containerd客户端的Resolver配置。日志里的http: server gave HTTP response to HTTPS client说明snapshotter在尝试用HTTPS访问你的HTTP仓库，导致失败。

2. 配置soci snapshotter的不安全仓库 #

找到soci snapshotter的配置文件（默认路径是/etc/soci-snapshotter-grpc/config.toml），添加以下配置：

[plugins."io.containerd.soci.v1.soci"]
  [plugins."io.containerd.soci.v1.soci".registry]
    [plugins."io.containerd.soci.v1.soci".registry.mirrors]
      [plugins."io.containerd.soci.v1.soci".registry.mirrors."localhost:5000"]
        endpoint = ["http://localhost:5000"]
    [plugins."io.containerd.soci.v1.soci".registry.configs]
      [plugins."io.containerd.soci.v1.soci".registry.configs."localhost:5000".tls]
        insecure_skip_verify = true

• endpoint指定用HTTP协议访问该仓库
• insecure_skip_verify跳过证书验证（适用于无证书或自签证书的HTTP仓库）

3. 重启soci snapshotter服务 #

配置修改后，重启服务使配置生效：

systemctl restart soci-snapshotter-grpc

4. 代码优化（可选） #

你的代码中DefaultScheme已经设为http，没必要再配置DefaultTLS，可以简化Resolver配置：

options := docker.ResolverOptions{
    Hosts: config.ConfigureHosts(context.TODO(), config.HostOptions{
        DefaultScheme: "http",
    }),
}

另外，确认sociIndexDigest与拉取的镜像完全匹配，否则会出现no ztoc for layer的警告，导致无法用FUSE挂载快照。

内容的提问来源于stack exchange，提问作者user17297103