CentOS 8下Rootless Podman配合GitLab Runner时部分命令无响应问题

• 我使用gitlab-runner用户在GitLab中运行Podman命令时遇到问题，个人用户无法查看创建的容器，于是修改了GitLab CI Runner的运行用户。
• 重新运行GitLab流水线，前3个步骤执行成功，创建了1个Podman Pod及2个所属容器，执行最后一个容器的podman run命令时流程突然卡住。登录服务器排查：
  • 执行podman ps -a按下回车后无任何返回，命令行仍可正常输入内容
  • 执行podman images list可正常返回镜像列表，但存在2个ID完全相同、无法删除的镜像
  • 执行sudo podman ps则可正常输出结果
• Rootless模式下Podman部分命令可用、部分无法正常运行：同一用户在服务器本地可正常执行podman run hello-world，但在GitLab流水线中执行podman run就会无任何报错卡住，只能手动取消任务。
• 运行环境：CentOS 8，Podman版本为3.3.1，sudo podman info输出如下：

host:
  arch: amd64
  buildahVersion: 1.22.3
  cgroupControllers:
  - cpuset
  - cpu
  - cpuacct
  - blkio
  - memory
  - devices
  - freezer
  - net_cls
  - perf_event
  - net_prio
  - hugetlb
  - pids
  - rdma
  cgroupManager: systemd
  cgroupVersion: v1
  conmon:
    package: conmon-2.0.29-1.module+el8.5.0+12582+56d94c81.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.29, commit: 0f5bee61b18d4581668e5bf18b910cda3cff5081'
  cpus: 40
  distribution:
    distribution: '"rhel"'
    version: "8.5"
  eventLogger: file
  hostname: DELL2
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 4.18.0-348.2.1.el8_5.x86_64
  linkmode: dynamic
  memFree: 55734177792
  memTotal: 134510010368
  ociRuntime:
    name: runc
    package: runc-1.0.2-1.module+el8.5.0+12582+56d94c81.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.2
      spec: 1.0.2-dev
      go: go1.16.7
      libseccomp: 2.5.1
  os: linux
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_NET_RAW,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /bin/slirp4netns
    package: slirp4netns-1.1.8-1.module+el8.5.0+12582+56d94c81.x86_64
    version: |-
      slirp4netns version 1.1.8
      commit: d361001f495417b880f20329121e3aa431a8f90f
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.1
  swapFree: 4294963200
  swapTotal: 4294963200
  uptime: 11h 30m 52.88s (Approximately 0.46 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageStore:
    number: 2
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 3.3.1
  Built: 1632213702
  BuiltTime: Tue Sep 21 10:41:42 2021
  GitCommit: ""
  GoVersion: go1.16.7
  OsArch: linux/amd64
  Version: 3.3.1

• 故障截图：
  

故障根因 #

该问题属于典型的rootless Podman用户上下文不一致+残留锁文件问题：

1. 修改gitlab-runner运行用户前，该用户执行过的Podman操作残留了存储锁，导致后续非root调用时卡在锁等待状态
2. GitLab CI Runner执行任务时的环境变量和本地登录用户的环境变量不一致，XDG_RUNTIME_DIR路径不匹配，导致Podman读取到错误的运行时状态
3. 相同ID镜像无法删除是因为镜像被残留的Podman进程持有引用，无法执行删除操作

修复步骤 #

1. 停止所有Podman相关进程

podman system reset -f
pkill -u gitlab-runner podman conmon

2. 清理gitlab-runner用户的Podman运行时缓存和锁文件

rm -rf /run/user/$(id -u gitlab-runner)/containers
rm -rf /home/gitlab-runner/.local/share/containers

3. 给GitLab CI Runner任务显式指定rootless Podman需要的环境变量，在.gitlab-ci.yml对应的job中添加变量配置：

variables:
  XDG_RUNTIME_DIR: "/run/user/$(id -u gitlab-runner)"
  DOCKER_HOST: "unix://$XDG_RUNTIME_DIR/podman/podman.sock"

4. 验证gitlab-runner用户的subuid、subgid配置是否正确

grep gitlab-runner /etc/subuid /etc/subgid

如果没有输出，执行以下命令添加映射：

usermod --add-subuids 100000-165535 gitlab-runner
usermod --add-subgids 100000-165535 gitlab-runner

5. 重启gitlab-runner服务生效

systemctl restart gitlab-runner

可选优化 #

如果上述操作后仍有卡顿，可将Podman事件日志从file改为journald，避免日志写入锁冲突：
修改/etc/containers/containers.conf中events_logger = "journald"，重启Podman服务生效。

内容的提问来源于stack exchange，提问作者Iman