Nvidia Orin执行apt升级后Docker启动容器报错求助

问题概述 #

此前在Nvidia Orin上Docker运行正常，执行apt update和apt upgrade后，启动容器时出现资源占用报错，核心提示为无法删除overlay路径下的文件，提示device or resource busy。具体执行命令及报错如下：

ptc@orin:~$ docker run -d -it --rm -p 8888 --net=host -v /mnt/ssd:/mnt/ssd cronin/mlbase:version4
WARNING: Published ports are discarded when using host network mode
9e67d3eb58e7effddf79f4f20e4f60aa8cfb10f3c816bd7b8401753bc619019e
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #1: error running hook: exit status 1, stdout: , stderr: time="2024-12-19T10:30:03-05:00" level=info msg="Symlinking /mnt/ssd/docker/overlay2/7dede99aca991da3cbbe0f94db1c71c0e1dc5fc4c420bb6f6d20e6225c6f1a7d/merged/etc/vulkan/icd.d/nvidia_icd.json to /usr/lib/aarch64-linux-gnu/tegra/nvidia_icd.json"
time="2024-12-19T10:30:03-05:00" level=error msg="failed to create link [/usr/lib/aarch64-linux-gnu/tegra/nvidia_icd.json /etc/vulkan/icd.d/nvidia_icd.json]: failed to create symlink: failed to remove existing file: remove /mnt/ssd/docker/overlay2/7dede99aca991da3cbbe0f94db1c71c0e1dc5fc4c420bb6f6d20e6225c6f1a7d/merged/etc/vulkan/icd.d/nvidia_icd.json: device or resource busy": unknown.

修复步骤 #

无需直接卸载重装Docker，按以下顺序操作：

1. 卸载残留的overlay挂载点 #

报错中的文件处于挂载状态，先找到并卸载：

# 查找对应overlay层的挂载信息
mount | grep 7dede99aca991da3cbbe0f94db1c71c0e1dc5fc4c420bb6f6d20e6225c6f1a7d
# 执行卸载，替换为上一步查到的挂载路径
sudo umount /mnt/ssd/docker/overlay2/7dede99aca991da3cbbe0f94db1c71c0e1dc5fc4c420bb6f6d20e6225c6f1a7d/merged

2. 重启Docker服务 #

卸载挂载点后重启Docker，刷新服务状态：

sudo systemctl restart docker

3. 验证容器启动 #

重新运行原容器命令，检查是否正常启动：

docker run -d -it --rm -p 8888 --net=host -v /mnt/ssd:/mnt/ssd cronin/mlbase:version4

4. 重置NVIDIA容器运行时钩子（若步骤3无效） #

系统升级可能导致NVIDIA容器钩子配置异常，重新生成配置：

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

5. 清理Docker缓存（可选） #

若上述步骤均无效，清理未使用的镜像、容器及缓存：

sudo docker system prune -a

内容的提问来源于stack exchange，提问作者ptcesq