KServe ModelMesh部署时出现seldonio/mlserver镜像拉取失败问题
问题定性
该问题不属于Docker与K8s的核心配置异常,本质是容器镜像拉取环节的网络超时故障。
报错依据
部署过程输出的事件日志如下:
Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 4m20s default-scheduler Successfully assigned modelmesh-serving/modelmesh-serving-mlserver-0.x-77cc8fd548-xdgvr to minikube Normal Pulling 4m18s kubelet Pulling image "kserve/modelmesh:v0.9.0-rc0" Normal Pulled 3m18s kubelet Successfully pulled image "kserve/modelmesh:v0.9.0-rc0" in 59.419620166s Normal Created 3m18s kubelet Created container mm Normal Started 3m17s kubelet Started container mm Normal Pulling 3m17s kubelet Pulling image "seldonio/mlserver:0.5.2" Warning Failed 68s kubelet Failed to pull image "seldonio/mlserver:0.5.2": rpc error: code = Unknown desc = context deadline exceeded Warning Failed 68s kubelet Error: ErrImagePull Normal Pulling 68s kubelet Pulling image "kserve/modelmesh-runtime-adapter:v0.9.0-rc0"
从日志可明确判断故障特征:
- 第一个基础镜像
kserve/modelmesh:v0.9.0-rc0拉取耗时接近1分钟,已经达到kubelet默认拉取超时阈值的临界值,说明当前公网镜像拉取链路质量很差 - 拉取
seldonio/mlserver:0.5.2时直接触发context deadline exceeded超时错误,属于典型的公网镜像仓库连通性差、带宽不足导致的拉取失败,和Docker、K8s核心组件配置错误无关 - 后续拉取
kserve/modelmesh-runtime-adapter:v0.9.0-rc0是kubelet按顺序拉取Pod内其他容器镜像的正常行为,不属于额外异常
解决方法
- 验证网络连通性:执行
minikube ssh进入集群节点,手动运行docker pull seldonio/mlserver:0.5.2,确认手动拉取是否同样超时,锁定网络故障根因 - 配置镜像加速:给minikube节点内的Docker配置国内镜像加速源,重启Docker服务后重新触发Pod部署
- 本地导入镜像:在网络通畅的环境下提前拉取所需版本的镜像,通过
minikube image load <镜像名:标签>命令将本地镜像直接导入minikube节点,跳过公网拉取环节 - 调整超时阈值:修改kubelet的
--image-pull-progress-deadline启动参数,将默认1分钟的拉取超时时间适当调大,给大体积镜像拉取预留充足时间
内容的提问来源于stack exchange,提问作者bonijad383
相关产品推荐
相关产品推荐

