部署Kubeflow Training Operator遇CrashLoopBackOff及客户端限流报错

使用Rancher Desktop创建本地Kubernetes集群后，执行命令部署Kubeflow Training Operator：

kubectl apply -k "github.com/kubeflow/training-operator/manifests/overlays/standalone?ref=v1.6.0"

部署后Pod处于CrashLoopBackOff状态，日志如下：

➜ kubectl logs training-operator-xxx -n kubeflow

I0714 04:54:03.434723       1 request.go:682] Waited for 1.024840626s due to client-side throttling, not priority and fairness, request: GET:https://10.43.0.1:443/apis/packages.operators.coreos.com/v1?timeout=32s
1.689310446978421e+09   INFO    controller-runtime.metrics  Metrics server is starting to listen    {"addr": ":8080"}
I0714 04:54:14.225698       1 request.go:682] Waited for 1.047503167s due to client-side throttling, not priority and fairness, request: GET:https://10.43.0.1:443/apis/node.k8s.io/v1?timeout=32s
I0714 04:54:24.275500       1 request.go:682] Waited for 1.948469293s due to client-side throttling, not priority and fairness, request: GET:https://10.43.0.1:443/apis/artifact.apicur.io/v1alpha1?timeout=32s
I0714 04:54:34.325909       1 request.go:682] Waited for 2.849523377s due to client-side throttling, not priority and fairness, request: GET:https://10.43.0.1:443/apis/operators.coreos.com/v1?timeout=32s
I0714 04:54:44.724674       1 request.go:682] Waited for 1.047644251s due to client-side throttling, not priority and fairness, request: GET:https://10.43.0.1:443/apis/operators.coreos.com/v1?timeout=32s
I0714 04:54:54.774273       1 request.go:682] Waited for 1.947402376s due to client-side throttling, not priority and fairness, request: GET:https://10.43.0.1:443/apis/elasticsearch.k8s.elastic.co/v1?timeout=32s

核心问题是客户端限流导致Training Operator无法完成API发现和初始化，可按以下步骤修复：

1. 调整客户端限流参数 #

修改Training Operator的Deployment配置，调大Kubernetes API请求的限流阈值：

• 编辑Deployment：

  kubectl edit deployment training-operator -n kubeflow
  

• 在spec.template.spec.containers[0].args中添加：

  --kube-api-qps=50
  --kube-api-burst=100
  

  这两个参数分别控制每秒请求数和突发请求数，调大后可缓解限流问题。

2. 升级Rancher Desktop集群资源 #

本地集群资源不足会导致API响应缓慢，加剧限流：

• 打开Rancher Desktop的Resources设置页
• 调高CPU（至少2核）和内存（至少4GB）分配，保存后重启集群

3. 清理无关CRD #

日志中出现的非Kubeflow必需API组（如artifact.apicur.io、elasticsearch.k8s.elastic.co）会拖慢API发现速度：

• 列出所有CRD：

  kubectl get crd
  

• 删除无关的CRD：

  kubectl delete crd <目标CRD名称>
  

4. 切换兼容版本 #

v1.6.0可能与Rancher Desktop的Kubernetes版本存在兼容性问题，尝试降级到v1.5.0：

kubectl delete -k "github.com/kubeflow/training-operator/manifests/overlays/standalone?ref=v1.6.0"
kubectl apply -k "github.com/kubeflow/training-operator/manifests/overlays/standalone?ref=v1.5.0"

5. 验证修复效果 #

查看Pod状态确认是否恢复正常：

kubectl get pods -n kubeflow

若Pod进入Running状态，再检查日志确认无持续限流报错即可。

内容的提问来源于stack exchange，提问作者Hongbo Miao