K8s自定义控制器无法找到CR问题排查求助

问题描述 #

使用kubebuilder开发K8s控制器，负责调和集群级自定义资源（CR）test-pod-monitor-cr，但启动后出现CR找不到的错误：

podlogmonitors.monitoring.mydomain.com "test-pod-monitor-cr" not found

已验证控制器Pod的ServiceAccount具备该CR的get权限（通过kubectl使用其token测试正常）。

错误日志 #

2025-02-15T23:13:32Z    INFO    starting server {"name": "health probe", "addr": "[::]:8081"}
2025-02-15T23:13:32Z    INFO    Starting EventSource    {"controller": "podlogmonitor", "controllerGroup": "monitoring.mydomain.com", "controllerKind": "PodLogMonitor", "source": "kind source: *v1.PodLogMonitor"}
2025-02-15T23:13:32Z    INFO    Starting Controller     {"controller": "podlogmonitor", "controllerGroup": "monitoring.mydomain.com", "controllerKind": "PodLogMonitor"}
2025-02-15T23:13:32Z    INFO    Starting workers        {"controller": "podlogmonitor", "controllerGroup": "monitoring.mydomain.com", "controllerKind": "PodLogMonitor", "worker count": 1}
2025-02-15T23:13:32Z    ERROR   Reconciler error        {"controller": "podlogmonitor", "controllerGroup": "monitoring.mydomain.com", "controllerKind": "PodLogMonitor", "PodLogMonitor": {"name":"test-pod-monitor-cr"}, "namespace": "", "name": "test-pod-monitor-cr", "reconcileID": "66f143a5-a211-4eab-911c-f0109f0661d7", "error": "podlogmonitors.monitoring.mydomain.com \"test-pod-monitor-cr\" not found"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.0/pkg/internal/controller/controller.go:332
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.0/pkg/internal/controller/controller.go:279
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.0/pkg/internal/controller/controller.go:240

排查与解决方案 #

1. 处理缓存未同步的启动重试问题 #

控制器启动初期，内部缓存尚未完成集群资源同步，此时收到调和请求会出现CR找不到的错误。修改Reconcile函数的错误处理逻辑，添加延迟重试：

func (r *PodLogMonitorReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    _ = log.FromContext(ctx)

    log := r.Log.WithValues("podlogmonitor", req.NamespacedName)
    log.Info("Line - 1")
    // Fetch the PodLogMonitor resource
    var podLogMonitor monitoringv1.PodLogMonitor
    if err := r.Get(ctx, req.NamespacedName, &podLogMonitor); err != nil {
        log.Error(err, "unable to fetch PodLogMonitor")
        // 非NotFound错误直接返回
        if client.IgnoreNotFound(err) != nil {
            return reconcile.Result{}, err
        }
        // 缓存未同步，延迟5秒重试
        return reconcile.Result{RequeueAfter: 5 * time.Second}, nil
    }

    // 后续原有逻辑...
}

2. 验证CRD与CR的有效性 #

• 确认CRD已处于Established状态：

  kubectl get crd podlogmonitors.monitoring.mydomain.com
  

  检查输出的STATUS字段是否为Established，未处于该状态则说明CRD未正常生效。

• 确认CR确实存在：

  kubectl get podlogmonitors.monitoring.mydomain.com test-pod-monitor-cr
  

  若CR不存在需重新创建；若存在，检查是否有拼写错误或资源被意外删除。

3. 修复CRD的Status子资源配置 #

代码中存在更新CR状态的逻辑，但当前CRD未启用Status子资源，会导致后续状态更新失败。修改CRD定义，添加subresources配置：

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: podlogmonitors.monitoring.mydomain.com
spec:
  group: monitoring.mydomain.com
  names:
    kind: PodLogMonitor
    listKind: PodLogMonitorList
    plural: podlogmonitors
    singular: podlogmonitor
  scope: Cluster
  versions:
    - name: v1
      served: true
      storage: true
      # 添加Status子资源配置
      subresources:
        status: {}
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                namespace:
                  type: string
                logMessage:
                  type: string
            status:
              type: object
              properties:
                lastRestartedPodName:
                  type: string
                lastRestartTime:
                  type: string

4. 优化客户端使用（可选） #

代码中checkPodLogs函数自行创建clientset，可直接复用控制器内置的client.Client，减少资源消耗并利用缓存：

func (r *PodLogMonitorReconciler) checkPodLogs(ctx context.Context, pod *corev1.Pod, logMessage string, podLogMonitor *monitoringv1.PodLogMonitor) error {
   log := r.Log.WithValues("pod", pod.Name)
   log.Info("Line - 6")

   // 直接使用r.Client获取Pod日志，无需额外创建clientset
   podLogOpts := &corev1.PodLogOptions{}
   req := r.RESTClient().Get().
       Namespace(pod.Namespace).
       Resource("pods").
       Name(pod.Name).
       SubResource("log").
       VersionedParams(podLogOpts, scheme.ParameterCodec)

   podLogs, err := req.Stream(ctx)
   if err != nil {
       log.Error(err, "unable to stream pod logs")
       return err
   }
   defer podLogs.Close()

   // 后续原有逻辑...
}

内容的提问来源于stack exchange，提问作者Saikiran