K3s集群Pod仍尝试调度至已下线旧节点，求排查建议

在Hetzner上通过terraform-hcloud-kube-hetzner部署了v1.27.4+k3s1版本的K3s集群，添加硬件配置更优的新节点并入集群后，对旧节点执行了cordon、drain操作并关机，但部分Pod仍试图调度到已不在集群中的旧节点，报错nodeinfo not found for node name "agent-cx21-fsn1-iof"。删除HelmRelease后Flux重建，问题依旧，求排查建议。

apiVersion: v1
kind: Pod
status:
  phase: Pending
  conditions:
    - type: PodScheduled
      status: 'False'
      lastProbeTime: null
      lastTransitionTime: '2023-09-05T13:47:11Z'
      reason: SchedulerError
      message: nodeinfo not found for node name "vpl-agent-cx21-fsn1-iof"
  qosClass: Burstable
spec:
  volumes:
    - name: config
      persistentVolumeClaim:
        claimName: config-pgadmin-0
    - name: kube-api-access-zk8jq
      projected:
        sources:
          - serviceAccountToken:
              expirationSeconds: 3607
              path: token
          - configMap:
              name: kube-root-ca.crt
              items:
                - key: ca.crt
                  path: ca.crt
          - downwardAPI:
              items:
                - path: namespace
                  fieldRef:
                    apiVersion: v1
                    fieldPath: metadata.namespace
        defaultMode: 420
  containers:
    - name: pgadmin
      image: docker.io/dpage/pgadmin4:7
      ports:
        - name: http
          containerPort: 8080
          protocol: TCP
      envFrom:
        - secretRef:
            name: pgadmin-secret
      env:
        - name: PGADMIN_LISTEN_PORT
          value: '8080'
      resources:
        limits:
          memory: 512Mi
        requests:
          cpu: 10m
          memory: 128Mi
      volumeMounts:
        - name: config
          mountPath: /var/lib/pgadmin
        - name: kube-api-access-zk8jq
          readOnly: true
          mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      livenessProbe:
        httpGet:
          path: /misc/ping
          port: 8080
          scheme: HTTP
        initialDelaySeconds: 3
        timeoutSeconds: 1
        periodSeconds: 10
        successThreshold: 1
        failureThreshold: 3
      readinessProbe:
        httpGet:
          path: /misc/ping
          port: 8080
          scheme: HTTP
        initialDelaySeconds: 3
        timeoutSeconds: 1
        periodSeconds: 10
        successThreshold: 1
        failureThreshold: 3
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      imagePullPolicy: IfNotPresent
  restartPolicy: Always
  terminationGracePeriodSeconds: 30
  dnsPolicy: ClusterFirst
  serviceAccountName: default
  serviceAccount: default
  automountServiceAccountToken: true
  securityContext:
    runAsUser: 5050
    runAsGroup: 5050
    supplementalGroups:
      - 44
      - 109
      - 100
    fsGroup: 5050
    fsGroupChangePolicy: OnRootMismatch
  hostname: pgadmin-0
  subdomain: pgadmin
  schedulerName: default-scheduler
  tolerations:
    - key: node.kubernetes.io/not-ready
      operator: Exists
      effect: NoExecute
      tolerationSeconds: 300
    - key: node.kubernetes.io/unreachable
      operator: Exists
      effect: NoExecute
      tolerationSeconds: 300
  priority: 0
  enableServiceLinks: true
  preemptionPolicy: PreemptLowerPriority

• 确认旧节点是否彻底移除：执行kubectl get nodes查看旧节点状态，若仍在列表中（状态为NotReady），手动删除节点：kubectl delete node agent-cx21-fsn1-iof
• 检查Pod调度约束：查看对应Deployment/StatefulSet的spec.template.spec配置，确认是否存在nodeSelector、affinity硬编码旧节点，或nodeName直接指定旧节点名称
• 排查PersistentVolume绑定问题：该Pod使用PVCconfig-pgadmin-0，检查对应PV是否为Local类型并绑定到旧节点：kubectl describe pv <pv-name>，若是Local PV，需重新创建PV绑定新节点，或调整StorageClass支持动态迁移
• 检查Flux配置：查看HelmRelease或Kustomization资源的values配置，确认是否存在指定旧节点的调度规则，确保Flux同步的配置无硬编码内容
• 清除调度器缓存：在K3s master节点重启kube-scheduler组件（执行systemctl restart k3s），清除调度器中残留的旧节点信息
• 处理节点Finalizers：若删除节点卡住，检查节点finalizers：kubectl get node agent-cx21-fsn1-iof -o jsonpath='{.metadata.finalizers}'，手动移除残留项：kubectl patch node agent-cx21-fsn1-iof -p '{"metadata":{"finalizers":[]}}' --type=merge
• 检查K3s节点注册数据：在master节点查看/var/lib/rancher/k3s/server/db/目录下的节点相关数据，确保旧节点注册信息已完全清理

内容的提问来源于stack exchange，提问作者Tomaž