Kubernetes CronJob手动触发异常：修复未知Job与退避超限问题

问题背景 #

配置了定时重启Deployment的CronJob，手动触发后出现两个异常：

1. CronJob控制器报错：Saw a job that the controller did not create or forgot: vijay-deployment-restart-manual-a7s16
2. Job报错：Job has reached the specified backoff limit

集群版本信息：

• Client Version: v1.29.4
• Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
• Server Version: v1.29.4-eks-036c24b

修复方案 #

1. 修正CronJob的命名空间 #

将CronJob的命名空间改为odex，与ServiceAccount、RBAC资源保持一致，避免跨命名空间权限失效：

apiVersion: batch/v1
kind: CronJob
metadata:
  name: vijay-deployment-restart
  namespace: odex  # 统一为odex命名空间
spec:
  concurrencyPolicy: Forbid
  schedule: '48 10 * * *'
  jobTemplate:
    spec:
      backoffLimit: 3
      activeDeadlineSeconds: 800
      template:
        spec:
          serviceAccountName: vijay-deployment-restart
          restartPolicy: Never
          containers:
          - name: kubectl
            image: bitnami/kubectl:latest
            command:
            - 'kubectl'
            - 'rollout'
            - 'restart'
            - 'deployment/mango-deployment'

2. 修正Role的Deployment授权范围 #

将Role中授权的Deployment名称改为实际需要重启的mango-deployment，确保权限匹配：

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: Vijay-deployment-restart
  namespace: odex
rules:
  - apiGroups: ["apps", "extensions"]
    resources: ["deployments"]
    resourceNames: ["mango-deployment"]  # 替换为目标Deployment名称
    verbs: ["get", "patch", "list", "watch"]

3. 清理异常Job并重新触发 #

先删除已失败的手动Job：

kubectl delete job vijay-deployment-restart-manual-a7s16 -n odex

再重新手动触发CronJob：

kubectl create job --from=cronjob/vijay-deployment-restart vijay-deployment-restart-manual -n odex

4. 验证权限有效性 #

确认ServiceAccount拥有重启目标Deployment的权限：

kubectl auth can-i patch deployments/mango-deployment --as=system:serviceaccount:odex:vijay-deployment-restart -n odex

返回yes则权限配置正确。

问题根源 #

• 命名空间不匹配：CronJob与权限资源分属不同命名空间，导致Pod无法获取有效权限执行重启命令，不断重试触发退避限制。
• 授权资源不匹配：Role中授权的Deployment名称与实际操作的Deployment不一致，kubectl rollout restart命令执行失败。
• 手动Job跟踪问题：手动创建的Job命名不符合CronJob自动生成规则，被控制器识别为非自身创建的资源，触发UnexpectedJob警告。

内容的提问来源于stack exchange，提问作者Vijaykumar Jawale