Terraform count创建多kubernetes_job无法并行运行问题排查

问题背景 #

• 目标需求：通过Terraform编排kubernetes_job资源，使用count函数创建3个独立Kubernetes Job并行执行，每个Job通过count索引传入对应数组元素作为启动参数
• 异常表现：实际运行时Job未并行执行，日志观测显示任务为顺序运行

初始Terraform配置如下：

resource "kubernetes_job" "rotate_eks_nodes" {

  count = length(local.eks_asg)

  metadata {
    name      = "rotate-eks-nodes-job-${count.index}"
    namespace = kubernetes_namespace.rotate_eks_nodes.metadata[0].name
  }

  spec {
    parallelism                = 1
    completions                = length(local.eks_asg)
    backoff_limit              = 0
    ttl_seconds_after_finished = 1000

    template {
      metadata {
      }
      spec {

        container {
          security_context {
            run_as_user = 0
          }
          name    = "rotate-eks-nodes-job"
          image   = "artifactory.ie/alpine-awscli:1.0.1"
          command = ["/bin/sh", "-c", "echo \"Starting Rotate EKS Nodes Job ${count.index}\" && /script/rotate.sh ${local.eks_asg[count.index]} ${local.eks_asg_desired_size[count.index]} && echo \"Completed Rotate EKS Nodes Job ${count.index}\""]
          
          volume_mount {
            name       = kubernetes_config_map.rotate_script.metadata[0].name
            mount_path = "/script"
          }
        }

        volume {
          name = kubernetes_config_map.rotate_script.metadata[0].name
          config_map {
            name         = kubernetes_config_map.rotate_script.metadata[0].name
            default_mode = "0755"
          }
        }

        service_account_name = kubernetes_service_account.rotate_eks_nodes.metadata[0].name
      }
    }
  }
  wait_for_completion = true
  timeouts {
    create = "15m"
    update = "15m"
  }
}

排查验证过程 #

parallelism参数调试验证 #

• 测试操作：将Job spec中的parallelism参数设置为3
• 测试结果：每个count索引生成的独立Job都会各自启动3个运行实例，最终总计生成9个Pod，不符合预期3个Job各运行1个实例的要求

对应Pod运行状态如下：

[ components ] > aws-vault exec build -- kubectl get po -n rotate-eks-nodes
NAME                           READY   STATUS    RESTARTS   AGE
rotate-eks-nodes-job-0-qwkrk   1/1     Running   0          74s
rotate-eks-nodes-job-0-qxfs2   1/1     Running   0          73s
rotate-eks-nodes-job-0-wd7p5   1/1     Running   0          73s
rotate-eks-nodes-job-1-5hjnb   1/1     Running   0          73s
rotate-eks-nodes-job-1-db98p   1/1     Running   0          73s
rotate-eks-nodes-job-1-h7hfh   1/1     Running   0          73s
rotate-eks-nodes-job-2-5h4b6   1/1     Running   0          73s
rotate-eks-nodes-job-2-jmrjx   1/1     Running   0          73s
rotate-eks-nodes-job-2-lp86b   1/1     Running   0          73s

wait_for_completion参数调试验证 #

• 测试操作：将Terraform资源的wait_for_completion参数设置为false
• 测试结果：Terraform完成Job资源创建后会直接结束执行，不会等待Kubernetes Job运行完成，不符合流程要求

最终解决方案 #

1. 调整Terraform中kubernetes_job资源的配置参数：

spec {
  parallelism                = 1
  completions                = 1
  # 其余原有spec配置保持不变
}
wait_for_completion = false

2. 排查修复业务脚本逻辑：本次问题的实际根因是Job调用的rotate.sh脚本自身存在逻辑bug，修复脚本后即可实现3个Job并行执行的预期效果。

内容的提问来源于stack exchange，提问作者Theo Sweeny