Terraform配置托管EKS节点组时无法稳定覆盖最大Pod数

我尝试覆盖EKS节点的单节点最大Pod数，用eksctl创建托管节点组时操作顺畅，但用Terraform时，集群状态始终无法稳定复现：

• 首次执行terraform apply（无现有基础设施），t3.medium实例的单节点最大Pod数仍为默认值17；
• 对节点组Terraform配置做无关小修改后触发节点组重建，此时最大Pod数突然变为110，但我在启动模板中明确设置的是109。

尝试的两种方案 #

方案1：通过systemd配置覆盖kubelet参数 #

用户数据模板内容：

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: text/x-shellscript; charset="us-ascii"
#!/bin/bash
set -o xtrace
echo "KUBELET_EXTRA_ARGS=--max-pods=${max_pods}" >> /etc/systemd/system/kubelet.service.d/30-extra-args.conf
systemctl daemon-reload
systemctl restart kubelet
--//--

对应的Terraform启动模板配置：

resource "aws_launch_template" "eks" {
  name_prefix   = "${local.env}-${local.eks_name}-eks-nodes"
  instance_type = "t3.medium"
  user_data     = base64encode(templatefile("${path.module}/user_data.sh.tpl", {
    max_pods = 109, # 确保实际max pods值从此处获取
  }))

  tags = {
    Name = "${local.env}-${local.eks_name}-eks-nodes"
  }

  lifecycle {
    create_before_destroy = true
  }
}

output "launch_template_latest_version" {
  value = aws_launch_template.eks.latest_version
}

方案2：通过bootstrap命令传递kubelet参数 #

用户数据模板内容：

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: text/x-shellscript; charset="us-ascii"
#!/bin/bash

/etc/eks/bootstrap.sh ${cluster_name} \
  --use-max-pods false \
  --kubelet-extra-args '--max-pods=${max_pods}'
--//

但此方案直接导致节点组创建失败，出现超时错误。

已配置的VPC CNI补丁 #

通过kubectl provider的kubectl_manifest资源为aws-node DaemonSet启用前缀委派：

resource "kubectl_manifest" "aws_node_patch" {
  yaml_body = <<EOF
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: aws-node
  namespace: kube-system
spec:
  template:
    spec:
      containers:
      - name: aws-node
        env:
        - name: ENABLE_PREFIX_DELEGATION
          value: "true"
EOF

  depends_on = [
    aws_eks_addon.vpc_cni
  ]

}

节点组Terraform配置 #

resource "aws_eks_node_group" "general" {
  cluster_name    = aws_eks_cluster.eks.name
  version         = local.eks_version
  node_group_name = "general"
  node_role_arn   = aws_iam_role.nodes.arn

  subnet_ids = [
    aws_subnet.private_zone1.id,
    aws_subnet.private_zone2.id
  ]

  capacity_type = "ON_DEMAND"

  scaling_config {
    desired_size = 1
    max_size     = 2
    min_size     = 0
  }

  update_config {
    max_unavailable = 1
  }

  labels = {
    role = "general"
  }

  launch_template {
    id      = aws_launch_template.eks.id
    version = aws_launch_template.eks.latest_version
  }

  depends_on = [
    aws_iam_role_policy_attachment.amazon_eks_worker_node_policy,
    aws_iam_role_policy_attachment.amazon_eks_cni_policy,
    aws_iam_role_policy_attachment.amazon_ec2_container_registry_read_only
  ]

  # 允许外部调整节点数，避免Terraform计划产生差异
  lifecycle {
    ignore_changes = [scaling_config[0].desired_size]
  }
}

疑问 #

我是否遗漏了关键配置，或是参数应用的位置存在错误？

内容的提问来源于stack exchange，提问作者Cornul11