如何在Terraform中完成Kubernetes集群EC2实例Hosts文件填充后调用对应Ansible Playbook

Got it, let's tackle this problem step by step. The core issue here is that you need to first populate /etc/hosts on every cluster instance with all private IPs of the cluster nodes, then run your respective Ansible playbooks for master and workers. Your current null_resource is writing to your local machine's /etc/hosts instead of the EC2 instances, which is the first thing we need to fix. Here are two practical solutions:

Solution 1: Use Terraform remote-exec to Configure Hosts First, Then Run Playbooks #

This approach keeps most logic in Terraform, using remote-exec to update /etc/hosts on each EC2 instance before triggering the Ansible playbooks.

First, update your aws_instance resources to remove the embedded provisioner blocks (we'll move that logic to separate null resources):

# Create Kubernetes Master EC2 Instance
resource "aws_instance" "kubernetes-master" {
  provider = aws.region-master
  ami = "ami-0747bdcabd34c712a" # Ubuntu 18 x64
  instance_type = var.instance-type
  key_name = aws_key_pair.master-key.key_name
  associate_public_ip_address = "true"
  vpc_security_group_ids = [aws_security_group.kubernetes-sg.id]
  subnet_id = aws_subnet.subnet_1.id
  tags = {
    Name = "kubernetes_master_tf"
  }
  depends_on = [
    aws_main_route_table_association.set-master-default-rt-assoc
  ]
}

# Create Kubernetes Worker EC2 Instances
resource "aws_instance" "kubernetes-worker" {
  provider = aws.region-master
  count = var.workers-count
  ami = "ami-0747bdcabd34c712a" # Ubuntu 18 x64
  instance_type = var.instance-type
  key_name = aws_key_pair.master-key.key_name
  associate_public_ip_address = "true"
  vpc_security_group_ids = [aws_security_group.kubernetes-sg.id]
  subnet_id = aws_subnet.subnet_1.id
  tags = {
    Name = join("_", ["kubernetes_worker_tf", count.index + 1])
  }
  depends_on = [
    aws_main_route_table_association.set-master-default-rt-assoc
  ]
}

Next, define a local variable to combine master and worker instance details for easier iteration:

locals {
  cluster_instances = concat(
    [
      {
        private_ip = aws_instance.kubernetes-master.private_ip
        hostname   = aws_instance.kubernetes-master.tags.Name
        instance_id = aws_instance.kubernetes-master.id
      }
    ],
    [for idx, worker in aws_instance.kubernetes-worker : {
      private_ip = worker.private_ip
      hostname   = worker.tags.Name
      instance_id = worker.id
    }]
  )
}

Add a null resource to configure /etc/hosts on every cluster instance:

# Configure /etc/hosts on all cluster instances
resource "null_resource" "configure_cluster_hosts" {
  count = length(local.cluster_instances)

  depends_on = [aws_instance.kubernetes-master, aws_instance.kubernetes-worker]

  provisioner "remote-exec" {
    connection {
      type        = "ssh"
      user        = "ubuntu" # Ubuntu default username; adjust if using a different OS
      private_key = file(var.private_key_path) # Path to your SSH private key (define this variable)
      host        = local.cluster_instances[count.index].private_ip
    }

    inline = [
      # Clean up existing cluster entries to avoid duplicates
      "sudo sed -i '/kubernetes_master_tf/d' /etc/hosts",
      "sudo sed -i '/kubernetes_worker_tf/d' /etc/hosts",
      # Write all cluster instance entries to /etc/hosts
      "${join("\n", [for instance in local.cluster_instances : "echo '${instance.private_ip} ${instance.hostname}' | sudo tee -a /etc/hosts"])}"
    ]
  }
}

Finally, add null resources to run the Ansible playbooks after hosts are configured:

# Run Kubernetes Master Ansible Playbook
resource "null_resource" "run_master_playbook" {
  depends_on = [null_resource.configure_cluster_hosts]

  provisioner "local-exec" {
    command = <<EOF
aws --profile ${var.profile} ec2 wait instance-status-ok --region ${var.region-master} --instance-ids ${aws_instance.kubernetes-master.id}
ansible-playbook --extra-vars 'passed_in_hosts=tag_Name_${aws_instance.kubernetes-master.tags.Name}' ansible_templates/kubernetes-master.yml
EOF
  }
}

# Run Kubernetes Worker Ansible Playbooks
resource "null_resource" "run_worker_playbooks" {
  count = var.workers-count

  depends_on = [null_resource.configure_cluster_hosts]

  provisioner "local-exec" {
    command = <<EOF
aws --profile ${var.profile} ec2 wait instance-status-ok --region ${var.region-master} --instance-ids ${aws_instance.kubernetes-worker[count.index].id}
ansible-playbook --extra-vars 'passed_in_hosts=tag_Name_${aws_instance.kubernetes-worker[count.index].tags.Name}' ansible_templates/kubernetes-worker.yml
EOF
  }
}

Solution 2: Integrate Hosts Configuration into Ansible (Cleaner Approach) #

Since you're already using Ansible for cluster setup, it makes sense to move the /etc/hosts configuration into Ansible. This reduces Terraform complexity and leverages Ansible's strength in configuration management.

First, update your aws_instance resources like in Solution 1 (remove embedded provisioners). Then add a single null resource to trigger a master Ansible playbook:

# Trigger full cluster setup via Ansible
resource "null_resource" "run_cluster_setup" {
  depends_on = [aws_instance.kubernetes-master, aws_instance.kubernetes-worker]

  provisioner "local-exec" {
    command = <<EOF
# Wait for all instances to be ready
aws --profile ${var.profile} ec2 wait instance-status-ok --region ${var.region-master} --instance-ids ${aws_instance.kubernetes-master.id} ${join(" ", aws_instance.kubernetes-worker.*.id)}
# Pass cluster details to Ansible
ansible-playbook \
  --extra-vars 'master_ip=${aws_instance.kubernetes-master.private_ip} master_hostname=${aws_instance.kubernetes-master.tags.Name}' \
  --extra-vars 'worker_ips=${join(",", aws_instance.kubernetes-worker.*.private_ip)} worker_hostnames=${join(",", aws_instance.kubernetes-worker.*.tags.Name)}' \
  ansible_templates/cluster-setup.yml
EOF
  }
}

Create a master Ansible playbook cluster-setup.yml that configures hosts first, then runs your existing playbooks:

- name: Populate /etc/hosts for all cluster nodes
  hosts: all
  become: true
  tasks:
    - name: Add master node to /etc/hosts
      lineinfile:
        path: /etc/hosts
        line: "{{ master_ip }} {{ master_hostname }}"
        state: present
        owner: root
        group: root
        mode: '0644'

    - name: Add worker nodes to /etc/hosts
      lineinfile:
        path: /etc/hosts
        line: "{{ item.0 }} {{ item.1 }}"
        state: present
        owner: root
        group: root
        mode: '0644'
      with_together:
        - "{{ worker_ips.split(',') }}"
        - "{{ worker_hostnames.split(',') }}"

- name: Configure Kubernetes Master
  import_playbook: kubernetes-master.yml
  hosts: tag_Name_{{ master_hostname }}

- name: Configure Kubernetes Workers
  import_playbook: kubernetes-worker.yml
  hosts: tag_Name_kubernetes_worker_tf_*

Key Notes #

• For SSH connections: Ensure your private key has correct permissions (chmod 600) and that your security groups allow SSH access from your machine.
• Ansible Inventory: Make sure Ansible can discover your EC2 instances (use AWS dynamic inventory or tag-based targeting as you're already doing).
• OS Adjustments: If you're not using Ubuntu, update the SSH username (e.g., ec2-user for Amazon Linux).

内容的提问来源于stack exchange，提问作者Telion