如何配置AWS EC2实例启动时执行Kafka启动脚本？含两种方案

Alright, let's figure out how to get Kafka to automatically start up whenever your AWS EC2 instance boots. I've got two practical solutions for you—one using the Java AWS SDK to handle this remotely, and another using a bash script to set it up directly on the instance itself.

This approach is perfect if you need to automate setup across multiple EC2 instances without manually logging into each one. We'll use AWS Systems Manager (SSM) Run Command to send configuration commands to your instance through the Java SDK.

Prerequisites #

• Your EC2 instance has the SSM Agent installed (pre-installed on Amazon Linux 2, Amazon Linux, and Ubuntu 16.04+; manual installation may be needed for other distros).
• The IAM role attached to your EC2 instance has permissions to interact with SSM (e.g., the AmazonSSMFullAccess policy).
• Your Java environment includes the AWS SDK for SSM as a dependency.

Java Code Example #

import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.ssm.SsmClient;
import software.amazon.awssdk.services.ssm.model.SendCommandRequest;
import software.amazon.awssdk.services.ssm.model.SendCommandResponse;

public class KafkaAutoStartConfig {
    public static void main(String[] args) {
        // Update these values to match your environment
        Region awsRegion = Region.US_EAST_1;
        String instanceId = "i-1234567890abcdef0";
        String kafkaInstallDir = "/opt/kafka";
        String ec2User = "ec2-user"; // Use "ubuntu" for Ubuntu instances

        // Build the command to create a systemd service and enable auto-start
        String setupCommand = String.format("""
            sudo tee /etc/systemd/system/kafka.service << 'EOF'
            [Unit]
            Description=Apache Kafka Server
            Documentation=http://kafka.apache.org/documentation.html
            Requires=network.target
            After=network.target

            [Service]
            Type=simple
            User=%s
            ExecStart=%s/bin/kafka-server-start.sh %s/config/server.properties
            ExecStop=%s/bin/kafka-server-stop.sh
            Restart=on-failure

            [Install]
            WantedBy=multi-user.target
            EOF
            sudo systemctl daemon-reload
            sudo systemctl enable kafka.service
            """, ec2User, kafkaInstallDir, kafkaInstallDir, kafkaInstallDir);

        // Initialize SSM client and send the command
        try (SsmClient ssmClient = SsmClient.builder().region(awsRegion).build()) {
            SendCommandRequest request = SendCommandRequest.builder()
                    .instanceIds(instanceId)
                    .documentName("AWS-RunShellScript")
                    .parameters("commands", java.util.List.of(setupCommand))
                    .build();

            SendCommandResponse response = ssmClient.sendCommand(request);
            System.out.println("Successfully sent configuration command. Command ID: " + response.command().commandId());
        } catch (Exception e) {
            System.err.println("Failed to configure Kafka auto-start: " + e.getMessage());
            e.printStackTrace();
        }
    }
}

Key Notes #

• Double-check all placeholder values (region, instance ID, Kafka path, user) to match your setup.
• You can target multiple instances by adding more IDs to the instanceIds list.
• Verify the SSM Agent is running with sudo systemctl status amazon-ssm-agent.

If you're working with a single instance or prefer a hands-on approach, this bash script will configure a systemd service to start Kafka on boot. Systemd is the standard init system for most modern Linux distros (Amazon Linux 2, Ubuntu 16.04+, etc.).

Step 1: Create the Setup Script #

Create a file named setup-kafka-autostart.sh with the following content:

#!/bin/bash

# Update these variables to match your Kafka installation
KAFKA_DIR="/opt/kafka"
RUN_AS_USER="ec2-user" # Replace with "ubuntu" for Ubuntu instances

# Create systemd service file for Kafka
sudo tee /etc/systemd/system/kafka.service << 'EOF'
[Unit]
Description=Apache Kafka Server
Documentation=http://kafka.apache.org/documentation.html
Requires=network.target
After=network.target

[Service]
Type=simple
User=${RUN_AS_USER}
ExecStart=${KAFKA_DIR}/bin/kafka-server-start.sh ${KAFKA_DIR}/config/server.properties
ExecStop=${KAFKA_DIR}/bin/kafka-server-stop.sh
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

# Reload systemd to recognize the new service
sudo systemctl daemon-reload

# Enable Kafka to start on boot
sudo systemctl enable kafka.service

# Optional: Start Kafka immediately instead of waiting for the next reboot
# sudo systemctl start kafka.service

echo "Kafka auto-start configuration finished! Kafka will start on the next instance boot."

Step 2: Run the Script #

1. Make the script executable:

   chmod +x setup-kafka-autostart.sh
   

2. Execute the script with sudo privileges:

   sudo ./setup-kafka-autostart.sh
   

Alternative for Older Systems (rc.local) #

If your instance uses an older init system (e.g., Amazon Linux 1), add Kafka's start command to /etc/rc.local:

echo "su - ${RUN_AS_USER} -c '${KAFKA_DIR}/bin/kafka-server-start.sh ${KAFKA_DIR}/config/server.properties &'" | sudo tee -a /etc/rc.local
sudo chmod +x /etc/rc.local

Which Solution Should You Use? #

• Java SDK: Ideal for automation, scaling, or managing a fleet of EC2 instances.
• Bash Script: Great for quick one-off setups or including in EC2 user data (to configure auto-start when the instance is first launched).

内容的提问来源于stack exchange，提问作者andrea cenciarelli