使用AWS CodeDeploy在EC2实例部署Storm拓扑的技术求助

Let's break this down into actionable steps—you're absolutely right to use CodeDeploy's BeforeInstall hook for this pre-deployment setup. Here's a complete implementation tailored to your needs:

1. Create the BeforeInstall Hook Script #

Save this as install_storm.sh (don’t forget to make it executable with chmod +x install_storm.sh):

#!/bin/bash
set -e  # Exit immediately if any command fails

# --------------------------
# Step 1: Check & Install Java (Storm requires Java 8/11)
# --------------------------
if ! command -v java &> /dev/null; then
    echo "Java not found, installing OpenJDK 8..."
    sudo yum install -y java-1.8.0-openjdk-devel  # For Amazon Linux/RHEL
    # For Ubuntu/Debian, use: sudo apt-get update && sudo apt-get install -y openjdk-8-jdk
fi

# Verify Java installation
java_version=$(java -version 2>&1 | awk -F '"' '/version/ {print $2}')
echo "Installed Java version: $java_version"

# --------------------------
# Step 2: Check if Storm is already installed
# --------------------------
STORM_VERSION="2.4.0"  # Replace with your preferred Storm version
STORM_INSTALL_DIR="/opt/apache-storm"

if command -v storm &> /dev/null; then
    installed_version=$(storm version | awk '/Storm/ {print $2}')
    echo "Storm $installed_version is already installed. Skipping setup."
    exit 0
fi

# --------------------------
# Step 3: Download & Install Storm
# --------------------------
echo "Installing Storm $STORM_VERSION..."
sudo mkdir -p $STORM_INSTALL_DIR
cd /tmp
sudo wget https://dlcdn.apache.org/storm/apache-storm-$STORM_VERSION/apache-storm-$STORM_VERSION.tar.gz
sudo tar -xzf apache-storm-$STORM_VERSION.tar.gz -C $STORM_INSTALL_DIR --strip-components=1

# Add Storm to system PATH for all users
echo "export PATH=\$PATH:$STORM_INSTALL_DIR/bin" | sudo tee -a /etc/profile.d/storm.sh
source /etc/profile.d/storm.sh

# --------------------------
# Step 4: Configure Storm Cluster Settings
# --------------------------
STORM_CONFIG="$STORM_INSTALL_DIR/conf/storm.yaml"

# Set Nimbus host (replace with your Nimbus EC2 instance's private IP)
sudo sed -i 's|# nimbus.seeds: \["nimbus1"\]|nimbus.seeds: ["10.0.0.10"]|' $STORM_CONFIG

# Set Zookeeper quorum (replace with your ZK cluster addresses)
sudo sed -i 's|# storm.zookeeper.servers:|storm.zookeeper.servers:|' $STORM_CONFIG
sudo sed -i 's|#     - "zk1"|    - "10.0.0.11"|' $STORM_CONFIG
sudo sed -i 's|#     - "zk2"|    - "10.0.0.12"|' $STORM_CONFIG

# Configure supervisor worker ports (adjust number based on your instance capacity)
sudo sed -i 's|# supervisor.slots.ports:|supervisor.slots.ports:|' $STORM_CONFIG
sudo sed -i 's|#     - 6700|    - 6700|' $STORM_CONFIG
sudo sed -i 's|#     - 6701|    - 6701|' $STORM_CONFIG
sudo sed -i 's|#     - 6702|    - 6702|' $STORM_CONFIG

# --------------------------
# Step 5: Start Storm Supervisor Service
# --------------------------
echo "Starting Storm Supervisor..."
sudo $STORM_INSTALL_DIR/bin/storm supervisor > /var/log/storm-supervisor.log 2>&1 &

echo "Storm installation and configuration completed successfully!"

2. Update Your CodeDeploy AppSpec.yml #

Add the BeforeInstall hook to your appspec.yml to trigger the script before deploying your JAR:

version: 0.0
os: linux
files:
  - source: /your-storm-topology.jar
    destination: /opt/storm-topologies/
hooks:
  BeforeInstall:
    - location: scripts/install_storm.sh
      timeout: 300  # 5 minutes, adjust based on download speed
      runas: root
  AfterInstall:
    - location: scripts/submit_topology.sh
      timeout: 120
      runas: root

3. Create the AfterInstall Script for Topology Deployment #

Save this as submit_topology.sh (make it executable too):

#!/bin/bash
set -e

TOPOLOGY_JAR="/opt/storm-topologies/your-storm-topology.jar"
TOPOLOGY_CLASS="com.yourcompany.storm.YourTopology"  # Replace with your topology class
TOPOLOGY_NAME="MyProductionStormTopology"

# Optional: Kill existing topology before deploying the new one
# storm kill $TOPOLOGY_NAME -w 10  # Wait 10 seconds for graceful shutdown

# Submit the new topology to the cluster
storm jar $TOPOLOGY_JAR $TOPOLOGY_CLASS $TOPOLOGY_NAME

echo "Storm topology submitted successfully!"

Key Tips for Success: #

• Customize Values: Replace Storm version, Nimbus/Zookeeper IPs, and topology details with your actual cluster setup.
• OS Compatibility: Swap yum to apt-get commands if you’re using Ubuntu/Debian instances.
• Logging: Check /var/log/storm-supervisor.log for supervisor issues, and CodeDeploy logs in /opt/codedeploy-agent/deployment-root/ for deployment failures.
• Permissions: Ensure your CodeDeploy service role has permissions to execute commands on EC2 instances (e.g., a custom role with AmazonEC2FullAccess or restricted granular permissions).

内容的提问来源于stack exchange，提问作者Arun.D