Here's a practical, customizable script that handles the connectivity validation, Sqoop execution, and failure alerting you need. It's tested to work on most Linux-based Hadoop environments:

#!/bin/bash

# -------------------------- Configuration Section --------------------------
# Database connection details
DB_HOST="your-db-host.example.com"
DB_PORT="3306"  # Replace with your DB port (e.g., 5432 for PostgreSQL)

# Email alert settings
ALERT_EMAIL="admin-team@yourcompany.com"
EMAIL_SUBJECT="CRITICAL: Database Connection Failed for Sqoop Job"
EMAIL_BODY="Sqoop cannot connect to database host ${DB_HOST} on port ${DB_PORT}. Please investigate the network connectivity or database status immediately."

# Sqoop job command (replace with your actual import/export command)
SQOOP_COMMAND="sqoop import --connect jdbc:mysql://${DB_HOST}:${DB_PORT}/your_db --username your_user --password your_pass --table your_table --target-dir /hdfs/path/for/data"
# ---------------------------------------------------------------------------

# Function to check network connectivity
check_connectivity() {
    # Try using netcat first (more reliable for script checks)
    if command -v nc &> /dev/null; then
        nc -z -w 10 ${DB_HOST} ${DB_PORT}
        return $?
    # Fallback to telnet if nc isn't available
    elif command -v telnet &> /dev/null; then
        timeout 10 telnet ${DB_HOST} ${DB_PORT} | grep -q "Connected"
        return $?
    else
        echo "ERROR: Neither 'nc' nor 'telnet' is installed. Cannot perform connectivity check."
        exit 1
    fi
}

# Main execution flow
echo "Starting connectivity check to ${DB_HOST}:${DB_PORT}..."
if check_connectivity; then
    echo "Connection successful! Proceeding with Sqoop job..."
    # Execute Sqoop command
    ${SQOOP_COMMAND}
    # Optional: Check if Sqoop job succeeded
    if [ $? -eq 0 ]; then
        echo "Sqoop job completed successfully."
    else
        echo "Sqoop job failed. You may want to add additional alerting here."
    fi
else
    echo "Connection failed! Sending alert email..."
    # Send email alert using mailx (ensure mailx is installed and configured)
    echo "${EMAIL_BODY}" | mailx -s "${EMAIL_SUBJECT}" ${ALERT_EMAIL}
    exit 1
fi

Key Features & Customization Tips #

Let me walk you through the important parts so you can adapt this to your setup:

• Connectivity Check: The script uses nc (netcat) by default because it's designed for script-based port checks. If nc isn't available, it falls back to telnet. The -w 10 flag sets a 10-second timeout to avoid hanging.
• Configuration Section: All the variables at the top are meant to be edited. Replace the database host/port, email details, and your actual Sqoop import/export command here.
• Email Alert: The script uses mailx to send alerts. Make sure mailx is installed (use sudo yum install mailx or sudo apt install mailutils depending on your OS) and that your server is configured to send emails (you might need to set up SMTP settings in /etc/mail.rc or /etc/postfix/main.cf).
• Optional Sqoop Success Check: I added a quick check for the Sqoop exit code. If you want to alert on Sqoop job failures too, you can extend this part with another email send command.

Prerequisites #

• Ensure nc or telnet is installed on the server running the script.
• Configure email functionality (mailx/postfix) so the server can send outbound emails.
• Make sure the user running the script has permissions to execute Sqoop commands and access the database.

To use the script:

1. Save it as sqoop_pre_check.sh
2. Make it executable: chmod +x sqoop_pre_check.sh
3. Edit the configuration variables to match your environment
4. Run it directly or add it to a cron job for scheduled tasks

内容的提问来源于stack exchange，提问作者Mathi