EMR引导操作安装Python模块失败：返回非零错误码

Hey, I’ve run into this exact non-zero return code issue with EMR bootstrap actions before—let’s break down what might be going wrong and how to fix it:

1. First, Check Your EMR Cluster’s OS Version #

Your script uses apt-get, which is for Debian/Ubuntu systems. But most EMR clusters default to Amazon Linux/Amazon Linux 2 (based on RHEL/CentOS), where apt-get doesn’t exist at all. That’s a super common mistake!

• If you’re using Amazon Linux 2: Replace all apt-get commands with yum
• If you intentionally chose Ubuntu for your EMR cluster: Keep apt-get but fix other script issues (more on that below)

2. Fix Up Your Bootstrap Script #

Your original script has a few gaps that can cause failures. Here’s revised versions for both OS types:

For Ubuntu EMR Clusters: #

#!/bin/sh
set -e -x

# Update package cache first to avoid missing packages
sudo apt-get update -y

# Install Python 3 tools (Python 2 is deprecated, so use 3.x)
sudo apt-get install -y python3-setuptools python3-pip

# Upgrade pip to the latest version before installing packages
sudo pip3 install -U pip

# Install pandas (add any other modules you need here)
sudo pip3 install -U pandas

For Amazon Linux 2 EMR Clusters: #

#!/bin/sh
set -e -x

# Update yum cache
sudo yum update -y

# Install Python 3 and pip
sudo yum install -y python3 python3-pip

# Upgrade pip
sudo pip3 install -U pip

# Install pandas
sudo pip3 install -U pandas

Key fixes here:

• Added -y to all install commands: This skips interactive prompts (which will hang your bootstrap action if left out)
• Switched to Python 3: Python 2 is end-of-life, and most modern EMR versions prioritize Python 3
• Explicitly used pip3 to avoid version conflicts

3. Verify S3 Script Permissions #

Make sure your EMR cluster’s EC2 instance role (usually EMR_EC2_DefaultRole) has permission to read the script from s3://mybucket/bootstrap/install_python_modules.sh. You can:

• Check the IAM role’s policy to ensure it includes s3:GetObject for your bucket path
• For testing, temporarily set the script’s S3 object permissions to "Public read" (not recommended for production!)

4. Dig Into EMR Logs to Find Exact Errors #

The non-zero code just tells you something failed—but the logs will show what failed. Here’s how to access them:

1. Go to your EMR cluster’s detail page in the AWS Console
2. Switch to the Logs tab
3. Navigate to the bootstrap action logs path (usually s3://<your-log-bucket>/<cluster-id>/node/<node-id>/bootstrap-actions/1/)
4. Download stdout and stderr files—look for lines like command not found or error: command failed to pinpoint the issue

For example:

• If stderr says apt-get: command not found, you’re running Amazon Linux, not Ubuntu
• If pandas fails to install, you’re missing compile dependencies (see next section)

5. Add Compile Dependencies for Pandas #

Pandas requires some system libraries to compile during installation. If you see errors about missing headers or compilers, add these lines to your script before installing pandas:

For Ubuntu: #

sudo apt-get install -y build-essential python3-dev libssl-dev libffi-dev libxml2-dev libxslt1-dev zlib1g-dev

For Amazon Linux 2: #

sudo yum install -y gcc python3-devel openssl-devel libxml2-devel libxslt-devel zlib-devel

内容的提问来源于stack exchange，提问作者Simonidas