如何在AWS EMR上安装OpenSSL 1.1以适配Python 3.10

背景 #

我正在为运行PySpark的EMR项目编写Bootstrap脚本，目标是安装Python 3.10。但Python 3.10要求OpenSSL 1.1或更高版本，而EMR默认自带的是OpenSSL 1.0.2k-fips（可通过openssl version查看），且无法直接升级系统默认版本。

已尝试的无效方法 #

方法1：直接用yum安装openssl11-devel #

执行命令：

sudo yum install -y openssl11-devel <其他依赖包>

遇到EMR仓库优先级保护，且出现依赖冲突报错：

13 packages excluded due to repository priority protections
Package gcc-7.3.1-15.amzn2.x86_64 already installed and latest version
Package 2:tar-1.26-35.amzn2.0.2.x86_64 already installed and latest version
Package gzip-1.5-10.amzn2.0.1.x86_64 already installed and latest version
Package wget-1.14-18.amzn2.1.x86_64 already installed and latest version
Package 1:make-3.82-24.amzn2.x86_64 already installed and latest version
Resolving Dependencies
--> Running transaction check
---> Package bzip2-devel.x86_64 0:1.0.6-13.amzn2.0.3 will be installed
---> Package libffi-devel.x86_64 0:3.0.13-18.amzn2.0.2 will be installed
---> Package openssl11-devel.x86_64 1:1.1.1g-12.amzn2.0.15 will be installed
--> Processing Conflict: 1:openssl11-devel-1.1.1g-12.amzn2.0.15.x86_64 conflicts openssl-devel
--> Finished Dependency Resolution
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest

方法2：禁用优先级并添加--skip-broken参数 #

执行命令：

sudo yum install -y --disableplugin=priorities --skip-broken openssl11-devel <其他依赖包>

结果openssl11-devel因依赖问题被跳过：

Resolving Dependencies
--> Running transaction check
---> Package bzip2-devel.x86_64 0:1.0.6-13.amzn2.0.3 will be installed
---> Package libffi-devel.x86_64 0:3.0.13-18.amzn2.0.2 will be installed
---> Package openssl11-devel.x86_64 1:1.1.1g-12.amzn2.0.15 will be installed
--> Processing Conflict: 1:openssl11-devel-1.1.1g-12.amzn2.0.15.x86_64 conflicts openssl-devel
--> Running transaction check
---> Package openssl11-devel.x86_64 1:1.1.1g-12.amzn2.0.15 will be installed
--> Running transaction check
---> Package bzip2-devel.x86_64 0:1.0.6-13.amzn2.0.3 will be installed
---> Package libffi-devel.x86_64 0:3.0.13-18.amzn2.0.2 will be installed
--> Finished Dependency Resolution

Packages skipped because of dependency problems:
    1:openssl11-devel-1.1.1g-12.amzn2.0.15.x86_64 from amzn2-core

方法3：卸载系统默认OpenSSL后安装1.1版本 #

执行命令：

sudo yum remove openssl

发现会卸载大量系统依赖包（包括cloud-init、authconfig等关键组件），且Bootstrap脚本在此步骤冻结一小时后失败。

可行解决方案：源码编译OpenSSL 1.1并指定路径编译Python #

核心思路是不修改系统默认OpenSSL，而是从源码编译OpenSSL 1.1到自定义目录，然后编译Python时指定该OpenSSL的路径，确保Python使用新的SSL版本，同时不破坏系统依赖。

修改后的Bootstrap脚本如下：

#!/usr/bin/env bash

PYTHON_VERSION=3.10.6
OPENSSL_VERSION=1.1.1w  # 可选择最新的1.1.x稳定版本

echo "-----------------------------------------------"
echo "安装编译依赖"
echo "-----------------------------------------------"
sudo yum update -y
sudo yum install -y gcc bzip2-devel libffi-devel tar gzip wget make

echo "-----------------------------------------------"
echo "编译安装OpenSSL 1.1到自定义目录"
echo "-----------------------------------------------"
wget https://www.openssl.org/source/openssl-${OPENSSL_VERSION}.tar.gz
tar xzf openssl-${OPENSSL_VERSION}.tar.gz
cd openssl-${OPENSSL_VERSION}
./config --prefix=/usr/local/openssl11 --openssldir=/usr/local/openssl11
make -j$(nproc)
sudo make install

# 临时设置环境变量，让Python编译时找到新的OpenSSL
export LD_LIBRARY_PATH=/usr/local/openssl11/lib:$LD_LIBRARY_PATH
export CPPFLAGS="-I/usr/local/openssl11/include"
export LDFLAGS="-L/usr/local/openssl11/lib"

echo "-----------------------------------------------"
echo "编译安装Python ${PYTHON_VERSION}"
echo "-----------------------------------------------"
cd ..
wget https://www.python.org/ftp/python/${PYTHON_VERSION}/Python-${PYTHON_VERSION}.tgz
tar xzf Python-${PYTHON_VERSION}.tgz
cd Python-${PYTHON_VERSION}
# 配置Python时指定OpenSSL路径，并启用优化
./configure --enable-optimizations --with-openssl=/usr/local/openssl11
make -j$(nproc)
sudo make altinstall prefix=~/local  # 安装到用户本地目录，不覆盖系统Python
ln -s ~/local/bin/python3.10 ~/local/bin/python

# 验证Python的SSL版本
~/local/bin/python3.10 -c "import ssl; print(ssl.OPENSSL_VERSION)"

关键说明 #

1. 不卸载系统OpenSSL：避免破坏EMR的核心依赖组件，防止系统异常。
2. 自定义OpenSSL安装路径：将OpenSSL 1.1安装到/usr/local/openssl11，与系统版本隔离。
3. 编译Python指定OpenSSL路径：通过--with-openssl参数让Python使用新编译的SSL库，同时设置环境变量确保编译时能找到头文件和库。
4. 使用altinstall：避免覆盖系统默认的Python版本，防止影响EMR原有服务。

内容的提问来源于stack exchange，提问作者mbaggio