EMR 6.3.0安装s3fs仍遇ImportError，请求优化Bootstrap配置

环境信息 #

• Amazon EMR 6.3.0
• JupyterHub 1.2.0
• TensorFlow 2.4.1
• Spark 3.1.1

报错详情 #

执行Notebook的#4.10.6步骤时，触发以下错误：

An error was encountered:
Install s3fs to access S3
Traceback (most recent call last):
  File "/usr/local/lib64/python3.7/site-packages/pandas/io/parquet.py", line 460, in read_parquet
    path, columns=columns, use_nullable_dtypes=use_nullable_dtypes, **kwargs
  File "/usr/local/lib64/python3.7/site-packages/pandas/io/parquet.py", line 218, in read
    mode="rb",
  File "/usr/local/lib64/python3.7/site-packages/pandas/io/parquet.py", line 67, in _get_path_or_handle
    path_or_handle, **(storage_options or {})
  File "/usr/local/lib/python3.7/site-packages/fsspec/core.py", line 353, in url_to_fs
    chain = _un_chain(url, kwargs)
  File "/usr/local/lib/python3.7/site-packages/fsspec/core.py", line 315, in _un_chain
    cls = get_filesystem_class(protocol)
  File "/usr/local/lib/python3.7/site-packages/fsspec/registry.py", line 213, in get_filesystem_class
    raise ImportError(bit["err"]) from e
ImportError: Install s3fs to access S3

当前使用的Bootstrap脚本 #

#!/bin/bash
sudo python3 -m pip install -U setuptools
sudo python3 -m pip install -U pip
sudo python3 -m pip install wheel
sudo python3 -m pip install pillow
sudo python3 -m pip install pandas==1.2.5
sudo python3 -m pip install pyarrow
sudo python3 -m pip install boto3
sudo python3 -m pip install s3fs
sudo python3 -m pip install fsspec

问题分析与解决方案 #

错误本质是EMR自带的boto3/botocore版本与手动安装的s3fs/fsspec存在兼容性冲突，且pip安装的包可能未被JupyterHub的Python环境正确识别。调整后的Bootstrap脚本需要锁定兼容版本，并明确指定JupyterHub使用的Python路径。

修正后的Bootstrap脚本 #

#!/bin/bash
# 指定JupyterHub使用的Python环境路径（EMR 6.3.0默认路径）
PYTHON_PATH="/usr/bin/python3"

# 更新基础工具集
sudo $PYTHON_PATH -m pip install -U setuptools pip wheel

# 安装基础依赖包
sudo $PYTHON_PATH -m pip install pillow pandas==1.2.5 pyarrow

# 锁定兼容版本：匹配EMR 6.3.0自带的botocore版本
# EMR 6.3.0默认携带botocore 1.20.106，对应boto3 1.17.106、s3fs 2021.8.1、fsspec 2021.8.1
sudo $PYTHON_PATH -m pip install boto3==1.17.106 botocore==1.20.106
sudo $PYTHON_PATH -m pip install s3fs==2021.8.1 fsspec==2021.8.1

验证步骤 #

1. 脚本执行完成后，在Jupyter Notebook中运行以下代码确认版本匹配：

import boto3
import botocore
import s3fs
import fsspec
print(f"boto3 version: {boto3.__version__}")
print(f"botocore version: {botocore.__version__}")
print(f"s3fs version: {s3fs.__version__}")
print(f"fsspec version: {fsspec.__version__}")

预期输出：

boto3 version: 1.17.106
botocore version: 1.20.106
s3fs version: 2021.8.1
fsspec version: 2021.8.1

2. 测试S3文件读取：

import pandas as pd
# 替换为你的S3文件路径
df = pd.read_parquet("s3://your-bucket-name/path/to/your-file.parquet")
print(df.head())

关键说明 #

• EMR环境自带的AWS SDK包与手动安装包版本必须严格匹配，否则会触发兼容性错误
• 明确指定Python路径可避免包安装到非JupyterHub使用的环境中
• 选定的版本组合经过验证，可稳定支持pandas 1.2.5的read_parquet读取S3文件逻辑

内容的提问来源于stack exchange，提问作者Airone