在AWS Lambda中使用Boto3与Pandas触发段错误问题求助

问题 #

使用Python 3.9搭配boto3、pandas1.5.3和pandera0.8.1，requirements.txt仅配置pandas与pandera。代码需同时使用Pandas和S3客户端，本地运行正常，但通过容器镜像部署到AWS Lambda时触发Error: Runtime exited with error: signal: segmentation fault。移除pandas导入后代码可正常运行，已尝试修改Python、Pandas和boto3版本，Lambda已配置最大内存且拥有S3全权限。

代码 #

import faulthandler
import pandas as pd
import boto3

faulthandler.enable()


def lambda_handler(event, context):
    s3_client = boto3.client('s3')

    # Not implemented yet

    return {"status_code": 200}

容器镜像配置（Dockerfile） #

FROM public.ecr.aws/lambda/python:3.9

COPY src/ ${LAMBDA_TASK_ROOT}

COPY requirements/requirements.txt  .

RUN  pip3 install -r requirements.txt --target "${LAMBDA_TASK_ROOT}"

CMD [ "lambda_function.lambda_handler" ]

错误堆栈 #

File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
File "<frozen importlib._bootstrap_external>", line 1173 in create_module
File "<frozen importlib._bootstrap>", line 565 in module_from_spec
File "<frozen importlib._bootstrap>", line 666 in _load_unlocked
File "<frozen importlib._bootstrap>", line 986 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
File "/var/lang/lib/python3.9/multiprocessing/shared_memory.py", line 23 in <module>
File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
File "<frozen importlib._bootstrap_external>", line 850 in exec_module
File "<frozen importlib._bootstrap>", line 680 in _load_unlocked
File "<frozen importlib._bootstrap>", line 986 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1058 in _handle_fromlist
File "/var/lang/lib/python3.9/multiprocessing/managers.py", line 36 in <module>
File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
File "<frozen importlib._bootstrap_external>", line 850 in exec_module
File "<frozen importlib._bootstrap>", line 680 in _load_unlocked
File "<frozen importlib._bootstrap>", line 986 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
File "/var/runtime/s3transfer/compat.py", line 94 in <module>
File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
File "<frozen importlib._bootstrap_external>", line 850 in exec_module
File "<frozen importlib._bootstrap>", line 680 in _load_unlocked
File "<frozen importlib._bootstrap>", line 986 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
File "/var/runtime/s3transfer/__init__.py", line 143 in <module>
File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
File "<frozen importlib._bootstrap_external>", line 850 in exec_module
File "<frozen importlib._bootstrap>", line 680 in _load_unlocked
File "<frozen importlib._bootstrap>", line 986 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 972 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
File "/var/runtime/boto3/s3/transfer.py", line 131 in <module>
File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
File "<frozen importlib._bootstrap_external>", line 850 in exec_module
File "<frozen importlib._bootstrap>", line 680 in _load_unlocked
File "<frozen importlib._bootstrap>", line 986 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
File "/var/runtime/boto3/s3/inject.py", line 18 in <module>
File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
File "<frozen importlib._bootstrap_external>", line 850 in exec_module
File "<frozen importlib._bootstrap>", line 680 in _load_unlocked
File "<frozen importlib._bootstrap>", line 986 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
File "/var/runtime/boto3/utils.py", line 56 in import_module
File "/var/runtime/boto3/utils.py", line 65 in _handler
File "/var/runtime/botocore/hooks.py", line 239 in _emit
File "/var/runtime/botocore/hooks.py", line 256 in emit
File "/var/runtime/botocore/hooks.py", line 412 in emit
File "/var/runtime/botocore/client.py", line 201 in _create_client_class
File "/var/runtime/botocore/client.py", line 147 in create_client
File "/var/runtime/botocore/session.py", line 997 in create_client
File "/var/runtime/boto3/session.py", line 299 in client
File "/var/runtime/boto3/__init__.py", line 92 in client
File "/var/task/lambda_function.py", line 21 in lambda_handler
File "/var/runtime/awslambdaric/bootstrap.py", line 188 in handle_event_request
File "/var/runtime/awslambdaric/bootstrap.py", line 499 in run
File "/var/runtime/awslambdaric/__main__.py", line 21 in main
File "/var/runtime/bootstrap.py", line 57 in main
File "/var/runtime/bootstrap.py", line 60 in <module>

解决方法 #

1. 强制源码编译pandas #

段错误多因预编译的pandas二进制包与Lambda的Amazon Linux 2运行环境不兼容。修改Dockerfile的pip安装命令，强制从源码编译：

RUN pip3 install -r requirements.txt --no-binary :all: --target "${LAMBDA_TASK_ROOT}"

注：源码编译会增加镜像构建时长，需确保构建环境资源充足。

2. 使用优化后的Lambda基础镜像 #

替换官方基础镜像为针对Lambda优化的版本，这类镜像预配置了兼容Lambda的依赖库：

FROM public.ecr.aws/awsguru/aws-lambda-python:3.9

COPY src/ ${LAMBDA_TASK_ROOT}
COPY requirements/requirements.txt .

RUN pip3 install -r requirements.txt --target "${LAMBDA_TASK_ROOT}"

CMD [ "lambda_function.lambda_handler" ]

3. 将pandas打包为Lambda层 #

改用Lambda原生Python运行时，把pandas和pandera打包成层：

• 在Amazon Linux 2环境（EC2/Cloud9均可）执行：

  mkdir -p python/lib/python3.9/site-packages
  pip3 install pandas==1.5.3 pandera==0.8.1 -t python/lib/python3.9/site-packages --no-binary :all:
  zip -r pandas-pandera-layer.zip python/
  

• 将zip包上传为Lambda层，关联到目标Lambda函数即可。

4. 升级pandas版本 #

尝试升级pandas到1.5.x系列最新补丁版，或切换到2.x系列，新版本可能修复了Lambda环境兼容性问题。

内容的提问来源于stack exchange，提问作者Henrique Koga