升级Python3.11与Selenium4.18.0后AWS Lambda爬虫故障求助

我在AWS Lambda上使用Python+Selenium开发爬虫，将Python升级至3.11、Selenium升级至4.18.0后，爬虫直接停止工作。

我的Selenium代码 #

import os
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service

def get_headless_driver():
    options = Options()
    service = Service(executable_path=r'/opt/chromedriver')
    options.binary_location = '/opt/headless-chromium'
    options.add_argument('--headless')
    options.add_argument('--no-sandbox')
    options.add_argument('--single-process')
    options.add_argument('--disable-dev-shm-usage')
    options.add_argument('--window-size=1920x1080')
    options.add_argument('--start-maximized')

    return webdriver.Chrome(service=service, options=options)

def get_selenium_driver():
    return get_local_driver() if os.environ.get('STAGE') == 'local' else get_headless_driver()

ChromeDriver安装脚本 #

#! /bin/bash

# exit when any command fails
set -e

NOCOLOR='\033[0m'
GREEN='\033[0;32m'

echo -e "\n${GREEN}Installing 'headless' layer dependencies...${NOCOLOR}\n"
sudo mkdir -p layers/headless && sudo chmod 777 layers/headless && cd layers/headless

# https://github.com/adieuadieu/serverless-chrome/issues/133
echo -e "\n${GREEN}Installing Chrome Driver v2.37...${NOCOLOR}\n"
sudo curl -SL https://chromedriver.storage.googleapis.com/2.37/chromedriver_linux64.zip > chromedriver.zip
sudo chmod 755 chromedriver.zip
sudo unzip chromedriver.zip
sudo rm chromedriver.zip

echo -e "\n${GREEN}Installing Serverless Chrome v1.0.0-37...${NOCOLOR}\n"
sudo curl -SL https://github.com/adieuadieu/serverless-chrome/releases/download/v1.0.0-41/stable-headless-chromium-amazonlinux-2017-03.zip > headless-chromium.zip
sudo chmod 755 headless-chromium.zip
sudo unzip headless-chromium.zip
sudo rm headless-chromium.zip

cd ../../

错误信息 #

Message: Service /opt/chromedriver unexpectedly exited. Status code was: 127

问题根源 #

错误码127本质是二进制文件依赖缺失或版本不兼容，你的情况核心问题：

• 旧版ChromeDriver（v2.37）与Selenium4.18.0完全不兼容，Selenium4.x对Driver版本要求严格
• 无头Chrome（v1.0.0-41）过于老旧，无法适配Python3.11和Lambda当前的Amazon Linux 2运行环境

修复方案 #

1. 强制升级ChromeDriver与无头Chrome到兼容版本 #

必须保证ChromeDriver和无头Chrome版本完全匹配，同时适配Lambda环境和Selenium4.18.0。推荐选择Chrome 120系列版本（完美适配Selenium4.18.0和Python3.11）。

2. 修改安装脚本 #

替换原脚本的下载链接，使用适配Amazon Linux 2的版本：

#! /bin/bash

set -e

NOCOLOR='\033[0m'
GREEN='\033[0;32m'

echo -e "\n${GREEN}Installing 'headless' layer dependencies...${NOCOLOR}\n"
mkdir -p layers/headless && chmod 777 layers/headless && cd layers/headless

# 安装ChromeDriver v120.0.6099.109（对应Chrome 120）
echo -e "\n${GREEN}Installing Chrome Driver v120.0.6099.109...${NOCOLOR}\n"
curl -SL https://chromedriver.storage.googleapis.com/120.0.6099.109/chromedriver_linux64.zip > chromedriver.zip
chmod 755 chromedriver.zip
unzip chromedriver.zip
rm chromedriver.zip

# 安装对应版本的无头Chrome（适配Amazon Linux 2）
echo -e "\n${GREEN}Installing Headless Chromium v120.0.6099.109...${NOCOLOR}\n"
curl -SL https://github.com/adieuadieu/serverless-chrome/releases/download/v120.0.6099.109/stable-headless-chromium-amazonlinux-2.zip > headless-chromium.zip
chmod 755 headless-chromium.zip
unzip headless-chromium.zip
rm headless-chromium.zip

cd ../../

注：如果Lambda使用Amazon Linux 2023环境，需替换为对应版本的amazonlinux-2023压缩包

3. 优化Selenium代码（可选） #

针对Selenium4.18.0调整参数，提升稳定性：

import os
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service

def get_headless_driver():
    options = Options()
    service = Service(executable_path='/opt/chromedriver')
    options.binary_location = '/opt/headless-chromium'
    # Selenium4.8+推荐使用--headless=new，性能更优
    options.add_argument('--headless=new')
    options.add_argument('--no-sandbox')
    options.add_argument('--single-process')
    options.add_argument('--disable-dev-shm-usage')
    options.add_argument('--window-size=1920x1080')
    # 移除--start-maximized，与window-size冲突且无头模式无需最大化

    return webdriver.Chrome(service=service, options=options)

def get_selenium_driver():
    return get_local_driver() if os.environ.get('STAGE') == 'local' else get_headless_driver()

版本匹配规则 #

• Selenium4.18.0支持Chrome/ChromeDriver 114~122版本
• 必须严格保证ChromeDriver与无头Chrome版本号完全一致，否则会出现启动失败或通信错误
• Lambda运行环境为Amazon Linux 2则选对应amazonlinux-2包，Amazon Linux 2023选amazonlinux-2023包

内容的提问来源于stack exchange，提问作者taga