Puppeteer集成AWS Lambda突发Protocol error(Page.navigate):Target closed错误

问题背景 #

在AWS Lambda中使用Puppeteer时突然触发"Protocol error (Page.navigate): Target closed"错误，具体情况：

• 美国东部时间12月5日下午12点后生产环境出现该错误，近期无代码变更
• 初始预发布（staging）环境未出现错误，但重新运行最后一次无代码变更的GitHub Action后，预发布环境也出现完全相同的错误

已尝试的操作 #

• 更改Node.js版本
• 更新Chromium版本
• 修改Puppeteer启动参数
• 切换使用puppeteer-core

代码片段 #

import puppeteer from 'puppeteer-extra';
import stealthPlugin from 'puppeteer-extra-plugin-stealth';

const stealth = stealthPlugin();
puppeteer.use(stealth);

(async () => {
    try {
        const browser = await puppeteer.launch({
            headless: true, // 需要可视化窗口时设为false
            args: ['--no-sandbox', '--disable-setuid-sandbox']
        });

        const page = await browser.newPage();

        const userAgent = 'Mozilla/5.0 (Linux; Android 7.0; LGL84VL Build/NRD90U) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.125 Mobile Safari/537.36';
        await page.setUserAgent(userAgent);

        const url = 'https://example.com'; // 替换为目标URL
        await page.goto(url, { waitUntil: 'networkidle2' });

        // 页面上的其他操作...

        await browser.close();
    } catch (error) {
        console.error('Error occurred:', error);
    }
})();

Package.json 配置 #

{
  "name": "scrp",
  "version": "2.4.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "start": "node build/main.js",
    "build": "node node_modules/typescript/bin/tsc",
    "postbuild": "cp -r src/assets build/",
    "dev": "node node_modules/nodemon/bin/nodemon.js --files src/main.ts",
    "install": "node node_modules/rimraf/bin.js node_modules/mongoose/index.d.ts"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "dependencies": {
    "@google-cloud/vision": "^3.0.1",
    "@sentry/node": "^6.19.7",
    "@sentry/tracing": "^6.19.7",
    "@socket.io/redis-adapter": "^7.1.0",
    "@sparticuz/chromium": "^109.0.5",
    "aws-sdk": "^2.1034.0",
    "axios": "^0.21.1",
    "bcryptjs": "^2.4.3",
    "bullmq": "^1.54.3",
    "cors": "^2.8.5",
    "csvtojson": "^2.0.10",
    "dotenv": "^10.0.0",
    "express": "^4.17.1",
    "helmet": "^4.6.0",
    "ioredis": "^4.28.0",
    "jsonwebtoken": "^8.5.1",
    "moment": "^2.29.4",
    "mongoose": "^6.3.1",
    "mongoose-lean-virtuals": "^0.9.0",
    "multer": "^1.4.3",
    "nodemailer": "^6.4.16",
    "object-hash": "^2.2.0",
    "okrabyte": "^0.1.1",
    "pending-xhr-puppeteer": "^2.3.3",
    "phone": "^2.4.22",
    "puppeteer": "19.4",
    "puppeteer-extra": "^3.3.6",
    "puppeteer-extra-plugin-recaptcha": "^3.6.8",
    "puppeteer-extra-plugin-stealth": "^2.11.2",
    "redis": "^4.0.6",
    "reflect-metadata": "^0.1.13",
    "rimraf": "^3.0.2",
    "socket.io": "^4.0.0",
    "socket.io-emitter": "^3.2.0",
    "spinnies": "^0.5.1",
    "winston": "^3.7.2"
  },
  "devDependencies": {
    "@types/aws-lambda": "^8.10.114",
    "@types/aws-sdk": "^2.7.0",
    "@types/bcryptjs": "^2.4.2",
    "@types/cors": "^2.8.8",
    "@types/express": "^4.17.13",
    "@types/ioredis": "^4.28.1",
    "@types/nodemailer": "^6.4.0",
    "@types/object-hash": "^2.2.1",
    "@types/phone": "^2.4.1",
    "@types/rimraf": "^3.0.2",
    "nodemon": "^2.0.12",
    "ts-node": "^10.9.1",
    "typescript": "^4.9.3"
  }
}

可能的原因及排查步骤 #

1. AWS Lambda 运行环境变更 #

• 检查Lambda运行时版本：AWS可能自动更新Node.js小版本（如从16.x到18.x），导致Puppeteer与Chromium兼容性问题
• 核对执行角色权限：确认VPC安全组、NACL是否新增限制，阻止Chromium访问目标URL
• 调整资源配额：临时调高Lambda内存配置（如512MB→1GB），排查是否因资源不足导致Chromium进程被强制终止
• 检查环境变量：确认代理、时区等配置是否意外变更，影响网络请求

2. 依赖包隐式更新 #

• 锁定依赖版本：将package.json中所有依赖的版本号改为固定值（去掉^/~），重新构建测试
• 对比构建日志：查看GitHub Action中依赖安装的版本列表，与之前成功构建的版本做对比
• 排查插件兼容性：puppeteer-extra-plugin-stealth等插件的小版本更新可能引入冲突

3. Chromium 与 Puppeteer 版本不兼容 #

• 当前配置中puppeteer@19.4对应Chromium 108.x，但使用的@sparticuz/chromium@109.0.5版本不匹配，会导致进程通信失败
• 使用puppeteer-core时必须保证Chromium版本与Puppeteer版本严格对应

4. 目标网站反爬策略变更 #

• 目标站点可能在12月5日更新反爬机制，主动断开无头浏览器连接
• 增加日志记录：在page.goto前后添加页面响应码、控制台错误等日志
• 本地/EC2测试：在非Lambda环境运行代码，确认是否能正常访问目标网站

5. Puppeteer 启动参数缺失 #

• AWS Lambda环境下需补充Chromium启动参数：

  '--disable-dev-shm-usage', '--single-process', '--disable-gpu', '--no-zygote', '--disable-dev-tools'
  

• 改用新无头模式：将headless: true改为headless: 'new'（Puppeteer 19.x支持，兼容性更好）

6. 日志排查方向 #

• 查看CloudWatch日志：搜索错误栈，确认错误发生阶段（浏览器启动/页面创建/导航）
• 启用Puppeteer调试日志：添加process.env.DEBUG = 'puppeteer:*'，查看进程通信细节
• 监听页面事件：捕获页面的error和close事件，记录异常原因：

  page.on('error', err => console.error('Page error:', err));
  page.on('close', () => console.log('Page closed unexpectedly'));
  

内容的提问来源于stack exchange，提问作者Gio G