Render服务器中Playwright page.goto偶发页面关闭导航失败问题求助

本地运行Playwright网页抓取脚本无任何错误，但部署到Render服务器后，page.goto等操作时常出现报错，报错日志如下：

: GOTO: page.goto: Navigation failed because page was closed!
May 16 06:28:40 PM =========================== logs ===========================
May 16 06:28:40 PM navigating to "https://in.investing.com/equities/housing-development-finance-technical?timeFrame=60", waiting until "load"
May 16 06:28:40 PM ============================================================
May 16 06:28:40 PM at getResults (/usr/src/app/index.js:151:10)
May 16 06:28:40 PM at async scrape (/usr/src/app/index.js:223:23) {
May 16 06:28:40 PM name: 'Error'
May 16 06:28:40 PM }
May 16 06:28:40 PM ERROR => page.$eval: Target page, context or browser has been closed
May 16 06:28:40 PM at getResults (/usr/src/app/index.js:160:33)
May 16 06:28:40 PM at async scrape (/usr/src/app/index.js:223:23) {
May 16 06:28:40 PM name: 'Error'
May 16 06:28:40 PM }
May 16 06:28:40 PM ERROR => browserContext.newPage: Target page, context or browser has been closed
May 16 06:28:40 PM at getResults (/usr/src/app/index.js:133:34)
May 16 06:28:40 PM at async scrape (/usr/src/app/index.js:223:23) {
May 16 06:28:40 PM name: 'Error'
May 16 06:28:40 PM }

相关代码 #

链接列表 #

Link_Lst = [
  "https://in.investing.com/equities/axis-bank-technical",
  "https://in.investing.com/equities/tata-motors-ltd-technical",
  "https://in.investing.com/equities/icici-bank-ltd-technical",
  // ... 更多链接
];

核心抓取函数 #

async function getResults(lnk) {
  const results = [];
  const timeFrames = [1, 5, 15];

  for (const i of timeFrames) {
    try {
      const browser = await chromium.launch({
        args: [
          '--no-sandbox',
          '--disable-setuid-sandbox',
          '--disable-dev-shm-usage',
          '--single-process',
          "--no-zygote",
          '--ignore-certificate-errors',
          "--disable-features=AudioServiceOutOfProcess"
          // "--remote-debugging-port"
        ],   
        defaultViewport: chromium.defaultViewport,
        executablePath: await chromium.executablePath(),
        headless: chromium.headless,
      });
      const context = await browser.newContext({
        userAgent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36'
      });
      
      const page = await context.newPage();

      const url = `${lnk}?timeFrame=${i * 60}`;
      console.log(`Getting: ${url}`);

      await page.evaluate(() => navigator.userAgent);

      await page.setViewportSize({ width: 1920, height: 1080 });

      await page.setDefaultNavigationTimeout(0);
      await page
        .goto(url, { waitUntil: "load", timeout: 0 })
        .then(() => console.log("Goto Success!!"))
        .catch((err) => console.log("GOTO:", err));

      await page
        .waitForSelector("section.forecast-box-graph", {
          visible: true,
        })
        .then(() => console.log("Success"))
        .catch(async (err) => console.log("ERR", err));

      const status = await page.$eval(
        "section.forecast-box-graph .title",
        (el) => el.textContent
      );
      const bank_name = await page.$eval("h1.main-title.js-main-title", (el) =>
        el.textContent.trim()
      );

      results.push({
        bankName: bank_name,
        status: status,
        lnk: lnk.replace(/-/g, " ").split("/").pop(),
        url: url,
      });

      // await page.close();
      await browser.close();
    } catch (e) {
      console.log("ERROR =>", e);
    }
  }
  sendRequest();
  return results;
}

调度函数 #

async function scrape() {
  while (true) {
    for (const lnk of Link_Lst) {
      let Buy_ = true;
      let Sell_ = true;
      let focusCount = 0;

      const results = await getResults(lnk);
    }
  }
}

解决思路 #

• 避免重复创建浏览器实例：当前getResults的timeFrames循环中，每次迭代都新建浏览器实例，对服务器资源消耗极大，Render服务器资源有限时易导致进程被强制关闭。建议将浏览器/上下文实例移到循环外，甚至整个scrape过程复用一个浏览器实例，减少资源开销。
• 严谨处理错误流程：page.goto和waitForSelector用then/catch捕获错误后，后续代码仍会执行（比如page.$eval），此时页面可能已关闭，必然触发报错。需在捕获错误后终止当前迭代，比如在catch中添加continue，避免无效操作。
• 确保资源释放：用try/finally块包裹浏览器操作，无论是否报错都执行browser.close()，避免资源泄漏导致服务器资源耗尽。
• 优化Playwright启动参数：
  • 改用新版无头模式：headless: 'new'，比旧版更稳定、兼容性更好。
  • 移除--single-process参数，该参数强制浏览器单进程运行，稳定性较差，尤其在服务器环境中。
• 调整导航等待策略：将page.goto的waitUntil从"load"改为"domcontentloaded"或"networkidle"，减少不必要的等待时间，避免页面被判定为无响应。
• 限制并发数：scrape是无限循环遍历链接列表，加上getResults内部的循环，易导致并发过高。使用并发控制工具（如p-limit）限制同时运行的抓取任务数，避免服务器资源过载。

内容的提问来源于stack exchange，提问作者hobik