使用Puppeteer下载PDF遇403错误及超时问题求助

我尝试用Puppeteer把PDF下载到tmp目录，但出现403错误；调整后又出现超时问题。目标链接在浏览器里能正常打开，不需要认证或验证码，试过随机User-Agent、代理、延长超时都没用，PDF大小5-6MB。

代码如下：

const downloadPDFViaProxy = async (url: string, outputPath: string) => {
  console.log('Using Proxy');
  const proxyChain = require('proxy-chain');

  const oldProxyUrl = 'http://username:password@hostname:8000';
  const newProxyUrl = await proxyChain.anonymizeProxy(oldProxyUrl);
  // console.log(newProxyUrl);

  const browser = await puppeteer.launch({
    args: [
      '--no-sandbox',
      '--disable-setuid-sandbox',
      '--disable-infobars',
      '--window-position=0,0',
      '--ignore-certifcate-errors',
      '--ignore-certifcate-errors-spki-list',
    ],
    headless: true,
    // executablePath: '/opt/homebrew/bin/chromium',
    ignoreHTTPSErrors: true,
  });
  var userAgent = require('user-agents');

  const page = await browser.newPage();
  await page.setUserAgent(userAgent.random().toString());

  await page.setRequestInterception(true);
  console.log('reached here');
  // const browser = await puppeteer.launch({
  //   args: [
  //     '--no-sandbox',
  //     '--disable-setuid-sandbox',
  //     '--proxy-server=http://66.29.154.105:3128', // Specify your proxy server here
  //   ],
  // });
  // const page = await browser.newPage();

  try {
    await page.screenshot({ path: 'screenshot_new1.png' });

    await page.goto(url, {
      waitUntil: 'networkidle2',
      timeout: 60000, // 60 seconds
    });
    await page.screenshot({ path: 'screenshot_new2.png', fullPage: true });

    const pdfUrl = await page.evaluate(() => document.location.href);

    // Use Puppeteer built-in download mechanism instead of fetch
    const response = await page.goto(pdfUrl);
    if (!response || !response.ok()) {
      throw new Error(`Failed to download PDF. Status: ${response.status()}`);
    }

    // Save PDF to file system
    const pdfBuffer = await response.buffer();
    fs.writeFileSync(outputPath, pdfBuffer);
    console.log(`PDF downloaded successfully to: ${outputPath}`);
  } catch (error) {
    console.error(`Error: ${error.message}`);
  } finally {
    await browser.close();
  }
};

问题排查与修复步骤 #

1. 代理未实际生效 #

代码生成了newProxyUrl但未配置到浏览器启动参数中，等于白设置了代理。同时修正证书错误参数的拼写（原代码certifcate少了字母a）：

const browser = await puppeteer.launch({
  args: [
    '--no-sandbox',
    '--disable-setuid-sandbox',
    '--disable-infobars',
    '--window-position=0,0',
    '--ignore-certificate-errors',
    '--ignore-certificate-errors-spki-list',
    `--proxy-server=${newProxyUrl}`, // 新增代理配置
  ],
  headless: true,
  ignoreHTTPSErrors: true,
});

2. 请求拦截未处理导致阻塞 #

启用page.setRequestInterception(true)但未添加请求放行逻辑，所有请求会被挂起超时。要么移除该行代码，要么添加放行逻辑：

await page.setRequestInterception(true);
page.on('request', (req) => {
  req.continue(); // 放行所有请求
});

3. 重复调用page.goto触发反爬 #

第一次page.goto(url)已加载PDF页面，第二次page.goto(pdfUrl)属于重复请求，容易被拦截。直接复用第一次请求的响应：

try {
  await page.screenshot({ path: 'screenshot_new1.png' });

  // 直接获取goto的响应，延长超时到120秒适配5-6MB文件
  const response = await page.goto(url, {
    waitUntil: 'networkidle2',
    timeout: 120000,
  });
  
  await page.screenshot({ path: 'screenshot_new2.png', fullPage: true });

  if (!response || !response.ok()) {
    throw new Error(`Failed to download PDF. Status: ${response.status()}`);
  }

  const pdfBuffer = await response.buffer();
  fs.writeFileSync(outputPath, pdfBuffer);
  console.log(`PDF downloaded successfully to: ${outputPath}`);
} catch (error) {
  console.error(`Error: ${error.message}`);
}

4. 补充浏览器请求头模拟真实访问 #

除User-Agent外，添加Accept等头信息匹配浏览器行为：

await page.setExtraHTTPHeaders({
  'Accept': 'application/pdf,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
  'Accept-Language': 'en-US,en;q=0.5',
});

5. 避免headless模式被检测 #

新版Chrome的无头模式易被识别，可改用headless: 'new'（Puppeteer v19+支持）或添加反检测参数：

const browser = await puppeteer.launch({
  args: [
    // 原有参数
    '--start-maximized',
    '--disable-blink-features=AutomationControlled',
    '--disable-dev-shm-usage',
  ],
  headless: 'new',
  ignoreDefaultArgs: ['--enable-automation'], // 移除自动化标识
});

内容的提问来源于stack exchange，提问作者Sayan Patel