如何判断Headless Chrome(Puppeteer)页面未正常打开及异步爬取异常排查

Hey there, let's tackle your Puppeteer issues one by one with practical, actionable solutions:

Here are several reliable ways to spot failed page loads, which you can combine based on your scenario:

• Catch navigation errors directly: The page.goto() method throws errors for issues like DNS failures, connection resets, or server rejections. Wrap it in a try-catch block to detect these cases immediately:

  try {
    await page.goto(targetUrl);
  } catch (error) {
    console.log(`Failed to open page: ${error.message}`);
    // Add your error-handling logic here
  }
  

• Check HTTP response status codes: Grab the navigation response and verify if its status code falls in the normal range (200-399). 4xx/5xx codes mean the server returned an error, so the page likely didn't load correctly:

  const response = await page.goto(targetUrl);
  if (!response || response.status() < 200 || response.status() >= 400) {
    console.log(`Abnormal page response, status code: ${response?.status() || 'No response'}`);
  }
  

• Validate page load completion: Use page.evaluate() to check document.readyState — if it's not complete, the page hasn't fully loaded. Alternatively, use waitForNavigation with a condition like networkidle2 (fewer than 2 active network connections for 500ms) and catch timeouts:

  try {
    await page.goto(targetUrl, { waitUntil: 'networkidle2' });
    const readyState = await page.evaluate(() => document.readyState);
    if (readyState !== 'complete') {
      console.log('Page did not finish loading');
    }
  } catch (error) {
    console.log('Page load timed out or failed');
  }
  

• Check for critical page elements: If your target page has a fixed, essential element (like a main content container or page title), use page.waitForSelector() to wait for it. If it times out, the page didn't load properly:

  try {
    await page.waitForSelector('#main-content', { timeout: 15000 });
  } catch (error) {
    console.log('Critical page element not found — page load failed');
  }
  

Removing global timeouts prevents process termination but leaves you with hanging tasks or empty results. Try these fixes to stabilize your crawler:

• Per-task timeouts + retry logic: Don't ditch timeouts entirely — set a reasonable timeout (30-60 seconds) for each page.goto() and add retry logic for failed attempts. Add small delays between retries to avoid triggering anti-scraping measures:

  async function fetchPage(page, url, retryTimes = 3) {
    for (let i = 0; i < retryTimes; i++) {
      try {
        const response = await page.goto(url, { timeout: 30000, waitUntil: 'networkidle2' });
        if (response && response.status() >= 200 && response.status() < 400) {
          // Page loaded successfully — run your scraping logic
          return await page.evaluate(() => {
            // Your data extraction code here
          });
        }
      } catch (error) {
        console.log(`Attempt ${i+1} failed: ${error.message}`);
        await new Promise(resolve => setTimeout(resolve, 1000 * (i+1))); // Increase delay with each retry
      }
    }
    console.log(`Failed to load page after ${retryTimes} retries: ${url}`);
    return null; // Mark as failed for later processing
  }
  

• Limit concurrent tabs: Running thousands of tabs at once will drain your machine's memory and CPU, leading to random failures. Use a concurrency pool (like the p-limit library) to cap active tasks — start with 8-16 tabs (adjust based on your machine's specs):

  const pLimit = require('p-limit');
  const limit = pLimit(10); // Allow 10 concurrent tasks
  
  const urls = [/* Your list of target URLs */];
  const tasks = urls.map(url => limit(async () => {
    const page = await browser.newPage();
    try {
      return await fetchPage(page, url);
    } finally {
      await page.close(); // Close tab immediately after scraping to free resources
    }
  }));
  
  const results = await Promise.all(tasks);
  

• Monitor and recover from page crashes: Tabs can crash unexpectedly. Listen for error and close events to detect crashes, then re-run the task with a new tab:

  page.on('error', (error) => {
    console.log(`Page crashed: ${error.message}`);
    // Mark this task for re-execution
  });
  
  page.on('close', () => {
    console.log('Page closed unexpectedly');
  });
  

• Mimic real user behavior: Many sites block Headless Chrome. Reduce detection risk with these tweaks:
  • Set a real User-Agent: await page.setUserAgent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36')
  • Disable automation flags: Launch Chrome with --disable-blink-features=AutomationControlled
  • Add random delays: Wait 1-3 seconds between requests to avoid looking like a bot
• Log failed URLs: Save URLs that still fail after retries to a file. You can re-scrape these later (e.g., during off-peak hours) or check if they're invalid.

内容的提问来源于stack exchange，提问作者Sungryeol Park