使用Puppeteer for循环抓取Facebook Marketplace数据无响应求助

Let’s break down why your loop isn’t working and fix this step by step—your code has a few key issues that are keeping it from running as expected:

Key Problems in Your Current Code #

• Wrong Listing Selector: You’re targeting #marketplace-modal-dialog-title—that’s the title inside the popup, not the actual listing links/names on the main Marketplace page. This is likely why your loop isn’t finding multiple elements when you expect it to.
• Mixing Contexts: You tried using page.waitForSelector inside page.evaluate—but page is a Puppeteer object that lives in your Node.js environment, not the browser’s window context. That nested async function is defined but never actually runs.
• No Async Loop Handling: A plain for loop doesn’t wait for each popup’s actions to finish before moving to the next listing. This creates race conditions where multiple popups open at once, breaking your workflow.
• Uninvoked Async Code: The nested async (page) => { ... } block is never called, so your popup-closing logic never executes.

Corrected Working Code #

Here’s a revised version that fixes all these issues, with comments explaining each step:

const scrapeMarketplaceListings = async (page) => {
  // 1. Grab all listing elements from the main Marketplace page
  // Replace this selector with the actual one for listings (inspect the page to find it)
  const listingElements = await page.$$('a[href*="/marketplace/item/"]');
  const scrapedData = [];

  // 2. Use for...of to handle async operations sequentially
  for (const listing of listingElements) {
    try {
      // Click the listing to open the popup
      await listing.click();
      // Wait for the popup to fully load (adjust selector if needed)
      await page.waitForSelector('#marketplace-modal-dialog', { timeout: 10000 });
      // Short wait to ensure all popup content renders
      await page.waitForTimeout(2000);

      // 3. Scrape data from the popup (customize based on your needs)
      const popupData = await page.evaluate(() => {
        const title = document.querySelector('#marketplace-modal-dialog-title')?.textContent.trim();
        const price = document.querySelector('span[aria-label^="Price"]')?.textContent.trim();
        const description = document.querySelector('div.x1iorvi4.x1pi30zi.x1l90r2v.x1swvt13')?.textContent.trim();
        
        return { title, price, description };
      });

      scrapedData.push(popupData);

      // 4. Close the popup (use a reliable selector like aria-label)
      const closeButton = await page.waitForSelector('button[aria-label="Close"]', { timeout: 5000 });
      await closeButton.click();
      // Wait for the popup to disappear before next iteration
      await page.waitForSelector('#marketplace-modal-dialog', { hidden: true, timeout: 5000 });
    } catch (error) {
      console.error(`Failed to process listing: ${error.message}`);
      // Fallback to close popup if it gets stuck
      try {
        const stuckCloseButton = await page.$('button[aria-label="Close"]');
        if (stuckCloseButton) await stuckCloseButton.click();
      } catch (closeError) {
        console.error(`Couldn't close stuck popup: ${closeError.message}`);
      }
      continue;
    }
  }

  return scrapedData;
};

// Call the function before closing the browser
const marketplaceResults = await scrapeMarketplaceListings(page);
console.log('Scraped listings:', marketplaceResults);

Critical Tips for Success #

• Update Selectors: Facebook uses dynamic, auto-generated class names (like those long x193iq5w... strings). Use stable selectors instead—like aria-label attributes, partial hrefs, or parent containers that don’t change.
• Avoid Detection: Facebook will flag aggressive scraping. Add longer waits between actions, and don’t scrape too many listings at once.
• Sequential Execution: The for...of loop ensures each listing is processed fully before moving to the next, eliminating race conditions.
• Error Handling: The try/catch blocks prevent a single failed listing from crashing the entire scrape, and ensure popups are closed even if something goes wrong.

内容的提问来源于stack exchange，提问作者klaurtar1