如何从referers.txt随机加载Referer？附Puppeteer现有代码

Got it, let's tweak your Puppeteer script to pull a random Referer from your referers.txt file instead of sticking with that hardcoded Facebook URL. Here's a straightforward, robust way to do this:

Step 1: Add File Reading Logic #

First, we'll use Node.js's built-in fs/promises module to read the file asynchronously (so it plays nice with your existing async Puppeteer code). We'll parse the file into an array of valid Referer URLs, skipping any empty lines.

Step 2: Implement Random Selection #

We'll create a simple helper function to pick a random element from our parsed Referer array.

Full Modified Code #

const puppeteer = require('puppeteer');
const fs = require('fs/promises'); // Use promises for async file reading

// Helper function to get a random element from an array
const getRandomReferer = (referers) => {
  return referers[Math.floor(Math.random() * referers.length)];
};

(async () => {
  try {
    // Read and parse referers.txt
    const fileContent = await fs.readFile('./referers.txt', 'utf8');
    // Split into lines, filter out empty lines and trim whitespace
    const refererList = fileContent.split('\n').map(line => line.trim()).filter(line => line.length > 0);

    if (refererList.length === 0) {
      throw new Error('referers.txt is empty or contains only empty lines');
    }

    // Your existing Puppeteer setup
    const browser = await puppeteer.getBrowserInstance(port);
    const page = await browser.newPage();
    page.setDefaultTimeout(PAGE_DEFAULT_TIMEOUT * 1000);
    page.on('error', handlePageCrash(page));
    page.on('pageerror', handlePageCrash(page));

    // Set random Referer from the list
    const randomReferer = getRandomReferer(refererList);
    page.setExtraHTTPHeaders({ referer: randomReferer });

    // Rest of your script...
  } catch (error) {
    console.error('Error setting up Referer or Puppeteer:', error);
  }
})();

Key Notes: #

• Async File Reading: We use fs/promises.readFile to avoid blocking the event loop, which is critical for Puppeteer's async workflow.
• Cleaning Up the List: The split('\n').map(line => line.trim()).filter(...) chain ensures we only keep valid, non-empty URLs—no accidental blank Referers.
• Error Handling: We added checks for an empty referers.txt file and wrapped everything in a try/catch block to catch any file-reading or Puppeteer setup issues.
• Randomness: The getRandomReferer function uses Math.floor(Math.random() * referers.length) to pick a valid index from the array every time.

Make sure your referers.txt is formatted with one Referer URL per line, like this:

https://www.facebook.com/
https://twitter.com/
https://www.instagram.com/
https://www.linkedin.com/

内容的提问来源于stack exchange，提问作者Tran