如何用Node.js+Cheerio获取动态渲染的class值？含Cloudflare防护场景

First, let’s break down the two core issues you’re facing:

1. Cheerio doesn’t run JavaScript: The classes like flying-won-team2 and flying-past are added dynamically by client-side JS after the page loads. Cheerio only parses the static HTML sent by the server, so it can’t see these post-load changes.
2. Cloudflare bot detection: The basic request library sends minimal, bot-like HTTP headers, which Cloudflare’s DDoS protection often blocks. You’re probably not even getting the real page content to parse in the first place.

Here’s how to solve both problems:

Solution 1: Use a Headless Browser (Puppeteer) #

A headless browser (like Puppeteer) mimics a real user’s browser: it loads the page, runs all JavaScript, and can bypass basic Cloudflare checks. We’ll use the puppeteer-extra library with a stealth plugin to avoid bot detection.

Step 1: Install Dependencies #

npm install puppeteer puppeteer-extra puppeteer-extra-plugin-stealth

Step 2: Working Code #

const puppeteer = require('puppeteer-extra');
const StealthPlugin = require('puppeteer-extra-plugin-stealth');
const fs = require('fs');

// Add stealth to avoid Cloudflare bot detection
puppeteer.use(StealthPlugin());

(async () => {
  const browser = await puppeteer.launch({
    headless: 'new', // Uses the modern, undetectable headless mode
    args: ['--no-sandbox', '--disable-setuid-sandbox'], // Required for some environments
    userAgent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36' // Mimic a real browser
  });

  try {
    const page = await browser.newPage();
    // Wait for the page to fully load (including JS execution)
    await page.goto('YOUR_TARGET_LINK', { waitUntil: 'networkidle2' });

    // Wait for the element with dynamic classes to appear
    await page.waitForSelector('div.bma-fly.flying.flying-won-team2.flying-past', { timeout: 10000 });

    // Get the element's full class attribute
    const elementClasses = await page.$eval('div.bma-fly.flying.flying-won-team2.flying-past', el => el.className);

    // Save the result to your file
    fs.writeFileSync('4.txt', elementClasses);
    console.log('Successfully saved dynamic classes!');
  } catch (error) {
    console.error('Error:', error.message);
  } finally {
    await browser.close();
  }
})();

Solution 2: Alternative Headless Browser (Playwright) #

If Puppeteer gives you trouble, Playwright is another great option—it supports Chrome, Firefox, and Safari, and has built-in anti-bot features.

Step 1: Install Dependencies #

npm install playwright

Step 2: Working Code #

const { chromium } = require('playwright');
const fs = require('fs');

(async () => {
  const browser = await chromium.launch({
    headless: true,
    args: ['--no-sandbox']
  });

  try {
    const page = await browser.newPage({
      userAgent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36'
    });

    await page.goto('YOUR_TARGET_LINK', { waitUntil: 'networkidle' });
    await page.waitForSelector('div.bma-fly.flying.flying-won-team2.flying-past');

    const elementClasses = await page.$eval('div.bma-fly.flying.flying-won-team2.flying-past', el => el.className);
    fs.writeFileSync('4.txt', elementClasses);

    console.log('Operation successful!');
  } catch (error) {
    console.error('Error:', error);
  } finally {
    await browser.close();
  }
})();

Extra Tips for Cloudflare Bypass #

• Add small delays: If Cloudflare still blocks you, add await page.waitForTimeout(2000) after page load to mimic human browsing speed.
• Use proxies: For repeated requests, rotate IP addresses with a proxy service to avoid being flagged.
• Manual CAPTCHA solving: If Cloudflare shows a CAPTCHA, you may need to run the browser in non-headless mode (headless: false) to solve it manually, or use a CAPTCHA-solving service (note: this may violate Cloudflare’s terms of service).

内容的提问来源于stack exchange，提问作者Roma N