如何用Node.js在下拉框onchange事件触发后抓取网站数据？

Hey there, let's break down why your approach with Cheerio isn't working and how to fix it.

Cheerio is fantastic for parsing static HTML, but it doesn't run JavaScript or simulate a real browser environment. When you try to modify the dropdown's value in Cheerio, you're just changing a property in a static DOM tree—there's no browser to trigger the onchange event, which is what loads the new data you're after.

Here are two solid solutions to get the data you need:

1. Use a Headless Browser (Puppeteer/Playwright) #

These tools mimic real user interactions, so they can actually select the dropdown option, trigger the JS event, and wait for the dynamic data to load. Here's a quick example with Puppeteer:

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch({ headless: 'new' }); // Use the updated headless mode
  const page = await browser.newPage();
  await page.goto('your-target-url-here');

  // Wait for the dropdown to load
  await page.waitForSelector('select#your-dropdown-id');
  // Select your target option (use the option's value or text)
  await page.select('select#your-dropdown-id', 'target-option-value');

  // Wait for the dynamic data to load—replace with your data's selector
  await page.waitForSelector('.dynamic-data-container');

  // Extract the desired data
  const desiredData = await page.evaluate(() => {
    return document.querySelector('.dynamic-data-container').innerText;
  });

  console.log(desiredData);
  await browser.close();
})();

2. Directly Call the Underlying API #

Often, when a dropdown change loads new data, it's making an AJAX request to an API. You can bypass the browser entirely by finding this API and calling it directly:

• Open your browser's DevTools (F12), go to the Network tab, and filter by XHR/fetch.
• Change the dropdown value and watch for the new request that loads the data.
• Copy the request URL, method, and any required parameters.

Then use a library like axios or node-fetch to call the API directly:

const axios = require('axios');

async function getDynamicData() {
  try {
    const response = await axios.get('the-api-url-you-found', {
      params: {
        // Replace with the parameter name and value tied to your dropdown
        dropdownParam: 'target-option-value'
      }
    });
    console.log(response.data);
  } catch (err) {
    console.error('Error fetching data:', err);
  }
}

getDynamicData();

This second approach is usually faster since you're skipping the overhead of loading the entire page.

Cheerio just isn't built for dynamic JS interactions—save it for static HTML parsing, and use one of these methods when you need to handle user-triggered JS events.

内容的提问来源于stack exchange，提问作者Sachin Shah