如何爬取动态值？能否用观察者模式监测动态更新数据？

Great question—this is such a common headache when dealing with dynamic web content that doesn’t shout its update mechanism from the rooftops. Let’s break down your options, including how to implement an observer-style approach to avoid wasteful polling.

First: Diagnose the Data Update Mechanism #

Before jumping into solutions, let’s figure out why you’re not seeing network requests change even as data updates. There are two likely scenarios:

1. Server-side push: The site uses WebSockets or Server-Sent Events (SSE) to stream updates directly to the browser, so no new "fetch" requests show up in the Network tab.
2. Client-side DOM updates: The page loaded all necessary data upfront, and a frontend script is periodically updating the DOM without making new network calls.

1. Leverage Server Push (The Most Efficient Option) #

Start by checking for WebSocket/SSE connections in Chrome DevTools:

• Go to the Network tab, then filter by WS (for WebSocket) or look for EventStream entries under the Other category.
• If you find either, you can bypass polling entirely by connecting directly to that stream:
  • In Python, use the websocket-client library to listen to WebSocket messages in real time.
  • For SSE, send a persistent GET request to the EventStream endpoint and process incoming events as they arrive.
    This is the gold standard—you only get data when the server has something new to send, no wasted requests or resources.

2. Use MutationObserver (Observer Pattern for DOM Changes) #

If there’s no server push, MutationObserver (a browser-native API) is exactly the observer-style solution you’re asking for. It lets you watch for changes to specific DOM elements and trigger actions only when updates happen.

Here’s how to implement this in a browser automation tool like Selenium or Playwright:

// Replace with your target element's selector (e.g., "#live-stats-panel")
const targetElement = document.querySelector('#your-target-selector');

// Create the observer instance
const dataObserver = new MutationObserver((mutations) => {
  mutations.forEach(mutation => {
    // Check if the change affects the data we care about
    if (mutation.type === 'childList' || mutation.type === 'characterData') {
      const newData = targetElement.textContent.trim();
      console.log('Data updated:', newData);
      
      // If using Selenium/Playwright, pass this data back to your script
      // Example for Selenium: window.lastUpdatedData = newData;
    }
  });
});

// Configure the observer to watch for relevant changes
dataObserver.observe(targetElement, {
  childList: true,          // Watch for added/removed child nodes
  subtree: true,            // Include nested elements in the watch
  characterData: true       // Watch for text content changes
});

Run this script via your automation tool, then you can wait for updates instead of polling on a fixed schedule. When you’re done, don’t forget to call dataObserver.disconnect() to clean up resources.

Pro Tips for MutationObserver:

• Use the most specific selector possible to avoid triggering on irrelevant DOM changes (e.g., a single data cell instead of the entire page).
• If the data lives inside an iframe, switch to the iframe’s context before setting up the observer.

3. Optimize Polling (If All Else Fails) #

If you can’t use push or MutationObserver, you can still make polling less wasteful:

• Dynamic intervals: Start with a short interval (e.g., 2s) when you first load the page, then lengthen it to 10s+ after detecting an update. If no updates happen for a while, gradually shorten the interval again.
• Hash comparison: Instead of parsing the entire data set every time, compute a hash (like SHA-1) of the target element’s content. Only process the data if the hash differs from the previous one.

内容的提问来源于stack exchange，提问作者King in the world.