Chrome扩展：页面完全加载等待（含加载屏）及动态元素抓取问题

Hey there! Let's dive into solving those two Chrome Extension headaches you're dealing with—dynamic page loads and late-appearing elements are such a pain with modern websites, but we've got solid automated fixes to skip the manual button clicks.

The changeInfo.status === "complete" event often fires before loading screens are gone, which leaves your script staring at an empty page. Here are two reliable automated workarounds:

• Watch for the loading screen to vanish
  Most sites use a specific element (like a div with class loading-overlay) as their loader. You can use MutationObserver to track when this element is hidden or removed from the DOM:

  // In your content script
  const loaderSelector = '.loading-overlay'; // Replace with your site's loader selector
  const observer = new MutationObserver((mutations) => {
    const loader = document.querySelector(loaderSelector);
    // Check if loader is gone or hidden
    if (!loader || getComputedStyle(loader).display === 'none') {
      observer.disconnect(); // Stop watching once we're done
      scrapeTargetElements(); // Run your scraping logic here
    }
  });
  
  // Start observing the body for changes to the loader
  observer.observe(document.body, {
    childList: true,
    subtree: true,
    attributes: true,
    attributeFilter: ['style']
  });
  

• Wait for your target content to exist
  If the loader's structure is inconsistent, skip watching it entirely and wait for the elements you actually need to scrape. This is more direct:

  function waitForCriticalElement(selector, callback) {
    const element = document.querySelector(selector);
    if (element) {
      callback(element);
      return;
    }
    // If element doesn't exist yet, watch the DOM for it
    const observer = new MutationObserver((mutations) => {
      const element = document.querySelector(selector);
      if (element) {
        observer.disconnect();
        callback(element);
      }
    });
    observer.observe(document.body, { childList: true, subtree: true });
  }
  
  // Usage: Wait for your core content element, then scrape
  waitForCriticalElement('.product-list', (element) => {
    scrapeTargetElements();
  });
  

Lots of sites load content incrementally (lazy loading, infinite scroll) or use frames, so complete doesn't mean all elements are ready. Here's how to automate this:

• Observe parent containers for new elements
  If your target elements get added to a specific container (like a .search-results div), use MutationObserver to watch that container for new child nodes:

  const targetContainer = document.querySelector('.search-results');
  if (targetContainer) {
    const observer = new MutationObserver((mutations) => {
      mutations.forEach((mutation) => {
        // Check if any new elements were added
        const newTargetElements = Array.from(mutation.addedNodes)
          .filter(node => node.nodeType === Node.ELEMENT_NODE && node.matches('.target-item'));
        
        if (newTargetElements.length > 0) {
          processElements(newTargetElements); // Handle the newly loaded elements
        }
      });
    });
  
    // Start watching the container for new children
    observer.observe(targetContainer, { childList: true, subtree: true });
  }
  

• Handle multi-frame scenarios
  If elements are hidden inside iframes, you need to inject your content script into every frame. Update your manifest.json to enable this:

  // manifest.json
  "content_scripts": [
    {
      "matches": ["<all_urls>"], // Adjust to your target sites
      "js": ["content.js"],
      "all_frames": true, // Inject script into every frame on the page
      "run_at": "document_idle"
    }
  ]
  

  In your content script, you can check if the current frame contains your target elements, or use chrome.runtime.sendMessage to send data back to the main frame if needed.

• Smart polling (fallback option)
  If MutationObserver isn't feasible (e.g., the site's DOM changes are unpredictable), use a debounced poll to check for elements. It's less efficient but works as a backup:

  let pollTimer;
  function pollForTargetElements() {
    const elements = document.querySelectorAll('.target-item');
    if (elements.length > 0) {
      clearTimeout(pollTimer);
      scrapeTargetElements(elements);
      // Optional: Stop polling once you've collected all needed data
    } else {
      pollTimer = setTimeout(pollForTargetElements, 500); // Adjust interval as needed
    }
  }
  
  // Start polling after the initial page load
  window.addEventListener('load', pollForTargetElements);
  

Pro Tip
Combine these methods for maximum reliability: first wait for the loading screen to disappear, then start observing the target container for new elements. This way, your script runs fully automatically without any user input.

内容的提问来源于stack exchange，提问作者leafpanda