Selenium无法加载完整页面，爬取lobstersnowboards网站遇阻

Hey Luis, let's work through this problem you're facing—getting those available model fields to load in Selenium has definitely stumped me before with dynamic sites, so here are actionable steps to fix it:

1. Wait Explicitly for Dynamic Content (Don’t Rely on time.sleep()) #

Chances are Selenium is trying to grab elements before the page’s JavaScript finishes fetching and rendering the model data. Ditch the unreliable time.sleep() and use explicit waits with expected conditions:

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

# Replace with the actual CSS selector for your model elements (find via browser dev tools)
wait = WebDriverWait(driver, 15)
model_elements = wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, ".model-selector-class")))

This tells Selenium to wait up to 15 seconds for the model elements to exist in the DOM before throwing an error.

2. Check for Hidden API Calls #

Many dynamic sites load data via background API requests instead of rendering it directly in the initial HTML. Here’s how to find them:

• Open your browser’s DevTools (F12), go to the Network tab, then refresh the page.
• Filter for XHR or Fetch requests—look for endpoints that return JSON data containing the model list.
• If you find such an endpoint, skip Selenium entirely and use requests or httpx to call it directly. This is faster, more reliable, and avoids browser-related issues.

3. Simulate User Interaction to Trigger Loading #

Some content only loads when you interact with the page (scroll, click a tab, hover over an element):

• If models are in a dropdown or tab that needs activation:

  # Wait for the trigger button to be clickable, then click it
  trigger_button = wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, ".model-tab-button")))
  trigger_button.click()
  

• If content is lazy-loaded when scrolled into view:

  # Scroll to the section containing models
  model_section = wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, ".model-section")))
  driver.execute_script("arguments[0].scrollIntoView({behavior: 'smooth'});", model_section)
  

4. Bypass Anti-Scraping Measures #

Sites often detect headless browsers. Add these settings to make your Selenium instance look like a real user:

For Chrome: #

from selenium.webdriver.chrome.options import Options

options = Options()
options.add_argument("--headless=new")  # New headless mode mimics regular Chrome
options.add_argument("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36")
options.add_argument("--disable-blink-features=AutomationControlled")
driver = webdriver.Chrome(options=options)

For Firefox: #

from selenium.webdriver.firefox.options import Options

options = Options()
options.add_argument("--headless")
options.set_preference("general.useragent.override", "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:118.0) Gecko/20100101 Firefox/118.0")
driver = webdriver.Firefox(options=options)

5. Check for Hidden Content in Page Source #

Sometimes the model data is already in the HTML but hidden with CSS (e.g., display: none). Run print(driver.page_source) and search for the model text—if it’s there, you can extract it directly from the source without waiting for it to be visible.

内容的提问来源于stack exchange，提问作者Luis Ramon Ramirez Rodriguez