Python脚本无法搜索指定HTML标签，Chrome浏览器可正常查看

Hey there, I’ve run into similar headless browsing quirks before—let’s break down why that class="iw_component" id="c1417094965155" element isn’t showing up in your script’s page_source, even though it’s visible in desktop Chrome. Here are the most likely fixes to try:

1. Swap Firefox for Chrome Headless (Fix Rendering Differences) #

Firefox’s headless mode can behave differently from desktop Chrome, especially when it comes to dynamic content or responsive layout triggers. Chrome’s modern headless mode tends to match desktop Chrome’s behavior much closer. Give this a shot:

import time
from selenium import webdriver
from selenium.webdriver.chrome.options import Options

chrome_options = Options()
chrome_options.add_argument("--headless=new")  # Use the updated headless mode
chrome_options.add_argument("--window-size=1920,1080")  # Match desktop resolution
chrome_options.add_argument("--user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36")

driver = webdriver.Chrome(options=chrome_options)
driver.get('your-target-url')
time.sleep(5)
print(driver.page_source)
driver.close()

2. Replace Fixed Sleep with Explicit Waits #

time.sleep(5) is unreliable—sometimes the element takes longer to load, or the sleep finishes before the element renders. Use Selenium’s explicit wait to wait directly for the element to exist:

import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from pyvirtualdisplay import Display

display = Display(visible=0, size=(1920,1080))  # Larger resolution avoids responsive layout issues
display.start()
driver = webdriver.Firefox()
driver.get('your-target-url')

# Wait up to 10 seconds for the element to be present in the DOM
try:
    WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.ID, "c1417094965155"))
    )
    print(driver.page_source)
except Exception as e:
    print(f"Element failed to load: {e}")

driver.close()
display.stop()

3. Bump Up pyvirtualdisplay Resolution #

Your current size=(800,800) might trigger the site’s mobile/tablet responsive layout, which could hide or skip loading the iw_component element. Switching to a desktop-sized resolution (like 1920x1080) often resolves this.

4. Enable Firefox Features for Dynamic Content #

Firefox’s headless mode might disable JavaScript or other critical features by default. Try enabling them explicitly:

from selenium.webdriver.firefox.options import Options

firefox_options = Options()
firefox_options.add_argument("--headless")
firefox_options.set_preference("javascript.enabled", True)
firefox_options.add_argument("--user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:118.0) Gecko/20100101 Firefox/118.0")

driver = webdriver.Firefox(options=firefox_options)

Start with the explicit wait and resolution adjustment first—those are the quickest fixes for most cases!

内容的提问来源于stack exchange，提问作者Mary