Selenium Python中get()无法获取完整HTML，仅抓取前21个<li>元素问题

问题根源 #

目标页面采用滚动加载机制：初始仅渲染视口内的内容，剩余<li>元素需要页面滚动到对应位置才会动态加载；同时headless模式默认窗口尺寸过小，进一步限制了初始加载的内容数量。

修复方案 #

1. 调整headless浏览器窗口尺寸：设置更大的窗口分辨率，让初始加载更多内容
2. 模拟页面滚动加载：通过JavaScript执行滚动操作，循环滚动至页面底部，触发所有动态内容加载
3. 显式等待元素加载完成：确保所有<li>元素加载完毕后再进行抓取

修改后的代码 #

# Import necessary modules from Selenium
import selenium.webdriver.common.by
from selenium import webdriver, common
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

# Set up headless browser with larger window size
options = webdriver.ChromeOptions()
options.add_argument("--headless")
options.add_argument("--window-size=1920,1080")  # 设置桌面级窗口尺寸

# Initialize Chrome browser
browser = webdriver.Chrome(options=options)
browser.get("https://dex.pokemonshowdown.com/pokemon/")

# Wait for the chart element to load
chart = WebDriverWait(browser, 10).until(
    EC.presence_of_element_located((selenium.webdriver.common.by.By.CLASS_NAME, 'utilichart'))
)

# Simulate scrolling to load all content
last_height = browser.execute_script("return document.body.scrollHeight")
while True:
    # Scroll to bottom of page
    browser.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    # Wait for new content to load
    browser.implicitly_wait(2)
    # Calculate new scroll height and compare with last height
    new_height = browser.execute_script("return document.body.scrollHeight")
    if new_height == last_height:
        break
    last_height = new_height

# Get all li elements after full load
rows = chart.find_elements(by=selenium.webdriver.common.by.By.TAG_NAME, value='li')

# Extract Pokémon names
for row in rows:
    try:
        poke = row.find_element(by=selenium.webdriver.common.by.By.TAG_NAME, value='a')
    except selenium.common.exceptions.NoSuchElementException:
        continue
    poke_name = poke.get_attribute('data-entry')[8:]
    print(poke_name)

# Close the browser
browser.quit()

额外说明 #

• --window-size参数确保headless模式下浏览器有足够的视口空间，避免初始加载内容过少
• 循环滚动通过对比滚动前后的页面高度，判断是否已加载完所有内容
• 显式等待WebDriverWait确保图表元素完全加载后再进行后续操作，比隐式等待更可靠

内容的提问来源于stack exchange，提问作者gorkius_pD