如何用Python Selenium突破页面限制获取完整页面内容

我尝试获取页面https://www.csaregistries.ca/GHG_VR_Listing/CleanProjectProjects的完整内容，但该页面存在内容加载限制——未滚动时仅显示A-C开头的项目名称，滚动到页面底部后又直接跳到L-W开头的内容，中间的项目丢失。我不需要依赖page_source、手动滚动或time.sleep的方案，想知道如何用Selenium突破这类未知的页面限制。

我当前使用的代码 #

未滚动至页面底部的代码 #

url = "https://www.csaregistries.ca/GHG_VR_Listing/CleanProjectProjects"
browser = webdriver.Chrome(ChromeDriverManager().install())
browser.maximize_window()
browser.get(url)
time.sleep(10)
       
content = browser.page_source.encode('utf-8')
file_ = open('result1.html', 'wb')
file_.write(content)
file_.close()
browser.close()

滚动至页面底部的代码 #

url = "https://www.csaregistries.ca/GHG_VR_Listing/CleanProjectProjects"
browser = webdriver.Chrome(ChromeDriverManager().install())
browser.maximize_window()
browser.get(url)
time.sleep(10)

elem = browser.find_element_by_xpath(
    '/html/body/div/div/div/div/div/div/div[2]/footer/div/div/div/div[3]')
actions = ActionChains(browser)
actions.click(elem).perform()
time.sleep(5)
  
content = browser.page_source.encode('utf-8')
file_ = open('result1.html', 'wb')
file_.write(content)
file_.close()
browser.close()

解决方案 #

这类页面通常采用**虚拟滚动（Virtual Scrolling）**机制，仅渲染视口内的内容，滚动时直接替换DOM元素，导致普通滚动跳过中间内容。以下是几种无需依赖page_source、手动粗暴滚动或time.sleep的Selenium解决方案：

方法1：逐步滚动到每个项目元素，触发内容渲染 #

通过定位每个项目行，逐个滚动到元素位置，确保中间内容被依次加载：

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.action_chains import ActionChains
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

url = "https://www.csaregistries.ca/GHG_VR_Listing/CleanProjectProjects"
browser = webdriver.Chrome(ChromeDriverManager().install())
browser.maximize_window()
browser.get(url)

# 等待列表加载完成
WebDriverWait(browser, 10).until(EC.presence_of_element_located((By.CSS_SELECTOR, "tr[role='row']")))

# 获取所有项目行
project_rows = browser.find_elements(By.CSS_SELECTOR, "tr[role='row']")

# 逐个滚动到项目行，确保内容渲染
for row in project_rows:
    ActionChains(browser).move_to_element(row).perform()
    # 显式等待行内容可见，替代time.sleep
    WebDriverWait(browser, 2).until(EC.visibility_of(row))

# 提取所有项目名称
all_project_names = [row.find_element(By.CSS_SELECTOR, "td:nth-child(2)").text for row in project_rows]
print(all_project_names)

browser.close()

方法2：执行JavaScript触发页面内置加载逻辑 #

利用页面的滚动加载机制，通过JS滚动到当前最后一个元素，等待新内容加载，直到获取全部项目：

from selenium import webdriver
from selenium.webdriver.common.by import By
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

url = "https://www.csaregistries.ca/GHG_VR_Listing/CleanProjectProjects"
browser = webdriver.Chrome(ChromeDriverManager().install())
browser.maximize_window()
browser.get(url)

# 获取总项目数（从页面信息栏提取）
info_text = WebDriverWait(browser, 10).until(
    EC.visibility_of_element_located((By.CSS_SELECTOR, ".dataTables_info"))
).text
total_items = int(info_text.split("of ")[1])

current_item_count = len(browser.find_elements(By.CSS_SELECTOR, "tr[role='row']"))

# 循环加载直到所有项目都被渲染
while current_item_count < total_items:
    # 滚动到当前最后一个项目
    last_item = browser.find_elements(By.CSS_SELECTOR, "tr[role='row']")[-1]
    browser.execute_script("arguments[0].scrollIntoView({block: 'end'});", last_item)
    
    # 等待新内容加载完成
    WebDriverWait(browser, 5).until(
        lambda d: len(d.find_elements(By.CSS_SELECTOR, "tr[role='row']")) > current_item_count
    )
    current_item_count = len(browser.find_elements(By.CSS_SELECTOR, "tr[role='row']"))

# 提取完整内容
all_projects = [row.find_element(By.CSS_SELECTOR, "td:nth-child(2)").text for row in browser.find_elements(By.CSS_SELECTOR, "tr[role='row']")]
print(all_projects)

browser.close()

方法3：直接获取页面全局变量中的原始数据 #

很多虚拟滚动页面会将完整数据存储在全局变量中，通过执行JS直接提取，无需渲染所有元素：

from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager

url = "https://www.csaregistries.ca/GHG_VR_Listing/CleanProjectProjects"
browser = webdriver.Chrome(ChromeDriverManager().install())
browser.maximize_window()
browser.get(url)

# 执行JS获取页面存储的原始项目数据
raw_project_data = browser.execute_script("return window.dtCleanProjects.data;")

# 解析数据，提取项目名称（数据格式为二维数组，第二个元素是名称）
all_project_names = [item[1] for item in raw_project_data]
print(all_project_names)

browser.close()

内容的提问来源于stack exchange，提问作者pythong