使用Selenium与Python滚动爬取VMH站点表格全量内容求助

目标站点的表格使用了ExtJS虚拟化渲染组件，仅当前可视范围内的行会被加载到DOM中，且表格自身是独立滚动容器，滚动整个页面无法触发表格行加载，因此会出现缺数问题。

核心逻辑为：针对表格独立滚动容器执行滚动操作，待单页所有行渲染完成后再提取数据，再执行翻页操作。

from selenium import webdriver
from bs4 import BeautifulSoup
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.support.ui import WebDriverWait
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
import pandas as pd
import time

# 初始化驱动
driver = webdriver.Chrome(ChromeDriverManager().install())
url = 'https://www.vmh.life/#downloadview'
driver.get(url)

# 等待微生物tab可点击后切换
WebDriverWait(driver, 15).until(EC.element_to_be_clickable((By.ID, "tab-1281-btnEl"))).click()
# 等待表格加载完成，定位表格滚动容器
scroll_container = WebDriverWait(driver, 15).until(
    EC.presence_of_element_located((By.CSS_SELECTOR, "#gridview-1261-body"))
)

organism = []
reconstruction = []
genes = []
reactions = []
metabolites = []
page_total = 17 # 固定爬17页可直接设置终止条件

for page in range(page_total):
    # 滚动表格到底部，触发所有行渲染
    last_height = driver.execute_script("return arguments[0].scrollHeight", scroll_container)
    while True:
        driver.execute_script("arguments[0].scrollTop = arguments[0].scrollHeight", scroll_container)
        time.sleep(0.8) # 可适配网络情况调整等待时长
        new_height = driver.execute_script("return arguments[0].scrollHeight", scroll_container)
        if new_height == last_height:
            break
        last_height = new_height
    
    # 提取当前页所有数据
    soup = BeautifulSoup(driver.page_source, 'lxml')
    for child in soup.find_all('td',attrs={'data-columnid':'gridcolumn-1251'}):
        organism.append(child.text)
    for child in soup.find_all('td',attrs={'data-columnid':'gridcolumn-1252'}):
        reconstruction.append(child.text)
    for child in soup.find_all('td',attrs={'data-columnid':'gridcolumn-1253'}):
        genes.append(child.text)
    for child in soup.find_all('td',attrs={'data-columnid':'gridcolumn-1254'}):
        reactions.append(child.text)
    for child in soup.find_all('td',attrs={'data-columnid':'gridcolumn-1255'}):
        metabolites.append(child.text)
    
    # 最后一页不需要点击下一页
    if page == page_total - 1:
        break
    
    # 点击下一页，等待表格重置完成
    WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.ID, "button-1266-btnIconEl"))).click()
    # 等待表格回到顶部、内容刷新
    WebDriverWait(driver, 10).until(
        lambda d: d.execute_script("return arguments[0].scrollTop", scroll_container) == 0
    )
    time.sleep(1)

# 导出数据
df = pd.DataFrame(
    {'Organism': organism,
     'Reconstruction': reconstruction,
     'Genes': genes,
     'Reactions': reactions,
     'Metabolites': metabolites
    })
print(df)
# 可自行添加 df.to_csv("microbe_data.csv", index=False, encoding='utf-8-sig') 保存到本地
driver.quit()

• 如果后续站点更新导致元素ID变更，可通过F12开发者工具重新获取对应元素的选择器替换即可
• 若出现丢数情况可适当调大滚动后的等待时长，确保行渲染完成后再提取数据

内容的提问来源于stack exchange，提问作者coderowing