如何解决Selenium抓取Notion表格动态行加载不全的问题？

Notion表格采用懒加载机制，仅当行进入视口才会渲染DOM节点，这是你只能检测到部分行的核心原因。以下是针对该问题的具体修复方案，同时适配400行以上的大表格：

1. 修复总行数获取逻辑 #

放弃一次性获取全部行的方式，改为循环滚动加载直到无法获取新行：

def get_total_rows(driver: webdriver.Chrome) -> int:
    """通过循环滚动加载所有行，返回最终总行数"""
    print("正在加载所有表格行...")
    scroll_container_xpath = "//div[@role='table']/div[last()]"  # Notion表格的滚动容器
    scroll_container = WebDriverWait(driver, 15).until(
        EC.presence_of_element_located((By.XPATH, scroll_container_xpath))
    )
    
    total_rows = 0
    while True:
        # 获取当前已渲染的行（排除隐藏行）
        current_rows = driver.find_elements(By.XPATH, "//div[@role='row' and not(@aria-hidden='true')]")
        current_count = len(current_rows)
        
        if current_count == total_rows:
            # 行数不再增加，说明已加载全部
            break
        
        total_rows = current_count
        # 滚动到容器底部触发懒加载
        driver.execute_script("arguments[0].scrollTop = arguments[0].scrollHeight", scroll_container)
        # 等待新行加载完成
        time.sleep(2)
    
    print(f"表格总行数：{total_rows}")
    return total_rows

2. 改进行内容提取逻辑 #

移除硬编码的行号XPATH，改为直接操作行元素，避免索引错位：

from selenium.webdriver.common.action_chains import ActionChains

def extract_content_from_cell(driver: webdriver.Chrome, row_element) -> str:
    """传入行元素，提取对应页面内容"""
    print("处理当前行...")
    
    try:
        # 滚动行到视口中央，确保可交互
        driver.execute_script("arguments[0].scrollIntoView({block: 'center'});", row_element)
        # 悬停行元素触发"Open"按钮显示
        hover = ActionChains(driver).move_to_element(row_element)
        hover.perform()
        print("已悬停在行上")
    except Exception as e:
        print(f"悬停失败：{e}")
        return ""
    
    # 定位并点击当前行的"Open in side peek"按钮
    try:
        open_button = WebDriverWait(driver, 10).until(
            EC.element_to_be_clickable((By.XPATH, ".//div[@aria-label='Open in side peek']"))
        )
        open_button.click()
        print("已点击打开按钮")
    except Exception as e:
        print(f"打开按钮定位失败：{e}")
        return ""
    
    # 等待侧边栏内容加载
    time.sleep(3)
    
    # 提取页面内容并关闭侧边栏
    try:
        content_element = WebDriverWait(driver, 15).until(
            EC.presence_of_element_located((By.CLASS_NAME, "notion-page-content"))
        )
        page_text = content_element.text
        print("已提取页面内容")
        
        # 关闭侧边栏，避免干扰下一行操作
        close_button = driver.find_element(By.XPATH, "//div[@aria-label='Close']")
        close_button.click()
        time.sleep(1)
        
        return page_text
    except Exception as e:
        print(f"内容提取失败：{e}")
        return ""

3. 主流程调用示例 #

def main():
    driver = webdriver.Chrome()
    driver.get("https://web3sec.notion.site/b201fe69f84e4050bf3915c6030f0fdf?v=2149cfc2302342ebb6124022c1b1950c")
    driver.maximize_window()
    time.sleep(5)  # 等待页面初始加载完成
    
    # 自动加载所有行并获取总行数
    total_rows = get_total_rows(driver)
    # 排除表头行，只处理数据行
    rows = driver.find_elements(By.XPATH, "//div[@role='row' and not(@aria-hidden='true')]")[1:]
    
    dataset = []
    for idx, row in enumerate(rows):
        print(f"正在处理第{idx+1}/{total_rows-1}行")
        content = extract_content_from_cell(driver, row)
        if content:
            dataset.append(content)
    
    # 保存数据集到本地文件
    with open("notion_dataset.txt", "w", encoding="utf-8") as f:
        f.write("\n---\n".join(dataset))
    
    driver.quit()
    print("数据提取完成")

if __name__ == "__main__":
    main()

关键优化点 #

• 滚动容器精准定位：直接操作Notion表格的滚动容器，确保滚动触发懒加载
• 循环加载检测：通过对比滚动前后的行数，确认所有行已加载完成
• 行元素直接操作：避免硬编码行号XPATH，彻底解决索引错位问题
• 侧边栏自动关闭：每次提取后关闭侧边栏，消除对后续操作的干扰

内容的提问来源于stack exchange，提问作者meowlicious