如何使用Selenium抓取Notion页面中无ID的指定表格数据？

一、修复超时问题并抓取第一个表格 #

原代码超时主要是未处理Cookie弹窗遮挡元素，以及未区分多表格的单元格范围。修正后的代码如下：

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.by import By

url = 'https://theiagen.notion.site/Docker-Image-and-Reference-Materials-for-SARS-CoV-2-Genomic-Characterization-98328c61f5cb4f77975f512b55d09108'
options = Options()
options.add_argument("--headless=new")
options.add_argument("--window-size=1920,1080")  # 无头模式下设置窗口大小，避免元素不可见
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=options)
driver.get(url)

try:
    # 处理Cookie弹窗（如果存在）
    cookie_button = WebDriverWait(driver, 10).until(
        EC.element_to_be_clickable((By.XPATH, "//button[contains(@class, 'notion-cookie-banner-accept-button')]"))
    )
    cookie_button.click()
except:
    # 无弹窗则跳过
    pass

# 定位第一个表格容器，再抓取内部单元格
first_table = WebDriverWait(driver, 20).until(
    EC.visibility_of_element_located((By.XPATH, "(//div[contains(@class, 'notion-table')])[1]"))
)
notion_table_cells = [elem.get_attribute("innerText") for elem in first_table.find_elements(By.CLASS_NAME, "notion-table-cell-text")]

# 原有数据处理逻辑
workspace_keys = {'Pangolin Docker Image','Nextclade Docker Image','Nextclade Dataset Tag','VADR Docker Image'}
workspace_val_indices = [i+1 for i, item in enumerate(notion_table_cells) if item in workspace_keys]
ws_update_vals = [notion_table_cells[i].strip('"') for i in workspace_val_indices]

print(ws_update_vals)
finally:
    driver.quit()

关键调整说明 #

• 新增Cookie弹窗处理，避免元素被遮挡导致超时
• 先定位目标表格容器，再在容器内抓取单元格，确保数据仅来自指定表格
• 无头模式下设置窗口大小，避免因视口问题导致元素不可见

二、抓取任意指定表格的方法 #

方法1：按表格索引抓取 #

修改XPath中的索引即可定位第N个表格，比如抓取第2个表格：

target_table = WebDriverWait(driver, 20).until(
    EC.visibility_of_element_located((By.XPATH, "(//div[contains(@class, 'notion-table')])[2]"))
)
table_cells = [elem.get_attribute("innerText") for elem in target_table.find_elements(By.CLASS_NAME, "notion-table-cell-text")]

方法2：按表格标题定位 #

如果表格有明确标题，可通过标题关联表格，比如定位标题为Reference Materials的表格：

# 定位标题元素
table_title = WebDriverWait(driver, 20).until(
    EC.visibility_of_element_located((By.XPATH, "//h3[text()='Reference Materials']"))
)
# 通过标题找到后续的表格容器
target_table = table_title.find_element(By.XPATH, "./following-sibling::div[contains(@class, 'notion-table')]")
table_cells = [elem.get_attribute("innerText") for elem in target_table.find_elements(By.CLASS_NAME, "notion-table-cell-text")]

方法3：按表格内唯一内容定位 #

通过表格内的专属文本反向定位表格，比如找包含GISAID的表格：

# 定位包含目标文本的单元格
target_cell = WebDriverWait(driver, 20).until(
    EC.visibility_of_element_located((By.XPATH, "//div[@class='notion-table-cell-text' and text()='GISAID']"))
)
# 回溯到表格容器
target_table = target_cell.find_element(By.XPATH, "./ancestor::div[contains(@class, 'notion-table')]")
table_cells = [elem.get_attribute("innerText") for elem in target_table.find_elements(By.CLASS_NAME, "notion-table-cell-text")]

内容的提问来源于stack exchange，提问作者Erik D