Selenium中无Next Page按钮时如何遍历HTML页面完成爬取？

目标爬取链接：https://www.napier.ac.uk/research-and-innovation/research-search?fv=BE7C946393454607AD2CC3D91E65303F%7eBusiness+School&dtFrom=2021-01&dtTo=2022-12&t1sz=100&tab=1&tabpg1=3#rms的Outputs标签内容。

当前能手动修改XPATH中的页码值爬取指定页面，但无法自动迭代所有页面（页面无Next Page按钮），运行现有代码时出现TimeoutException: TimedPromise timed out after 300000 ms错误。

现有实现代码：

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as ec
import time
import pandas as pd

output_tab = wait.until(ec.element_to_be_clickable((By.XPATH, "(//a[@class='r-tabs-anchor'][normalize-space()='Outputs'])[1]")))
output_tab.click()
time.sleep(2)

df = pd.DataFrame({'Titles': [''], 'SubTitle': [''], 'Abstract': [''], 'Intro': [''], 'Links': ['']})

counter = 0
while counter < 4:
    
    driver.refresh()
    post_blocks = driver.find_elements(By.XPATH, "(//div[@class='output bgGrey'])")
    
    for post_block in post_blocks:
        title = post_block.find_element(By.XPATH, "./div/h3").text # (//div[@class='output bgGrey'])/div/h3
        sub_title = post_block.find_element(By.XPATH, "./div[3]").text # (//div[@class='output bgGrey'])/div[3]
        try:
            post_abstract = post_block.find_element(By.XPATH, "./div[4]").text # (//div[@class='output bgGrey'])/div[4]
        except Exception:
            continue
        try:
            post_intro = post_block.find_element(By.XPATH, "./div[5]").text # # (//div[@class='output bgGrey'])/div[5]
        except Exception:
            continue
        post_link = post_block.find_element(By.XPATH, "./parent::a").get_attribute('href') # (//div[@class='output bgGrey'])/parent::a
        
        df = df.append({'Titles': title, 'SubTitle': sub_title, 'Abstract': post_abstract, 'Intro': post_intro, 'Links': post_link}, ignore_index = True)
    
        
    next_page = wait.until(ec.element_to_be_clickable((By.XPATH, "(//a[contains(@class,'')][normalize-space()='3'])[2]"))).click()
   
    counter += 1
    


df.to_csv('C:/Users/testuser/napier_outputs.csv')

核心思路：利用URL参数翻页 #

观察目标链接可知，tabpg1参数控制Outputs标签的页码（如tabpg1=3对应第3页），直接修改该参数加载页面，无需依赖页面元素定位，彻底避免Timeout问题。

优化后代码 #

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as ec
import pandas as pd

# 初始化浏览器和等待对象
driver = webdriver.Chrome()
wait = WebDriverWait(driver, 10)

# 基础URL，用占位符替换页码部分
base_url = "https://www.napier.ac.uk/research-and-innovation/research-search?fv=BE7C946393454607AD2CC3D91E65303F%7eBusiness+School&dtFrom=2021-01&dtTo=2022-12&t1sz=100&tab=1&tabpg1={page_num}#rms"

# 用列表存储数据，替代频繁的df.append操作，提升效率
data_list = []

page_num = 1
while True:
    # 加载当前页码的页面
    driver.get(base_url.format(page_num=page_num))
    
    # 首次加载时切换到Outputs标签，后续页面加载后会保持该标签状态
    if page_num == 1:
        output_tab = wait.until(ec.element_to_be_clickable((By.XPATH, "(//a[@class='r-tabs-anchor'][normalize-space()='Outputs'])[1]")))
        output_tab.click()
        # 等待内容块加载完成
        wait.until(ec.presence_of_element_located((By.XPATH, "(//div[@class='output bgGrey'])")))
    
    # 获取当前页面的所有内容块
    post_blocks = driver.find_elements(By.XPATH, "(//div[@class='output bgGrey'])")
    
    # 没有内容块说明已到最后一页，终止循环
    if not post_blocks:
        break
    
    # 遍历内容块提取数据
    for post_block in post_blocks:
        title = post_block.find_element(By.XPATH, "./div/h3").text
        sub_title = post_block.find_element(By.XPATH, "./div[3]").text
        post_abstract = ""
        try:
            post_abstract = post_block.find_element(By.XPATH, "./div[4]").text
        except Exception:
            pass
        post_intro = ""
        try:
            post_intro = post_block.find_element(By.XPATH, "./div[5]").text
        except Exception:
            pass
        post_link = post_block.find_element(By.XPATH, "./parent::a").get_attribute('href')
        
        # 将数据添加到列表
        data_list.append({
            'Titles': title,
            'SubTitle': sub_title,
            'Abstract': post_abstract,
            'Intro': post_intro,
            'Links': post_link
        })
    
    # 页码递增，进入下一页
    page_num += 1

# 转换为DataFrame并保存
df = pd.DataFrame(data_list)
df.to_csv('C:/Users/testuser/napier_outputs.csv', index=False)

# 关闭浏览器
driver.quit()

关键优化点 #

1. URL参数控制翻页：彻底抛弃依赖页面元素定位页码的方式，直接修改URL参数加载页面，稳定性大幅提升
2. 数据存储优化：用列表批量存储数据，最后一次性转DataFrame，比df.append效率提升数倍
3. 自动终止循环：当页面无内容块时自动停止，无需提前知道总页数
4. 异常处理优化：缺失的Abstract/Intro字段赋值为空字符串，避免跳过整行有效数据

内容的提问来源于stack exchange，提问作者Temidayo Amure