Selenium Python爬虫报错：无法定位元素（翻页异常）

我用Python编写网页爬虫脚本实现页面跳转，前4页运行正常，但处理第5页（最后一页）时出现如下报错：

Unable to locate element: {"method":"xpath","selector":"//ul[@class="pagination"]/li/span[@onclick="GoToPage(5)"]"}

我的代码如下：

import os
import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys


def main():
    url = "https://offcampus.uwo.ca/listings/"

    os.environ['PATH'] += r"D:/Usuario/Desktop/web_s"
    options = webdriver.ChromeOptions()
    options.add_experimental_option('excludeSwitches', ['enable-logging'])
    driver = webdriver.Chrome(options=options)

    driver.get(url)

    for k in range(1, 6):
        print(f'page # {k}')
        address = driver.find_elements(
            By.XPATH, '//div[@class=\"rental-listing-details\"]/h2//a[not(@class)]')
        price = driver.find_elements(
            By.XPATH, '//div[@class=\"rental-listing-details\"]/h3/strong')
        n_of_bed = driver.find_elements(
            By.XPATH, '//div[@class=\"rental-listing-details\"]/h3')

        for i in range(len(address)):
            try:
                print(
                    f'{i} {address[i].text} {price[i].text} {n_of_bed[i].text}')

            except IndexError:
                pass

        next = driver.find_element(
            By.XPATH, '//ul[@class=\"pagination\"]/li/span[@onclick=\"GoToPage(' + str(k) + ')\"]')
        next.click()

        window_after = driver.window_handles[0]

        driver.switch_to.window(window_after)

        time.sleep(2)

    driver.close()


if __name__ == "__main__":
    mian()

核心问题 #

当爬取到最后一页时，分页控件中不会存在onclick="GoToPage(5)"的元素——当前页面已是第5页，分页栏不会显示指向自身的可点击跳转按钮。你的循环从1到5，当k=5时仍尝试定位该按钮，自然触发找不到元素的报错。另外代码末尾存在笔误：mian()应改为main()。

修正后的代码 #

import os
import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.common.exceptions import NoSuchElementException


def main():
    url = "https://offcampus.uwo.ca/listings/"

    os.environ['PATH'] += r"D:/Usuario/Desktop/web_s"
    options = webdriver.ChromeOptions()
    options.add_experimental_option('excludeSwitches', ['enable-logging'])
    driver = webdriver.Chrome(options=options)

    driver.get(url)

    page_num = 1
    while True:
        print(f'page # {page_num}')
        address = driver.find_elements(
            By.XPATH, '//div[@class="rental-listing-details"]/h2//a[not(@class)]')
        price = driver.find_elements(
            By.XPATH, '//div[@class="rental-listing-details"]/h3/strong')
        n_of_bed = driver.find_elements(
            By.XPATH, '//div[@class="rental-listing-details"]/h3')

        for i in range(len(address)):
            try:
                print(f'{i} {address[i].text} {price[i].text} {n_of_bed[i].text}')
            except IndexError:
                pass

        # 尝试定位下一页按钮，找不到则退出循环（已到最后一页）
        try:
            next_btn = driver.find_element(
                By.XPATH, '//ul[@class="pagination"]/li/span[contains(@onclick, "GoToPage") and not(contains(@class, "active"))]')
            next_btn.click()
            page_num += 1
            time.sleep(2)
        except NoSuchElementException:
            print("已到最后一页，停止爬取")
            break

    driver.close()


if __name__ == "__main__":
    main()

优化说明 #

1. 改用while True循环，通过检测是否存在下一页按钮判断是否停止爬取，避免固定页码的局限性
2. 调整分页元素定位逻辑，找包含跳转事件且非当前激活状态的按钮，适配任意页数的场景
3. 修复了mian()的拼写错误
4. 捕获NoSuchElementException优雅处理最后一页的结束逻辑

额外建议 #

• 替换固定time.sleep()为WebDriverWait显式等待，提升爬虫稳定性和执行效率
• 可进一步优化元素定位方式，比如通过按钮文本（如"Next"）或图标属性定位，降低xpath维护成本

内容的提问来源于stack exchange，提问作者AskMeee12