滚动Opensea页面时如何避免Selenium的过时元素引用错误？

问题背景 #

尝试从Opensea的NFT交易藏品页面获取精确交易日期：

• 初始加载的日期可通过鼠标悬停概略日期（如“2个月前”）从弹窗获取
• 页面滚动加载新内容后，触发stale element reference: element is not attached to the page document错误

原代码 #

import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.action_chains import ActionChains

options = Options()
options.add_experimental_option("detach", True)
options.add_argument("start-maximized")
options.add_experimental_option('excludeSwitches', ['enable-logging'])

driver = webdriver.Chrome(options=options)

driver.get(
    "https://opensea.io/activity/artifex-fvckrender?search[collections][0]=artifex-fvckrender&search[eventTypes][0]=AUCTION_SUCCESSFUL")
driver.implicitly_wait(5)

# Code for scrolling down the page
pre_scroll_height = driver.execute_script('return document.body.scrollHeight;')
run_time, max_run_time = 0, 1
list_of_dates = []

while True:

    # Gets all available items that have loaded on the page.
    list_items = driver.find_elements(By.XPATH, "//div[@role='listitem']")

    # Getting values from parent element by digging down to children elements
    for item in list_items:

        # Getting time stamps
        WebDriverWait(driver, 10).until(
            EC.presence_of_all_elements_located((By.XPATH, ".//a[@class='sc-1f719d57-0 fKAlPV EventTimestamp--link']")))  # Timestamp
        time_stamp = item.find_element(
            By.XPATH, ".//a[@class='sc-1f719d57-0 fKAlPV EventTimestamp--link']")
        ActionChains(driver).move_to_element(time_stamp).perform()
        dates_container = item.find_element(
            By.XPATH, ".//div[@data-tippy-root]")
        dates_parent = item.find_element(
            By.XPATH, ".//div[@class='tippy-content']/div[1]")
        list_of_dates.append(dates_parent.get_attribute("innerHTML"))

    print(list_of_dates)
    iteration_start = time.time()
    # Scroll webpage, the 100 allows for a more 'aggressive' scroll
    driver.execute_script(
        'window.scrollTo(0, 100*document.body.scrollHeight);')

    time.sleep(5)

    post_scroll_height = driver.execute_script(
        'return document.body.scrollHeight;')

    scrolled = post_scroll_height != pre_scroll_height
    timed_out = run_time >= max_run_time

    if scrolled:
        run_time = 0
        pre_scroll_height = post_scroll_height
    elif not scrolled and not timed_out:
        run_time += time.time() - iteration_start
    elif not scrolled and timed_out:
        break

print(list_of_dates)

输出结果 #

['April 26, 2022 at 12:30 PM', 'March 9, 2022 at 11:45 PM', 'January 14, 2022 at 6:18 PM', 'October 18, 2021 at 9:03 PM', 'October 22, 2021 at 11:08 AM', 'March 9, 2022 at 5:10 PM', 'March 14, 2022 at 11:48 AM', 'March 13, 2022 at 4:18 AM', 'March 18, 2022 at 8:05 AM', 'November 29, 2022 at 7:37 AM', 'July 8, 2022 at 8:13 AM', 'December 8, 2022 at 4:07 PM']

报错信息 #

Traceback (most recent call last):
  File "c:\Users\persi\Projects\Python\data-scrape-opensea\main.py", line 37, in <module>
    time_stamp = item.find_element(
                 ^^^^^^^^^^^^^^^^^^
  File "C:\Users\persi\AppData\Local\Programs\Python\Python311\Lib\site-packages\selenium\webdriver\remote\webelement.py", line 433, in find_element
    return self._execute(Command.FIND_CHILD_ELEMENT, {"using": by, "value": value})["value"]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\persi\AppData\Local\Programs\Python\Python311\Lib\site-packages\selenium\webdriver\remote\webelement.py", line 410, in _execute
    return self._parent.execute(command, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\persi\AppData\Local\Programs\Python\Python311\Lib\site-packages\selenium\webdriver\remote\webdriver.py", line 444, in execute
    self.error_handler.check_response(response)
  File "C:\Users\persi\AppData\Local\Programs\Python\Python311\Lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 249, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: element is not attached to the page document
  (Session info: chrome=109.0.5414.75)
Stacktrace:
Backtrace:
        (No symbol) [0x00986643]
        (No symbol) [0x0091BE21]
        (No symbol) [0x0081DA9D]
        (No symbol) [0x008209E4]
        (No symbol) [0x008208AD]
        (No symbol) [0x00820B30]
        (No symbol) [0x00850FAC]
        (No symbol) [0x0085147B]
        (No symbol) [0x008464C1]
        (No symbol) [0x0086FDC4]
        (No symbol) [0x0084641F]
        (No symbol) [0x008700D4]
        (No symbol) [0x00886B09]
        (No symbol) [0x0086FB76]
        (No symbol) [0x008449C1]
        (No symbol) [0x00845E5D]
        GetHandleVerifier [0x00BFA142+2497106]
        GetHandleVerifier [0x00C285D3+2686691]
        GetHandleVerifier [0x00C2BB9C+2700460]
        GetHandleVerifier [0x00A33B10+635936]
        (No symbol) [0x00924A1F]
        (No symbol) [0x0092A418]
        (No symbol) [0x0092A505]
        (No symbol) [0x0093508B]
        BaseThreadInitThunk [0x771B00F9+25]
        RtlGetAppContainerNamedObjectPath [0x77457BBE+286]
        RtlGetAppContainerNamedObjectPath [0x77457B8E+238]

解决方案 #

问题根源 #

滚动页面后，Opensea会动态渲染DOM元素，之前获取的list_items元素引用已失效，导致操作旧元素时触发异常。

修复要点 #

1. 每次循环重新获取元素：避免使用缓存的旧元素引用
2. 记录已处理元素数量：防止重复处理同一批交易项
3. 优化等待逻辑：用显式等待替代固定sleep，提升稳定性

修改后的代码 #

import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.action_chains import ActionChains

options = Options()
options.add_experimental_option("detach", True)
options.add_argument("start-maximized")
options.add_experimental_option('excludeSwitches', ['enable-logging'])

driver = webdriver.Chrome(options=options)
driver.get(
    "https://opensea.io/activity/artifex-fvckrender?search[collections][0]=artifex-fvckrender&search[eventTypes][0]=AUCTION_SUCCESSFUL")

pre_scroll_height = driver.execute_script('return document.body.scrollHeight;')
run_time, max_run_time = 0, 2
list_of_dates = []
processed_count = 0  # 记录已处理的元素数量

while True:
    # 每次循环重新获取所有交易项
    list_items = WebDriverWait(driver, 10).until(
        EC.presence_of_all_elements_located((By.XPATH, "//div[@role='listitem']"))
    )
    
    # 只处理新增的元素
    for item in list_items[processed_count:]:
        try:
            # 等待时间戳元素加载完成
            time_stamp = WebDriverWait(item, 10).until(
                EC.presence_of_element_located((By.XPATH, ".//a[contains(@class, 'EventTimestamp--link')]"))
            )
            # 鼠标悬停显示精确日期
            ActionChains(driver).move_to_element(time_stamp).perform()
            # 等待弹窗加载并获取日期
            dates_parent = WebDriverWait(item, 10).until(
                EC.presence_of_element_located((By.XPATH, ".//div[@class='tippy-content']/div[1]"))
            )
            list_of_dates.append(dates_parent.get_attribute("innerHTML"))
        except Exception as e:
            print(f"处理元素出错: {e}")
            continue
    
    # 更新已处理元素数量
    processed_count = len(list_items)
    
    # 滚动页面加载新内容
    iteration_start = time.time()
    driver.execute_script('window.scrollTo(0, document.body.scrollHeight);')
    
    # 等待页面加载完成（替代固定sleep）
    WebDriverWait(driver, 10).until(
        lambda d: d.execute_script('return document.body.scrollHeight;') > pre_scroll_height or time.time() - iteration_start > 5
    )
    
    post_scroll_height = driver.execute_script('return document.body.scrollHeight;')
    scrolled = post_scroll_height != pre_scroll_height
    timed_out = run_time >= max_run_time
    
    if scrolled:
        run_time = 0
        pre_scroll_height = post_scroll_height
    elif not scrolled and not timed_out:
        run_time += time.time() - iteration_start
    elif not scrolled and timed_out:
        break

print("获取到的交易日期列表:")
for date in list_of_dates:
    print(date)

关键改进说明 #

• 用WebDriverWait替代固定time.sleep()，等待元素加载完成后再操作
• 记录processed_count，每次只处理新增的交易项，避免重复操作
• 每次循环重新获取list_items，确保元素引用有效
• 添加异常捕获，单个元素处理失败不影响整体流程

内容的提问来源于stack exchange，提问作者omno