Selenium爬虫点击赛事弹窗仅爬取首个项报错求助

问题场景 #

需要爬取https://www.wagertalk.com/odds上每场赛事点击“Open”弹出的弹窗标题（比如“College football”），但爬虫处理完第一场后就报错停止，代码及错误信息如下：

原代码 #

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
#from selenium.webdriver.firefox.options import Options
import time
from datetime import datetime
import pandas as pd

date = datetime.today().strftime('%Y-%m-%d')
url = 'https://www.wagertalk.com/odds'
options = Options()
driver = webdriver.Chrome(options=options)
#driver = webdriver.Firefox(options=options)
# row.find_elements("css selector",'.time-started')
driver.get(url)
driver.maximize_window()
driver.implicitly_wait(60)
for row in driver.find_elements("css selector",'tr[id^="g"]'):
    

    books = row.find_elements("css selector",'.book')
    for b in books:
        print('--- popup ---')

        # open .popupDiv
        b.click()
        
        time.sleep(1)
        # ... scrape table from .popupDiv ...
        tds = driver.find_elements("css selector",'.popupDiv th.header')
        print(tds[0].text)
        
        # close .popupDiv
        driver.find_elements("css selector",'.popupDiv button').click()

driver.quit()

报错信息 #

selenium.common.exceptions.InvalidArgumentException: Message: invalid argument: invalid locator
  (Session info: chrome=108.0.5359.71)
Stacktrace:
0   chromedriver                        0x000000010ad9ff38 chromedriver + 4910904
1   chromedriver                        0x000000010ad1fa03 chromedriver + 4385283
2   chromedriver                        0x000000010a964747 chromedriver + 472903
3   chromedriver                        0x000000010a9a8d5d chromedriver + 752989
4   chromedriver                        0x000000010a9a95a1 chromedriver + 755105
5   chromedriver                        0x000000010a9eced1 chromedriver + 1031889
6   chromedriver                        0x000000010a9cf13d chromedriver + 909629
7   chromedriver                        0x000000010a9ea28e chromedriver + 1020558
8   chromedriver                        0x000000010a9ceee3 chromedriver + 909027
9   chromedriver                        0x000000010a99930c chromedriver + 688908
10  chromedriver                        0x000000010a99a88e chromedriver + 694414
11  chromedriver                        0x000000010ad6d1de chromedriver + 4702686
12  chromedriver                        0x000000010ad71b19 chromedriver + 4721433
13  chromedriver                        0x000000010ad7928e chromedriver + 4752014
14  chromedriver                        0x000000010ad7291a chromedriver + 4725018
15  chromedriver                        0x000000010ad46b02 chromedriver + 4545282
16  chromedriver                        0x000000010ad91888 chromedriver + 4851848
17  chromedriver                        0x000000010ad91a05 chromedriver + 4852229
18  chromedriver                        0x000000010ada7e5f chromedriver + 4943455
19  libsystem_pthread.dylib             0x00007fff203598fc _pthread_start + 224
20  libsystem_pthread.dylib             0x00007fff20355443 thread_start + 15

错误原因 #

1. 元素引用失效：弹窗操作后页面DOM可能更新，最初获取的row和books元素变为过时元素，无法继续操作
2. 定位器调用错误：driver.find_elements("css selector",'.popupDiv button').click()直接对列表调用click()，不符合Selenium语法，必须指定具体元素
3. 等待机制不稳定：仅用time.sleep(1)无法保证弹窗元素完全加载，容易出现元素未找到的情况

修复后的代码 #

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.common.exceptions import StaleElementReferenceException
import time
from datetime import datetime

date = datetime.today().strftime('%Y-%m-%d')
url = 'https://www.wagertalk.com/odds'
options = Options()
driver = webdriver.Chrome(options=options)
driver.get(url)
driver.maximize_window()
wait = WebDriverWait(driver, 20)

# 循环处理所有赛事行，避免元素过时
processed_rows = 0
while True:
    try:
        # 每次循环重新获取行列表，确保元素引用有效
        rows = wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, 'tr[id^="g"]')))
        if processed_rows >= len(rows):
            break
        
        row = rows[processed_rows]
        books = row.find_elements(By.CSS_SELECTOR, '.book')
        
        for b in books:
            print('--- popup ---')
            # 点击打开弹窗
            b.click()
            # 显式等待弹窗标题加载完成
            popup_header = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, '.popupDiv th.header')))
            print(popup_header.text)
            # 等待关闭按钮可点击并关闭弹窗
            close_btn = wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, '.popupDiv button')))
            close_btn.click()
            # 等待弹窗完全关闭
            wait.until(EC.invisibility_of_element_located((By.CSS_SELECTOR, '.popupDiv')))
        
        processed_rows +=1
    except StaleElementReferenceException:
        # 遇到过时元素时重新获取行列表，继续循环
        continue
    except Exception as e:
        print(f"处理出错: {str(e)}")
        break

driver.quit()

关键修复说明 #

• 避免元素过时：每次循环重新获取赛事行列表，或捕获StaleElementReferenceException后重试，确保元素引用始终有效
• 替换固定等待为显式等待：用WebDriverWait等待元素可见/可点击，适配页面加载速度，避免因加载慢导致的元素未找到问题
• 修正按钮点击逻辑：对find_elements返回的列表指定具体元素（第一个关闭按钮）后再调用click()
• 增加异常处理：捕获过时元素异常和通用异常，避免单个行出错导致整个爬虫停止

内容的提问来源于stack exchange，提问作者Jerry