Selenium爬取足球数据时NoSuchElementException错误修复方案

我用以下Python代码爬取https://www.adamchoi.co.uk/overs/detailed的足球赛事数据：

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.support.ui import Select
import pandas as pd
import time

# define the website to scrape and path where the chromedriver is located
website = 'https://www.adamchoi.co.uk/overs/detailed'
path = 'C:/users/Administrator/Downloads/chromedriver-win64/chromedriver.exe'  # write your path here
service = Service(executable_path=path)  # selenium 4
driver = webdriver.Chrome(service=service)  # define 'driver' variable
# open Google Chrome with chromedriver
driver.get(website)

# locate and click on a button
all_matches_button = driver.find_element(by='xpath', value='//label[@analytics-event="All matches"]')
all_matches_button.click()

# select elements in the table
matches = driver.find_elements(by='xpath', value='//tr')

# storage data in lists
date = []
home_team = []
score = []
away_team = []

# looping through the matches list
for match in matches:
    date.append(match.find_element(by='xpath', value='./td[1]').text)
    home = match.find_element(by='xpath', value='./td[2]').text
    home_team.append(home)
    print(home)
    score.append(match.find_element(by='xpath', value='./td[3]').text)
    away_team.append(match.find_element(by='xpath', value='./td[4]').text)
# quit drive we opened at the beginning
driver.quit()

# Create Dataframe in Pandas and export to CSV (Excel)
df = pd.DataFrame({'date': date, 'home_team': home_team, 'score': score, 'away_team': away_team})
df.to_csv('football_data.csv', index=False)
print(df)

运行时出现如下错误：

Traceback (most recent call last):
  文件 "C:\Users\Administrator\PycharmProjects\WebScraping\1.selenium4-adamchoi.py", 第31行, in <module>
    home = match.find_element(by='xpath', value='./td[2]').text
           ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  文件 "C:\Users\Administrator\PycharmProjects\WebScraping\.venv\Lib\site-packages\selenium\webdriver\remote\webelement.py", 第601行, in find_element
    return self._execute(Command.FIND_CHILD_ELEMENT, {"using": by, "value": value})["value"]
           ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  文件 "C:\Users\Administrator\PycharmProjects\WebScraping\.venv\Lib\site-packages\selenium\webdriver\remote\webelement.py", 第572行, in _execute
    return self._parent.execute(command, params)
           ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  文件 "C:\Users\Administrator\PycharmProjects\WebScraping\.venv\Lib\site-packages\selenium\webdriver\remote\webdriver.py", 第458行, in execute
    self.error_handler.check_response(response)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
  文件 "C:\Users\Administrator\PycharmProjects\WebScraping\.venv\Lib\site-packages\selenium\webdriver\remote\errorhandler.py", 第233行, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: 消息: 找不到元素: 无法定位元素: {"method":"xpath","selector":"./td[2]"}
  (会话信息: chrome=142.0.7444.60)

进程结束，退出代码1

错误原因 #

• 使用//tr获取了页面所有<tr>元素，但其中包含表头行或无<td>的空行，这些行不存在./td[2]子元素，导致报错。
• 点击"All matches"按钮后，页面未完全加载就开始获取元素，也可能触发该问题。

修改后的代码 #

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
import pandas as pd

# 定义目标网站和chromedriver路径
website = 'https://www.adamchoi.co.uk/overs/detailed'
path = 'C:/users/Administrator/Downloads/chromedriver-win64/chromedriver.exe'  # 替换为你的实际路径
service = Service(executable_path=path)
driver = webdriver.Chrome(service=service)
driver.get(website)

# 等待按钮加载完成并点击
all_matches_button = WebDriverWait(driver, 10).until(
    EC.element_to_be_clickable((By.XPATH, '//label[@analytics-event="All matches"]'))
)
all_matches_button.click()

# 等待表格加载，仅获取包含赛事数据的行（排除表头和空行）
matches = WebDriverWait(driver, 10).until(
    EC.presence_of_all_elements_located((By.XPATH, '//table//tr[td]'))
)

# 存储数据的列表
date = []
home_team = []
score = []
away_team = []

# 遍历赛事行，捕获异常跳过无效行
for match in matches:
    try:
        date.append(match.find_element(By.XPATH, './td[1]').text)
        home = match.find_element(By.XPATH, './td[2]').text
        home_team.append(home)
        print(home)
        score.append(match.find_element(By.XPATH, './td[3]').text)
        away_team.append(match.find_element(By.XPATH, './td[4]').text)
    except Exception:
        continue

# 关闭浏览器
driver.quit()

# 生成CSV文件
df = pd.DataFrame({'date': date, 'home_team': home_team, 'score': score, 'away_team': away_team})
df.to_csv('football_data.csv', index=False)
print(df)

修改说明 #

• 添加显式等待：用WebDriverWait确保按钮和表格完全加载后再操作，避免页面未就绪导致的元素定位失败。
• 精准定位有效行：通过//table//tr[td]筛选出包含<td>的行，直接排除表头和空行。
• 异常捕获机制：遍历行时添加try-except，遇到结构异常的行直接跳过，保证程序正常执行。

内容的提问来源于stack exchange，提问作者user20308305