Selenium选择赛季后无法打印网页表格数据求助

原代码 #

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
#from selenium.webdriver.support.ui import Select
import time


#驱动路径
PATH = "C:\Program Files (x86)\chromedriver.exe"
driver = webdriver.Chrome(PATH) 


driver.get("https://www.adamchoi.co.uk/overs/detailed")
print(driver.title)


all_matches=driver.find_element(By.XPATH,"//label[@analytics-event='All matches']")
all_matches.click()

driver.find_element(By.XPATH,"//select[@id='season']/option[@label='20/21']").click()


matches=driver.find_elements(By.XPATH,'//tr')

for match in matches:
    print(match.text)


#driver.quit()

终端输出 #

DevTools listening on ws://127.0.0.1:61934/devtools/browser/30c3f68e-40f3-4a78-b193-d358ba7bbee2
Overs - Total Goals
PS C:\Users\mrmad\Desktop\selenium test> [9208:2332:0731/115957.877:ERROR:device_event_log_impl.cc(214)] [11:59:57.877] USB: usb_device_handle_win.cc:1048 Failed to read descriptor from node connection: A device attached to the system is not functioning. (0x1F)
[9208:2332:0731/115957.879:ERROR:device_event_log_impl.cc(214)] [11:59:57.878] USB: usb_device_handle_win.cc:1048 Failed to read descriptor from node connection: A device attached to the system is not functioning. (0x1F)

使用环境 #

chrome - 103.0.5060.134
chromewebdriver - 103.0.5060.134
python- 3.10.5
selenium - 4.3.0

问题分析 #

选择20/21赛季后，页面会动态渲染赛事表格数据，原代码直接调用find_elements时，表格数据还未完全加载完成，导致获取到的<tr>元素为空或不完整，因此无法打印出内容。另外，页面中可能存在多个表格，直接用//tr定位会包含无关的行，需要更精确的定位。

解决方案 #

1. 添加显式等待：等待表格行元素加载完成，确保获取到完整的赛事数据
2. 精确元素定位：定位到赛事表格内的<tr>元素，避免抓取无关内容
3. 可选：添加页面稳定等待：确保动态渲染完成

修改后的代码：

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time

# 驱动路径
PATH = "C:\Program Files (x86)\chromedriver.exe"
driver = webdriver.Chrome(PATH) 

driver.get("https://www.adamchoi.co.uk/overs/detailed")
print(driver.title)

# 点击"All matches"
all_matches = driver.find_element(By.XPATH, "//label[@analytics-event='All matches']")
all_matches.click()

# 选择20/21赛季
driver.find_element(By.XPATH, "//select[@id='season']/option[@label='20/21']").click()

# 显式等待：等待赛事表格的行加载完成，最长等待10秒
wait = WebDriverWait(driver, 10)
# 定位到赛事表格内的所有tr元素（更精确的定位）
matches = wait.until(EC.presence_of_all_elements_located((By.XPATH, "//table[contains(@class,'table')]//tr")))

# 打印所有赛事数据
for match in matches:
    if match.text:  # 跳过空文本的行
        print(match.text)

driver.quit()

修改说明 #

• 引入WebDriverWait和expected_conditions实现显式等待，确保表格数据加载完成后再获取元素
• 调整XPATH为//table[contains(@class,'table')]//tr，仅定位赛事表格内的行，避免抓取页面其他无关的<tr>元素
• 添加if match.text判断，跳过无内容的行，输出更整洁
• 最后调用driver.quit()关闭浏览器，释放资源

内容的提问来源于stack exchange，提问作者MAD