如何用Python+Selenium爬取德甲2021/22赛季全比赛日积分榜

我是Python和Selenium的新手，想要爬取德国德甲2021/22赛季所有34个比赛日的积分榜特定列数据。目前能获取单个比赛日的全量表格行数据，但遍历所有比赛日时，每个比赛日仅返回前5行数据，不确定原因，也纠结是否该用XPATH定位元素（或许用class更合适）。预期结果是获取页面第一个表格的所有行数据，共18×34行。

以下是我的尝试代码：

element_list = []
  
for matchday in range(1, 35, 1):
    
    url = 'https://www.kicker.de/bundesliga/tabelle/2021-22/' + str(matchday)
    driver = webdriver.Chrome()
    driver.get(url)
    position = driver.find_elements(By.XPATH, "/html[@class=' kick__nodark']/body/div[@id='kick__page-container']/div[@id='kick__page']/div[@class='kick__area--main']/div[@class='kick__card '][1]/div[@class='kick__data-grid']/div[@class='kick__data-grid__main ']/div[@class='kick__site-padding']/div[@id='1']/div[@class='kick__module-margin']/table[@class='kick__table kick__table--ranking kick__table--alternate kick__table--resptabelle']/tbody/tr/td[@class='kick__table--ranking__rank kick__respt-m-o-1 kick__respt-m-w-25']")
    team = driver.find_elements(By.XPATH, "/html[@class=' kick__nodark']/body/div[@id='kick__page-container']/div[@id='kick__page']/div[@class='kick__area--main']/div[@class='kick__card '][1]/div[@class='kick__data-grid']/div[@class='kick__data-grid__main ']/div[@class='kick__site-padding']/div[@id='1']/div[@class='kick__module-margin']/table[@class='kick__table kick__table--ranking kick__table--alternate kick__table--resptabelle']/tbody/tr/td[@class='kick__table--ranking__teamname kick__table--ranking__index kick__t__a__l kick__respt-m-o-4 kick__respt-m-w-120 kick__t__a__l']/a/span[@class='kick__table--show-desktop']")
    matchday = driver.find_elements(By.XPATH, "/html[@class=' kick__nodark']/body/div[@id='kick__page-container']/div[@id='kick__page']/div[@class='kick__area--main']/div[@class='kick__card '][1]/div[@class='kick__data-grid']/div[@class='kick__data-grid__main ']/div[@class='kick__site-padding']/div[@id='1']/div[@class='kick__module-margin']/table[@class='kick__table kick__table--ranking kick__table--alternate kick__table--resptabelle']/tbody/tr/td[@class='kick__table--ranking__number'][1]")
    goals = driver.find_elements(By.XPATH, "/html[@class=' kick__nodark']/body/div[@id='kick__page-container']/div[@id='kick__page']/div[@class='kick__area--main']/div[@class='kick__card '][1]/div[@class='kick__data-grid']/div[@class='kick__data-grid__main ']/div[@class='kick__site-padding']/div[@id='1']/div[@class='kick__module-margin']/table[@class='kick__table kick__table--ranking kick__table--alternate kick__table--resptabelle']/tbody/tr/td[@class='kick__table--ranking__number'][2]")
    points = driver.find_elements(By.XPATH, "/html[@class=' kick__nodark']/body/div[@id='kick__page-container']/div[@id='kick__page']/div[@class='kick__area--main']/div[@class='kick__card '][1]/div[@class='kick__data-grid']/div[@class='kick__data-grid__main ']/div[@class='kick__site-padding']/div[@id='1']/div[@class='kick__module-margin']/table[@class='kick__table kick__table--ranking kick__table--alternate kick__table--resptabelle']/tbody/tr/td[@class='kick__table--ranking__master kick__respt-m-o-5']")
  
    for i in range(len(team)):
        element_list.append([position[i].text, team[i].text, matchday[i].text, goals[i].text, points[i].text])
  
# closing the driver
driver.close()

问题原因 #

1. 页面加载不完整：循环中每次打开新页面后未等待页面完全渲染，仅获取到已加载的前5行数据。
2. 定位器过于脆弱：使用绝对XPATH定位，页面结构稍有变动就会失效，且容易出现定位偏差。
3. 变量名冲突：循环变量matchday被重新赋值为元素列表，导致后续循环逻辑异常。
4. 资源浪费：每次循环新建Chrome实例，速度慢且可能触发网站反爬机制。

优化后的代码 #

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

element_list = []

# 仅初始化一次浏览器实例
driver = webdriver.Chrome()
wait = WebDriverWait(driver, 10)  # 设置10秒等待超时

for matchday_num in range(1, 35):
    url = f'https://www.kicker.de/bundesliga/tabelle/2021-22/{matchday_num}'
    driver.get(url)
    
    # 等待目标表格加载完成
    table = wait.until(EC.presence_of_element_located((By.CLASS_NAME, 'kick__table--ranking')))
    rows = table.find_element(By.TAG_NAME, 'tbody').find_elements(By.TAG_NAME, 'tr')
    
    for row in rows:
        # 逐行提取目标数据
        position = row.find_element(By.CLASS_NAME, 'kick__table--ranking__rank').text
        team = row.find_element(By.CLASS_NAME, 'kick__table--show-desktop').text
        played = row.find_elements(By.CLASS_NAME, 'kick__table--ranking__number')[0].text
        goals = row.find_elements(By.CLASS_NAME, 'kick__table--ranking__number')[1].text
        points = row.find_element(By.CLASS_NAME, 'kick__table--ranking__master').text
        
        element_list.append([position, team, played, goals, points])

driver.quit()  # 关闭浏览器释放资源

关键改进点 #

• 添加等待机制：用WebDriverWait确保表格完全加载后再提取数据，避免获取不完整内容。
• 简化定位逻辑：改用class名称和相对定位，代码更简洁且不易受页面结构变动影响。
• 避免变量冲突：将循环变量改为matchday_num，防止与元素列表变量重名。
• 复用浏览器实例：仅创建一次Chrome驱动，提升爬取效率并降低反爬风险。
• 逐行解析数据：先获取所有行，再遍历每行提取列数据，避免因列数不一致导致的索引错误。

内容的提问来源于stack exchange，提问作者Sarah