使用Python&Selenium爬取网站后列缺失问题求助

运行代码后，生成的DataFrame只显示了4个目标列中的2个。我原本已经调整逻辑——当//a[@class='outcome']元素缺失时，给home_team、away_team、home_odd、away_odd赋值'N/A'再追加到对应列表，确保所有列表长度一致，但客队列和主队赔率列依旧缺失，相关代码如下：

games = driver.find_elements(By.XPATH,
                         '//div[@class="event-card__wrapper" and not(contains(., "//a[@class=\'outcome\']"))]')

for game in games:
    home_team_element = None
    away_team_element = None
    home_odd_element = None
    away_odd_element = None

    try:
        home_team_element = game.find_element(By.CSS_SELECTOR, '.competitors__wrapper > div:nth-child(1)')
        home_team = home_team_element.text
    except NoSuchElementException:
        home_team = 'N/A'

    try:
        away_team_element = game.find_element(By.CSS_SELECTOR,'.competitors__wrapper > div:nth-child(2)')
        away_team = away_team_element.text
    except NoSuchElementException:
        away_team = 'N/A'

    try:
        home_odd_element = game.find_element(By.CSS_SELECTOR,'.market__list > div:nth-child(1) > div:nth-child(1)')
        home_odd = home_odd_element.text
    except NoSuchElementException:
        home_odd = 'N/A'

    try:
        away_odd_element = game.find_element(By.CSS_SELECTOR, '.market__list > div:nth-child(3) > div:nth-child(1)')
        away_odd = away_odd_element.text
    except NoSuchElementException:
        away_odd = 'N/A'

pm_homebbwin_teams_list.append(home_team)
pm_awaybbwin_teams_list.append(away_team)
pm_homebbwin_odds_list.append(home_odd)
pm_awaybbwin_odds_list.append(away_odd)

pm_dict = {
    'Parimatch BBHome Team (winmkt)': pm_homebbwin_teams_list,
    'Parimatch BBAway Team (winmkt)': pm_awaybbwin_teams_list,
    'Parimatch BBHome Odd (winmkt)': pm_homebbwin_odds_list,
    'Parimatch BBAway Odd (winmkt)': pm_awaybbwin_odds_list
}

df = pd.DataFrame.from_dict(pm_dict)
print(df)

问题排查与修复方案 #

1. 核心错误：append语句缩进错误 #

原代码中，四个列表的append操作写在for循环外面，这意味着无论循环多少次，只有最后一次循环的变量值会被追加到列表里，导致列表长度异常，最终DataFrame无法正确显示所有列。

修复： 将append语句移到for循环内部，确保每个游戏的数据都被追加到对应列表中。

2. 必须初始化空列表 #

在循环开始前，需要先创建空列表，否则会引发NameError（列表未定义）或追加到已有旧数据的列表中，导致长度不一致。

添加初始化代码：

pm_homebbwin_teams_list = []
pm_awaybbwin_teams_list = []
pm_homebbwin_odds_list = []
pm_awaybbwin_odds_list = []

3. 修正HTML转义字符 #

原代码中的"和>是HTML转义字符，需要替换为正常的双引号"和大于号>，否则CSS选择器和XPATH无法正确解析。

4. 验证CSS选择器有效性 #

可以在浏览器控制台中测试选择器是否能正确定位元素：

• 测试客队选择器：document.querySelector(".competitors__wrapper > div:nth-child(2)")
• 测试主队赔率选择器：document.querySelector(".market__list > div:nth-child(1) > div:nth-child(1)")

如果返回null，说明选择器错误，需要根据页面实际HTML结构调整。

修复后的完整代码 #

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.common.exceptions import NoSuchElementException
import pandas as pd

# 初始化空列表
pm_homebbwin_teams_list = []
pm_awaybbwin_teams_list = []
pm_homebbwin_odds_list = []
pm_awaybbwin_odds_list = []

games = driver.find_elements(By.XPATH,
                         '//div[@class="event-card__wrapper" and not(contains(., "//a[@class=\'outcome\']"))]')

for game in games:
    home_team = 'N/A'
    away_team = 'N/A'
    home_odd = 'N/A'
    away_odd = 'N/A'

    try:
        home_team_element = game.find_element(By.CSS_SELECTOR, '.competitors__wrapper > div:nth-child(1)')
        home_team = home_team_element.text
    except NoSuchElementException:
        pass

    try:
        away_team_element = game.find_element(By.CSS_SELECTOR,'.competitors__wrapper > div:nth-child(2)')
        away_team = away_team_element.text
    except NoSuchElementException:
        pass

    try:
        home_odd_element = game.find_element(By.CSS_SELECTOR,'.market__list > div:nth-child(1) > div:nth-child(1)')
        home_odd = home_odd_element.text
    except NoSuchElementException:
        pass

    try:
        away_odd_element = game.find_element(By.CSS_SELECTOR, '.market__list > div:nth-child(3) > div:nth-child(1)')
        away_odd = away_odd_element.text
    except NoSuchElementException:
        pass

    # 移到循环内部，每个游戏数据都追加
    pm_homebbwin_teams_list.append(home_team)
    pm_awaybbwin_teams_list.append(away_team)
    pm_homebbwin_odds_list.append(home_odd)
    pm_awaybbwin_odds_list.append(away_odd)

# 调试：检查所有列表长度是否一致
print("各列表长度：", len(pm_homebbwin_teams_list), len(pm_awaybbwin_teams_list), len(pm_homebbwin_odds_list), len(pm_awaybbwin_odds_list))

pm_dict = {
    'Parimatch BBHome Team (winmkt)': pm_homebbwin_teams_list,
    'Parimatch BBAway Team (winmkt)': pm_awaybbwin_teams_list,
    'Parimatch BBHome Odd (winmkt)': pm_homebbwin_odds_list,
    'Parimatch BBAway Odd (winmkt)': pm_awaybbwin_odds_list
}

df = pd.DataFrame.from_dict(pm_dict)
print(df)

内容的提问来源于stack exchange，提问作者Geoffrey