使用Python爬取ESPN NFL网页：为比赛结果添加日期字段

问题背景 #

使用Python爬取ESPN网站的NFL 2022赛季（常规赛，seasontype=2）第1到18周的比赛比分，当前输出缺少比赛日期，需要将日期字段添加到结果中，最终保存为CSV。目标页面示例：https://www.espn.com/nfl/scoreboard/_/week/17/year/2022/seasontype/2

当前输出 #

Week #, Away Team, Away Score, Home Team, Home Score
Week 17, Cowboys, 27, Titans, 13
Week 17, Cardinals, 19, Falcons, 20
Week 17, Bears, 10, Lions, 41

期望输出 #

Week #, Date, Away Team, Away Score, Home Team, Home Score
Week 17, 12/29/2022, Cowboys, 27, Titans, 13
Week 17, 1/1/2023, Cardinals, 19, Falcons, 20
Week 17, 1/1/2023, Bears, 10, Lions, 41

修改后的代码 #

import bs4
from bs4 import BeautifulSoup
import requests
import pandas as pd
from datetime import datetime

# 生成1-18周的URL列表
url_list = [f"https://www.espn.com/nfl/scoreboard/_/week/{week}/year/2022/seasontype/2" for week in range(1, 19)]

# 初始化存储数据的列表
away_team = []
home_team = []
away_team_score = []
home_team_score = []
week_num = []
game_dates = []

for week_idx, url in enumerate(url_list, start=1):
    response = requests.get(url)
    soup = bs4.BeautifulSoup(response.text, 'lxml')
    print(f"Processing week {week_idx}: {url}")

    # 获取所有日期区块及对应比赛组
    date_sections = soup.find_all('div', class_='Scoreboard')
    for section in date_sections:
        # 提取日期文本并转换为目标格式
        date_text = section.find('div', class_='Scoreboard__Date').text.strip()
        # 根据周数判断年份，第18周部分比赛跨到2023年
        target_year = 2023 if week_idx == 18 and 'Jan' in date_text else 2022
        date_obj = datetime.strptime(f"{date_text} {target_year}", "%a, %b %d %Y")
        # 转换为无前置零的格式，如1/1/2023
        formatted_date = date_obj.strftime("%m/%d/%Y").lstrip('0').replace('/0', '/')

        # 获取该日期下的所有比赛
        games = section.find_all('div', class_='Scoreboard__Column')
        for game in games:
            # 提取客队和主队名称
            teams = game.find_all('div', class_='ScoreCell__TeamName ScoreCell__TeamName--shortDisplayName truncate db')
            if len(teams) != 2:
                continue  # 跳过无效比赛条目
            away_name = teams[0].text.strip()
            home_name = teams[1].text.strip()

            # 提取客队和主队得分
            scores = game.find_all('div', class_='ScoreCell__Score h4 clr-gray-01 fw-heavy tar ScoreCell_Score--scoreboard pl2')
            if len(scores) != 2:
                continue
            away_score = scores[0].text.strip()
            home_score = scores[1].text.strip()

            # 添加到列表
            away_team.append(away_name)
            home_team.append(home_name)
            away_team_score.append(away_score)
            home_team_score.append(home_score)
            week_num.append(f"Week {week_idx}")
            game_dates.append(formatted_date)

# 构建DataFrame并调整列顺序
df = pd.DataFrame({
    'Week #': week_num,
    'Date': game_dates,
    'Away Team': away_team,
    'Away Score': away_team_score,
    'Home Team': home_team,
    'Home Score': home_team_score
})

# 保存为CSV
df.to_csv('nfl_2022_regular_season_results.csv', index=False)
print("数据已保存为nfl_2022_regular_season_results.csv")

关键修改说明 #

• 日期抓取与格式转换：定位页面中Scoreboard__Date类的元素获取日期文本，结合赛季年份转换为无前置零的MM/DD/YYYY格式，同时处理第18周跨年度的特殊情况。
• 按日期分组匹配比赛：先按日期区块拆分页面，确保每场比赛对应正确的日期，避免日期与比赛条目错位。
• 优化遍历逻辑：替换原有的while循环，改为按比赛区块遍历，减少冗余的异常捕获，代码可读性更强。

内容的提问来源于stack exchange，提问作者Bhavyabl