改进Python爬虫获取247sports全年度大学橄榄球招募排名全量数据

现有脚本问题排查 #

• 存在未定义变量athlete，错误覆盖了team字段的取值
• 多余的嵌套while continue_loop逻辑完全无必要，还会干扰正常分页流程
• 对「Load More」按钮的判断逻辑错误，该按钮不属于rankings-page__list-item列表项，判断永远不会触发
• 缺少星级字段的正确选择器配置
• 没有请求延迟，容易被站点反爬拦截

需求对应实现方案 #

1. 抓取5/4/3星级招募人数 #

正确的字段选择器如下，增加异常兼容适配早年无星级统计的页面：

try:
    five_stars = tag.find('div', class_='rankings-page__star-count--five').find('span').text.strip()
except:
    five_stars = '0'
try:
    four_stars = tag.find('div', class_='rankings-page__star-count--four').find('span').text.strip()
except:
    four_stars = '0'
try:
    three_stars = tag.find('div', class_='rankings-page__star-count--three').find('span').text.strip()
except:
    three_stars = '0'

2. 突破前50条分页限制 #

247sports的排名页支持Page查询参数直接分页，每页返回50条数据，直接循环递增页码，直到返回的列表项为空即可停止，不需要模拟点击Load More按钮。

3. 批量抓取2000-2024年数据 #

直接遍历年份区间构造请求URL即可，每页请求后加1-2秒延迟，避免触发反爬规则。

修正后完整代码 #

import datetime
import time
import requests
from bs4 import BeautifulSoup
import pandas as pd

print(datetime.datetime.now().time())

# 配置需要抓取的年份范围
years = [str(y) for y in range(2000, 2025)]
rows = []
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
}

for year in years:
    url = f'https://247sports.com/Season/{year}-Football/CompositeTeamRankings/'
    print(f"正在抓取{year}年数据，起始URL：{url}")
    page = 1
    
    while True:
        payload = {'Page': page}
        try:
            response = requests.get(url, headers=headers, params=payload, timeout=10)
            response.raise_for_status()
        except Exception as e:
            print(f"请求出错：{e}，等待5秒后重试")
            time.sleep(5)
            continue
            
        soup = BeautifulSoup(response.text, 'html.parser')
        tags = soup.find_all('li', class_='rankings-page__list-item')
        
        # 没有返回数据说明当前年份已经爬完
        if len(tags) == 0:
            print(f"{year}年数据抓取完成，共{page-1}页")
            break
        
        for tag in tags:
            try:
                rank = tag.find('div', class_='primary').text.strip()
                team = tag.find('div', class_='team').find('a').text.strip()
                total_recruits = tag.find('div', class_='total').find('a').text.split(' ')[0].strip()
                avg_rating = tag.find('div', class_='avg').text.strip()
                total_rating = tag.find('div', class_='points').text.strip()
                
                # 抓取星级数据
                try:
                    five_stars = tag.find('div', class_='rankings-page__star-count--five').find('span').text.strip()
                except:
                    five_stars = '0'
                try:
                    four_stars = tag.find('div', class_='rankings-page__star-count--four').find('span').text.strip()
                except:
                    four_stars = '0'
                try:
                    three_stars = tag.find('div', class_='rankings-page__star-count--three').find('span').text.strip()
                except:
                    three_stars = '0'
                
                row = {
                    'Rank': rank,
                    'Team': team,
                    'Total Recruits': total_recruits,
                    'Five-Star Recruits': five_stars,
                    'Four-Star Recruits': four_stars,
                    'Three-Star Recruits': three_stars,
                    'Average Rating': avg_rating,
                    'Total Rating': total_rating,
                    'Year': year
                }
                print(row)
                rows.append(row)
            except Exception as e:
                print(f"解析行数据出错：{e}，跳过当前行")
                continue
        
        # 每页请求后加延迟，避免被封
        time.sleep(1.5)
        page += 1

# 导出结果
recruits_df = pd.DataFrame(rows)
recruits_df.to_csv('247sports_recruits_rank_2000-2024.csv', index=False, encoding='utf-8-sig')
print(f"全部数据抓取完成，共{len(rows)}条记录，已导出到csv文件")
print(datetime.datetime.now().time())

内容的提问来源于stack exchange，提问作者BLuta