Netflix Top10网页爬取异常：后续页面数据重复问题求助

问题现象 #

爬取top10.netflix.com过去2年数据时，第2页及之后的临时DataFrame（df）除手动设置的Date列外，其余数据均和首个DataFrame（Top10）完全重复，已确认URL格式正确。

用户提供的原始代码：

n_weeks= 104
types=['films','films-non-english','tv','tv-non-english']


for type in types:
    #First page
    url_base= 'https://top10.netflix.com/'
    week=datetime.date(2023,4,2)
    url= url_base + type + "?week=" + week.strftime('%Y-%m-%d')
    page= requests.get(url, headers=get_user_agent(), proxies=get_free_proxy())
    soup= BeautifulSoup(page.text,'lxml')
    table= soup.find('table', class_='w-full text-sm table-fixed md:text-base')
    Top10 = pd.read_html(table.prettify())[0]
    Top10['Date']=week
    Top10 = Top10.rename(columns={Top10.columns[1]: 'Title'})

    print(f"Completata la settimana {week} di {type}")

    #Other pages
    for i in range(n_weeks):
        week= week - datetime.timedelta(days=7)
        url= url_base + type + "?week=" + week.strftime('%Y-%m-%d')
        page= requests.get(url, headers=get_user_agent(), proxies=get_free_proxy())
        soup= BeautifulSoup(page.text,'lxml')
        table= soup.find('table', class_='w-full text-sm table-fixed md:text-base')

        if table:
            df = pd.read_html(table.prettify())[0]
            df['Date']=week
        else:
            print(f'Errore riscontrato alla settimana {week} di {type}')
            break
        df = df.rename(columns={df.columns[1]: 'Title'})

        Top10= pd.concat([Top10,df])
        print(f"Completata la settimana {week} di {type}")

    Top10.to_csv(f"C:/Users/User_/OneDrive/Desktop/Netflix Green/{type}.csv", index=False)
    print(f"Completato il caricamento di {type}")


print("Caricamento completato")

问题原因 #

核心问题出在代理/请求头的稳定性和请求校验缺失：

• get_free_proxy()可能返回了失效或重复的代理，导致服务器返回缓存页面
• get_user_agent()未正确轮换，被网站反爬机制识别，返回重复内容
• 未校验请求状态码，请求失败时仍解析旧页面内容
• 没有对爬取到的数据做重复校验，导致重复数据被写入

解决方案 #

1. 强制校验请求状态 #

每次请求后先检查响应状态码，确保请求成功：

page = requests.get(url, headers=get_user_agent(), proxies=get_free_proxy())
if page.status_code != 200:
    print(f"请求失败，状态码: {page.status_code}, 日期: {week}, 类型: {type}")
    continue  # 跳过当前周，不中断整个任务

2. 优化代理与User-Agent轮换 #

确保每次请求使用不同的有效标识：

• 给get_user_agent()添加随机User-Agent列表，避免被识别为爬虫
• 给get_free_proxy()增加代理有效性检测，或者暂时禁用不稳定的代理

3. 增加数据重复校验 #

对比当前周和上一周的标题数据，避免重复写入：

# 读取df后添加校验逻辑
last_week_data = Top10[Top10['Date'] == week + datetime.timedelta(days=7)]
if not last_week_data.empty and (df['Title'] == last_week_data['Title']).all():
    print(f"检测到重复数据，跳过 {type} 类型 {week} 周")
    continue

完整修正后的代码 #

import datetime
import requests
import random
from bs4 import BeautifulSoup
import pandas as pd

n_weeks= 104
types=['films','films-non-english','tv','tv-non-english']

def get_user_agent():
    # 随机返回合法User-Agent，避免被反爬
    user_agents = [
        'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36',
        'Mozilla/5.0 (Macintosh; Intel Mac OS X 14_2) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.1 Safari/605.1.15',
        'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:121.0) Gecko/20100101 Firefox/121.0'
    ]
    return {'User-Agent': random.choice(user_agents)}

def get_free_proxy():
    # 建议替换为稳定代理池，这里先使用本地IP避免代理问题
    return {}

for type in types:
    # 首次请求
    url_base= 'https://top10.netflix.com/'
    week=datetime.date(2023,4,2)
    url= url_base + type + "?week=" + week.strftime('%Y-%m-%d')
    page= requests.get(url, headers=get_user_agent(), proxies=get_free_proxy())
    
    if page.status_code != 200:
        print(f"首次请求失败，状态码: {page.status_code}, 日期: {week}, 类型: {type}")
        continue
    
    soup= BeautifulSoup(page.text,'lxml')
    table= soup.find('table', class_='w-full text-sm table-fixed md:text-base')
    if not table:
        print(f"首次请求未找到表格，日期: {week}, 类型: {type}")
        continue
    
    Top10 = pd.read_html(table.prettify())[0]
    Top10['Date']=week
    Top10 = Top10.rename(columns={Top10.columns[1]: 'Title'})
    print(f"完成 {type} 类型 {week} 周数据爬取")

    # 后续周请求
    for i in range(n_weeks):
        week= week - datetime.timedelta(days=7)
        url= url_base + type + "?week=" + week.strftime('%Y-%m-%d')
        page= requests.get(url, headers=get_user_agent(), proxies=get_free_proxy())
        
        # 校验请求状态
        if page.status_code != 200:
            print(f"请求失败，状态码: {page.status_code}, 日期: {week}, 类型: {type}")
            continue
        
        soup= BeautifulSoup(page.text,'lxml')
        table= soup.find('table', class_='w-full text-sm table-fixed md:text-base')

        if table:
            df = pd.read_html(table.prettify())[0]
            df['Date']=week
            df = df.rename(columns={df.columns[1]: 'Title'})
            
            # 校验重复数据
            last_week_data = Top10[Top10['Date'] == week + datetime.timedelta(days=7)]
            if not last_week_data.empty and (df['Title'] == last_week_data['Title']).all():
                print(f"检测到重复数据，跳过 {type} 类型 {week} 周")
                continue
            
            Top10= pd.concat([Top10,df])
            print(f"完成 {type} 类型 {week} 周数据爬取")
        else:
            print(f"未找到表格，日期: {week}, 类型: {type}")
            continue

    Top10.to_csv(f"C:/Users/User_/OneDrive/Desktop/Netflix Green/{type}.csv", index=False)
    print(f"{type} 类型数据已保存到CSV")

print("所有数据爬取完成")

内容的提问来源于stack exchange，提问作者Shattered