如何突破限制抓取Subreddit自创建以来的全部帖子？

问题背景 #

需要抓取目标Subreddit（示例为AfricanCichlids）自创建以来的所有帖子，但当前使用PRAW的subreddit.new(limit=None)只能获取最新1000条，无法实现全量抓取。

现有代码 #

import pandas as pd
import time
import praw

# 假设已完成Reddit API认证
reddit = praw.Reddit(client_id='YOUR_CLIENT_ID',
                     client_secret='YOUR_CLIENT_SECRET',
                     user_agent='YOUR_USER_AGENT')

subreddit = reddit.subreddit("AfricanCichlids")

def scrape_subreddit_data(subreddit):
    data = {
        'Title': [],
        'Author': [],
        'Upvotes': [],
        'Downvotes': [],
        'Score': [],
        'Number of Comments': [],
        'Creation Time': [],
        'Is NSFW': [],
        'Is Spoiler': [],
        'Is Locked': [],
        'Is Archived': [],
        'Subreddit Subscribers': [],
        'Subreddit Active Users': [],
        'Subreddit Creation Time': [],
        'Comments': []
    }
    
    for submission in subreddit.new(limit=None): # 仅能获取最新1000条
        data['Title'].append(submission.title)
        data['Author'].append(submission.author.name if submission.author else None)
        data['Upvotes'].append(submission.score)
        data['Downvotes'].append(submission.downs)
        data['Score'].append(submission.ups - submission.downs)
        data['Number of Comments'].append(submission.num_comments)
        data['Creation Time'].append(pd.to_datetime(submission.created_utc, unit='s'))
        data['Is NSFW'].append(submission.over_18)
        data['Is Spoiler'].append(submission.spoiler)
        data['Is Locked'].append(submission.locked)
        data['Is Archived'].append(submission.archived)
        data['Subreddit Subscribers'].append(submission.subreddit.subscribers)
        data['Subreddit Active Users'].append(submission.subreddit.accounts_active)
        data['Subreddit Creation Time'].append(pd.to_datetime(submission.subreddit.created_utc, unit='s'))

        time.sleep(2)
        
        submission.comments.replace_more(limit=None)
        comments = []
        for comment in submission.comments.list():
            comments.append(comment.body)
        data['Comments'].append(comments)
    
    return data

data = scrape_subreddit_data(subreddit)
df = pd.DataFrame(data)

可行解决方案 #

1. 通过分页参数循环抓取历史帖子 #

Reddit API支持通过before参数指定时间节点，获取该节点之前的帖子。通过循环执行以下逻辑，可实现全量抓取：

• 首次请求获取最新一批帖子（最多1000条）
• 记录这批帖子中最早的帖子的created_utc时间戳
• 下一次请求用before参数传入该时间戳，获取更早的帖子
• 重复上述步骤，直到返回的帖子数量为0

修改后的全量抓取代码示例：

def scrape_subreddit_full(subreddit):
    data = {
        'Title': [],
        'Author': [],
        'Upvotes': [],
        'Downvotes': [],
        'Score': [],
        'Number of Comments': [],
        'Creation Time': [],
        'Is NSFW': [],
        'Is Spoiler': [],
        'Is Locked': [],
        'Is Archived': [],
        'Subreddit Subscribers': [],
        'Subreddit Active Users': [],
        'Subreddit Creation Time': [],
        'Comments': []
    }
    
    # 初始化before参数，首次请求获取最新帖子
    before = None
    while True:
        # 获取当前时间节点之前的帖子，每次最多1000条
        submissions = list(subreddit.new(limit=1000, before=before))
        if not submissions:
            break  # 无更多帖子，终止循环
        
        # 更新before为当前批次中最早帖子的时间戳
        before = submissions[-1].created_utc
        
        for submission in submissions:
            data['Title'].append(submission.title)
            data['Author'].append(submission.author.name if submission.author else None)
            data['Upvotes'].append(submission.score)
            data['Downvotes'].append(submission.downs)
            data['Score'].append(submission.ups - submission.downs)
            data['Number of Comments'].append(submission.num_comments)
            data['Creation Time'].append(pd.to_datetime(submission.created_utc, unit='s'))
            data['Is NSFW'].append(submission.over_18)
            data['Is Spoiler'].append(submission.spoiler)
            data['Is Locked'].append(submission.locked)
            data['Is Archived'].append(submission.archived)
            data['Subreddit Subscribers'].append(submission.subreddit.subscribers)
            data['Subreddit Active Users'].append(submission.subreddit.accounts_active)
            data['Subreddit Creation Time'].append(pd.to_datetime(submission.subreddit.created_utc, unit='s'))

            time.sleep(1)  # 控制请求频率，避免触发API限流
            
            # 优化评论加载：限制replace_more的递归深度，减少请求量
            submission.comments.replace_more(limit=5)
            comments = []
            for comment in submission.comments.list():
                comments.append(comment.body)
            data['Comments'].append(comments)
    
    return data

# 执行全量抓取
full_data = scrape_subreddit_full(subreddit)
full_df = pd.DataFrame(full_data)

2. 关键注意事项 #

• API限流控制：Reddit普通用户API限制为每分钟60次请求，PRAW会自动处理部分限流，但仍需合理设置time.sleep时长，避免触发封禁。
• 评论加载优化：replace_more(limit=None)会递归加载所有嵌套评论，可能产生大量额外请求，建议根据需求设置合理的limit值，或仅抓取顶层评论。
• 数据持久化：全量抓取耗时较长，建议每抓取一批就将数据保存到本地文件（如CSV、JSON），防止程序中断导致数据丢失。

内容的提问来源于stack exchange，提问作者VenuBhaskar