如何抓取Subreddit特定时间段（2020-2021）的全部帖子？

你的代码只返回72条帖子，核心原因有两个：

1. subreddit.top("all")默认仅返回最多1000条全站热门帖子，而2020-2021年间的热门帖子本身占比极低；
2. Reddit原生API对单请求的返回数量有严格上限，直接遍历默认生成器无法获取全量历史数据。

以下是两种可行的解决方案：

方案一：改用Pushshift API（推荐） #

Pushshift是专门用于Reddit历史数据检索的工具，不受原生API的数量限制，支持按时间范围精准筛选，能获取远更多的目标帖子。

import requests
import pandas as pd
from datetime import datetime

def fetch_pushshift_posts(subreddit, start_time, end_time):
    url = "https://api.pushshift.io/reddit/search/submission/"
    params = {
        "subreddit": subreddit,
        "after": int(start_time),
        "before": int(end_time),
        "size": 1000,  # 单次请求最大返回量
        "sort_type": "created_utc",
        "sort": "asc"
    }
    all_posts = []
    
    while True:
        response = requests.get(url, params=params)
        if response.status_code != 200:
            break
        
        batch_posts = response.json()["data"]
        if not batch_posts:
            break
        
        all_posts.extend(batch_posts)
        # 更新起始时间，获取下一页数据
        params["after"] = batch_posts[-1]["created_utc"]
    
    return all_posts

# 转换目标时间为UTC时间戳
start_date = datetime(2020, 1, 1).timestamp()
end_date = datetime(2021, 12, 31, 23, 59, 59).timestamp()

# 获取2020-2021年的帖子
posts = fetch_pushshift_posts("SuicideWatch", start_date, end_date)

# 整理为DataFrame
posts_dict = {
    "Title": [post.get("title", "") for post in posts],
    "Post Text": [post.get("selftext", "") for post in posts],
    "ID": [post.get("id", "") for post in posts],
    "Score": [post.get("score", 0) for post in posts],
    "Total Comments": [post.get("num_comments", 0) for post in posts],
    "Post URL": [f"https://reddit.com{post.get('permalink', '')}" for post in posts]
}

top_posts = pd.DataFrame(posts_dict)
print(f"共获取{len(top_posts)}条帖子")

方案二：优化PRAW爬取逻辑 #

如果坚持使用PRAW，可通过遍历多种排序方式、去重的方式扩大获取范围，但仍受限于原生API的1000条单排序上限。

import datetime
import pandas as pd
import praw

# 初始化PRAW客户端（需补充你的认证信息）
reddit_read_only = praw.Reddit(
    client_id="YOUR_CLIENT_ID",
    client_secret="YOUR_CLIENT_SECRET",
    user_agent="YOUR_USER_AGENT"
)

subreddit = reddit_read_only.subreddit("SuicideWatch")

start_date = datetime.datetime(2020, 1, 1).timestamp()
end_date = datetime.datetime(2021, 12, 31, 23, 59, 59).timestamp()

posts_dict = {"Title": [], "Post Text": [], "ID": [], "Score": [], "Total Comments": [], "Post URL": []}
seen_post_ids = set()  # 用于去重

# 遍历多种排序方式，减少遗漏
for sort_method in ["top", "new", "controversial", "rising"]:
    # 获取对应排序下的最多帖子（limit=None表示拉取上限1000条）
    if sort_method == "top":
        posts = subreddit.top(time_filter="all", limit=None)
    else:
        posts = getattr(subreddit, sort_method)(limit=None)
    
    for post in posts:
        post_id = post.id
        if post_id in seen_post_ids:
            continue
        
        post_time = post.created
        if start_date < post_time < end_date:
            seen_post_ids.add(post_id)
            posts_dict["Title"].append(post.title)
            posts_dict["Post Text"].append(post.selftext)
            posts_dict["ID"].append(post_id)
            posts_dict["Score"].append(post.score)
            posts_dict["Total Comments"].append(post.num_comments)
            posts_dict["Post URL"].append(post.url)

top_posts = pd.DataFrame(posts_dict)
print(f"共获取{len(top_posts)}条帖子")

注意事项 #

• 请求频率限制：无论是Pushshift还是Reddit API，都有请求频率限制，避免短时间内大量请求，必要时添加time.sleep(1)间隔；
• 合规性：SuicideWatch内容涉及敏感心理话题，爬取和使用数据需遵守Reddit平台规则及伦理规范。

内容的提问来源于stack exchange，提问作者user21126867