如何从Reddit提取帖子与评论正文并生成CSV？工具使用遇阻求助

一、修复Pushshift无返回问题 #

Pushshift无返回通常是时间戳格式错误或未处理分页/限流导致的，以下是最简可运行的实现：

• 必须使用Unix时间戳（可通过datetime.timestamp()生成），不能用字符串日期
• 加入超时设置和分页参数，避免请求被拦截
• 只提取帖子/评论的纯正文，丢弃所有元数据

import requests
import csv
from datetime import datetime

# 配置参数
SUBREDDIT = "your_subreddit"
START_DATE = datetime(2023, 1, 1)
END_DATE = datetime(2023, 12, 31)
START_TIMESTAMP = int(START_DATE.timestamp())
END_TIMESTAMP = int(END_DATE.timestamp())

def fetch_pushshift_data(endpoint, params):
    response = requests.get(f"https://api.pushshift.io/reddit/{endpoint}/search", params=params, timeout=10)
    if response.status_code == 200:
        return response.json().get("data", [])
    return []

# 获取帖子
posts = []
after = START_TIMESTAMP
while after < END_TIMESTAMP:
    post_params = {
        "subreddit": SUBREDDIT,
        "after": after,
        "before": END_TIMESTAMP,
        "limit": 100,
        "fields": "selftext"  # 只取正文
    }
    batch = fetch_pushshift_data("submission", post_params)
    if not batch:
        break
    posts.extend([p["selftext"].replace("\n", " ") for p in batch if p["selftext"].strip()])
    after = batch[-1]["created_utc"]

# 获取评论
comments = []
after = START_TIMESTAMP
while after < END_TIMESTAMP:
    comment_params = {
        "subreddit": SUBREDDIT,
        "after": after,
        "before": END_TIMESTAMP,
        "limit": 100,
        "fields": "body"  # 只取正文
    }
    batch = fetch_pushshift_data("comment", comment_params)
    if not batch:
        break
    comments.extend([c["body"].replace("\n", " ") for c in batch if c["body"].strip()])

# 导出到CSV
with open("reddit_texts.csv", "w", encoding="utf-8", newline="") as f:
    writer = csv.writer(f)
    writer.writerow(["text"])  # 表头适配AntConc
    for text in posts + comments:
        writer.writerow([text])

二、修复PRAW时间戳卡顿问题 #

PRAW卡顿是因为未分页或未处理速率限制，以下是最简修复代码：

• 直接用datetime对象作为before/after参数，无需手动转Unix时间戳（PRAW会自动处理）
• 使用replace_more()加载所有嵌套评论，避免遗漏
• 只提取纯正文，丢弃元数据

import praw
import csv
from datetime import datetime

# 初始化PRAW（替换为你的API密钥）
reddit = praw.Reddit(
    client_id="your_client_id",
    client_secret="your_client_secret",
    user_agent="linguistic_study/1.0"
)

# 配置参数
SUBREDDIT = "your_subreddit"
START_DATE = datetime(2023, 1, 1)
END_DATE = datetime(2023, 12, 31)

all_texts = []

# 分页获取帖子
for submission in reddit.subreddit(SUBREDDIT).search(
    "", sort="new", limit=None, after=START_DATE, before=END_DATE
):
    # 提取帖子正文（过滤空内容）
    if submission.selftext.strip():
        all_texts.append(submission.selftext.replace("\n", " "))
    # 加载所有评论
    submission.comments.replace_more(limit=0)
    for comment in submission.comments.list():
        if comment.body.strip():
            all_texts.append(comment.body.replace("\n", " "))

# 导出到CSV
with open("reddit_texts.csv", "w", encoding="utf-8", newline="") as f:
    writer = csv.writer(f)
    writer.writerow(["text"])
    for text in all_texts:
        writer.writerow([text])

三、适配AntConc的注意事项 #

• 替换所有换行符为空格：AntConc处理单行文本文档更高效
• 过滤空文本：避免CSV中出现空行干扰语料分析
• 编码用UTF-8：确保特殊字符（如 emoji、外来语）正常显示

内容的提问来源于stack exchange，提问作者nabbolone