如何加速基于PRAW的Reddit跨帖子爬取代码？

问题描述：我正在爬取约1000个热门Subreddit的近期跨帖子，但当前代码运行耗时较长。以下是我的实现代码：

submission_data = []
sub_count = 0
for sub in popsublist:
    count = 0
    sub_count += 1
    print('============================')
    print('subs-looped count:',sub_count)
    print('current sub:',sub)
    print('============================')
    sub_loop = 0
    for post in reddit.subreddit(sub).hot(limit=500):
        sub_loop += 1
        print("posts-looped count",sub_loop)
        if hasattr(post, "crosspost_parent"):
            count += 1
            print('posts-loop count [ADDED!]:',count)
            op = reddit.submission(id=post.crosspost_parent.split("_")[1]).subreddit
            submission_data.append({
                'SOURCE_SUB': str(post.subreddit),
                'TARGET_SUB': str(op),
                'POST_ID': str(post.id),
                'POST_TITLE': str(post.title),
                'POST_DATE': datetime.utcfromtimestamp(int(post.created_utc)).strftime('%Y-%m-%d %H:%M:%S'),
                'POST_LINK': str('http://www.reddit.com'+post.permalink),
                'POST_SCORE': post.score,
                'POST_NSFW': post.over_18,
            })

关键优化措施 #

1. 改用多线程并发处理Subreddit #

原代码串行遍历每个Subreddit，每个请求都要等待前一个完成，效率极低。用多线程同时处理多个Subreddit，能大幅缩短总耗时。注意控制线程数（建议10-20个），避免触发Reddit API的速率限制（默认每分钟60次，注册应用后可提升至100次）：

from concurrent.futures import ThreadPoolExecutor
import datetime

def process_subreddit(sub):
    local_data = []
    try:
        for post in reddit.subreddit(sub).hot(limit=500):
            if hasattr(post, "crosspost_parent"):
                op_id = post.crosspost_parent.split("_")[1]
                op_sub = str(reddit.submission(id=op_id).subreddit)
                local_data.append({
                    'SOURCE_SUB': str(post.subreddit),
                    'TARGET_SUB': op_sub,
                    'POST_ID': str(post.id),
                    'POST_TITLE': str(post.title),
                    'POST_DATE': datetime.utcfromtimestamp(int(post.created_utc)).strftime('%Y-%m-%d %H:%M:%S'),
                    'POST_LINK': f'http://www.reddit.com{post.permalink}',
                    'POST_SCORE': post.score,
                    'POST_NSFW': post.over_18,
                })
    except Exception as e:
        print(f"处理{sub}失败: {str(e)}")
    return local_data

# 初始化线程池并执行
submission_data = []
with ThreadPoolExecutor(max_workers=15) as executor:
    for result in executor.map(process_subreddit, popsublist):
        submission_data.extend(result)

2. 缓存已查询的原帖子Subreddit #

每次遇到跨帖子都重新请求原帖子属于重复API调用，会浪费大量时间。用缓存存储已查询过的帖子ID对应的Subreddit名称，避免重复请求：

from functools import lru_cache

# 缓存最多10000条记录，平衡缓存效率与内存占用
@lru_cache(maxsize=10000)
def get_op_subreddit(op_id):
    return str(reddit.submission(id=op_id).subreddit)

# 在process_subreddit函数中替换原获取op_sub的代码：
op_sub = get_op_subreddit(op_id)

3. 移除冗余的打印操作 #

频繁的print是同步IO操作，会严重拖慢爬取速度。建议只保留关键错误日志，或每处理N个Subreddit/帖子才打印一次进度：

# 示例：每处理20个Subreddit打印一次进度
submission_data = []
with ThreadPoolExecutor(max_workers=15) as executor:
    for idx, result in enumerate(executor.map(process_subreddit, popsublist), 1):
        submission_data.extend(result)
        if idx % 20 == 0:
            print(f"已处理{idx}/{len(popsublist)}个Subreddit")

4. 优化日期格式化逻辑 #

将日期格式化操作封装成独立函数，减少重复调用的开销：

def format_utc_timestamp(timestamp):
    return datetime.utcfromtimestamp(int(timestamp)).strftime('%Y-%m-%d %H:%M:%S')

# 在添加数据时使用：
'POST_DATE': format_utc_timestamp(post.created_utc),

5. 启用只读模式提升API速率 #

如果爬取不需要写操作（发帖、评论等），开启PRAW的只读模式，可获得更宽松的API速率限制：

reddit.read_only = True

内容的提问来源于stack exchange，提问作者Albert AG