Steam Web API爬取Apex Legends评论不全问题求助

针对你爬取Apex Legends（AppID:1172470）评论仅能拿到1427条、游标进入死循环的问题，给你几个实操修复方案：

1. 修复游标编码逻辑 #

你当前代码对返回的游标二次编码会破坏格式，导致API无法识别。Steam返回的cursor已经是URL安全编码后的字符串，直接用requests的参数自动处理编码即可：

def fetch_game_reviews(appid, cursor='*', filter='recent', language='english'):
    url_params = {
        'json': 1,
        'cursor': cursor,
        'filter': filter,
        'language': language,
        'num_per_page': 100,
        'filter_offtopic_activity': 0
    }
    response = requests.get(f"https://store.steampowered.com/appreviews/{appid}", params=url_params)
    return response.json()

2. 处理游标死循环 #

当遇到返回0条评论且游标重复的情况，不要直接终止，尝试切换过滤规则或调整每页数量，跳过死循环游标：

def fetch_reviews_until_date(appid, end_date, filter='recent', language='all'):
    end_date = datetime.strptime(end_date, '%Y-%m-%d')
    all_reviews = []
    cursors = []
    cursor = '*'
    retry_count = 0
    max_retries = 3
    current_filter = filter

    while True:
        # 添加请求延迟，避免触发Steam限流
        time.sleep(1.5)
        response = fetch_game_reviews(appid, cursor=cursor, filter=current_filter, language=language)
        collected_timestamp = int(time.time())
        reviews = response.get('reviews', [])
        new_cursor = response.get('cursor')
        cursors.append(new_cursor)

        # 连续3次空结果+重复游标，切换过滤规则重置
        if not reviews:
            retry_count += 1
            if retry_count >= max_retries:
                current_filter = 'all' if current_filter == 'recent' else 'recent'
                cursor = '*'
                retry_count = 0
                continue
            cursor = new_cursor
            continue
        retry_count = 0

        # 处理日期判断：遇到早于结束日期的评论就终止
        for review in reviews:
            review_date = datetime.fromtimestamp(review['timestamp_created'])
            review['collected'] = collected_timestamp
            review['appid'] = appid
            if review_date < end_date:
                return all_reviews, cursors
            all_reviews.append(review)
            print(f"已收集评论数: {len(all_reviews)}")
            print(f"当前游标: {new_cursor}")

        cursor = new_cursor

3. 适配Steam API隐性限制 #

• 热门游戏的历史评论可能被归档，API不对外暴露，切换filter=all比recent能获取更多历史数据
• 必须添加请求延迟（1-2秒），Steam对高频请求会限流，直接返回空数据
• 若仍有数据限制，尝试将num_per_page调整为50，减少单页请求的数据量

4. 修正日期判断逻辑 #

原代码中遇到一条早于结束日期的评论就直接返回，会丢失当前页中早于该评论的新数据。上述修改后的逻辑会遍历完当前页所有符合时间要求的评论后再终止。

内容的提问来源于stack exchange，提问作者clarcia