使用snscrape的TwitterSearchScraper查询时请求失败的技术求助

问题背景 #

此前一直用以下Python代码通过snscrape爬取Twitter推文：

import snscrape.modules.twitter as sntwitter
query = "from:annewilltalk"
for i,tweet in enumerate(sntwitter.TwitterSearchScraper(query).get_items()):
     # 处理推文逻辑

今日未修改代码却出现报错，完整错误栈如下：

Traceback (most recent call last):
File "twitter.py", line 568, in <module>
    Scraper = TwitterScraper()
  File "twitter.py", line 66, in __init__
    self.get_tweets_talkshow(username = "annewilltalk")
  File "twitter.py", line 271, in get_tweets_talkshow
    for i,tweet in enumerate(sntwitter.TwitterSearchScraper(query).get_items()):
  File "/home/pi/.local/lib/python3.8/site-packages/snscrape/modules/twitter.py", line 1455, in get_items
    for obj in self._iter_api_data('https://api.twitter.com/2/search/adaptive.json', _TwitterAPIType.V2, params, paginationParams, cursor = self._cursor):
  File "/home/pi/.local/lib/python3.8/site-packages/snscrape/modules/twitter.py", line 721, in _iter_api_data
    obj = self._get_api_data(endpoint, apiType, reqParams)
  File "/home/pi/.local/lib/python3.8/site-packages/snscrape/modules/twitter.py", line 691, in _get_api_data
    r = self._get(endpoint, params = params, headers = self._apiHeaders, responseOkCallback = self._check_api_response)
  File "/home/pi/.local/lib/python3.8/site-packages/snscrape/base.py", line 221, in _get
    return self._request('GET', *args, **kwargs)
  File "/home/pi/.local/lib/python3.8/site-packages/snscrape/base.py", line 217, in _request
    raise ScraperException(msg)
snscrape.base.ScraperException: 4 requests to https://api.twitter.com/2/search/adaptive.json?include_profile_interstitial_type=1&include_blocking=1&include_blocked_by=1&include_followed_by=1&include_want_retweets=1&include_mute_edge=1&include_can_dm=1&include_can_media_tag=1&include_ext_has_nft_avatar=1&skip_status=1&cards_platform=Web-12&include_cards=1&include_ext_alt_text=true&include_quote_count=true&include_reply_count=1&tweet_mode=extended&include_entities=true&include_user_entities=true&include_ext_media_color=true&include_ext_media_availability=true&include_ext_sensitive_media_warning=true&include_ext_trusted_friends_metadata=true&send_error_codes=true&simple_quoted_tweet=true&q=from%3Aannewilltalk&tweet_search_mode=live&count=20&query_source=spelling_expansion_revert_click&pc=1&spelling_corrections=1&ext=mediaStats%2ChighlightedLabel%2ChasNftAvatar%2CvoiceInfo%2Cenrichments%2CsuperFollowMetadata%2CunmentionInfo failed, giving up.

问题分析与解答 #

1. URL长度是否是问题根源？ #

不是。HTTP规范无严格URL长度限制，Twitter服务器支持的URL长度远超过800字符。报错核心是请求连续失败，而非URL长度被拒绝，此猜测不成立。

2. 为何突然出现变化？ #

大概率是Twitter调整了API规则：

• snscrape依赖的非官方API端点（/2/search/adaptive.json）被限制或修改了参数要求；
• Twitter加强了反爬机制，未授权的频繁请求被拦截；
• snscrape版本过旧，未适配Twitter最新的API参数变化。

3. 解决办法 #

• 升级snscrape到最新版本：执行命令 pip install --upgrade snscrape，新版本通常会适配Twitter的API变更；
• 切换到Twitter官方API：若爬取需求稳定，建议申请Twitter Developer账号，使用官方API（如Twitter API v2），避免依赖非官方端点的不稳定问题；
• 添加请求延迟：在循环中加入 time.sleep(1) 之类的延迟，降低请求频率，减少被拦截概率；
• 检查代理/IP状态：若IP被Twitter拉黑，更换IP或使用代理请求。

内容的提问来源于stack exchange，提问作者Felix N