使用snscrape爬取Twitter英文推文遇404(blocked)错误求解决

5天前用snscrape爬取指定用户英文推文完全正常，现在突然报404(blocked)错误，已经升级过snscrape库，问题依旧。要求不使用官方API，修复该错误以正常爬取。

原代码 #

import pandas as pd
import snscrape.modules.twitter as sntwitter
scrapping_link = "https://twitter.com/tim_cook"
mycnt = 10
csv_file = "E:/django project/mywebsite/static/multiple posts/scraped_tweets.csv"   
username = scrapping_link.split('/')[-1]  # 从URL提取用户名   
count = 0  # 已爬取推文计数器
tweets_data = []  # 存储推文的列表    
for tweet in sntwitter.TwitterUserScraper(username).get_items():
  if count >= int(mycnt):
    break   
  if tweet.lang == "en" and tweet.content:
    tweet_text = tweet.content
    tweets_data.append({"username": username, "text": tweet_text})
    count += 1
# 将推文数据转为DataFrame
df = pd.DataFrame(tweets_data, columns=["username", "text"])
# 保存为CSV文件
df.to_csv(index=False, encoding="utf-8")                 

错误信息 #

ERROR:snscrape.base:Error retrieving https://twitter.com/i/api/graphql/7jT5GT59P8IFjgxwqnEdQw/SearchTimeline?variables=%7B%22rawQuery%22%3A%22from%3Atim_cook%22%2C%22count%22%3A20%2C%22product%22%3A%22Latest%22%2C%22withDownvotePerspective%22%3Afalse%2C%22withReactionsMetadata%22%3Afalse%2C%22withReactionsPerspective%22%3Afalse%7D&features=%7B%22rweb_lists_timeline_redesign_enabled%22%3Afalse%2C%22blue_business_profile_image_shape_enabled%22%3Afalse%2C%22responsive_web_graphql_exclude_directive_enabled%22%3Atrue%2C%22verified_phone_label_enabled%22%3Afalse%2C%22creator_subscriptions_tweet_preview_api_enabled%22%3Afalse%2C%22responsive_web_graphql_timeline_navigation_enabled%22%3Atrue%2C%22responsive_web_graphql_skip_user_profile_image_extensions_enabled%22%3Afalse%2C%22tweetypie_unmention_optimization_enabled%22%3Atrue%2C%22vibe_api_enabled%22%3Atrue%2C%22responsive_web_edit_tweet_api_enabled%22%3Atrue%2C%22graphql_is_translatable_rweb_tweet_is_translatable_enabled%22%3Atrue%2C%22view_counts_everywhere_api_enabled%22%3Atrue%2C%22longform_notetweets_consumption_enabled%22%3Atrue%2C%22tweet_awards_web_tipping_enabled%22%3Afalse%2C%22freedom_of_speech_not_reach_fetch_enabled%22%3Afalse%2C%22standardized_nudges_misinfo%22%3Atrue%2C%22tweet_with_visibility_results_prefer_gql_limited_actions_policy_enabled%22%3Afalse%2C%22interactive_text_enabled%22%3Atrue%2C%22responsive_web_text_conversations_enabled%22%3Afalse%2C%22longform_notetweets_rich_text_read_enabled%22%3Afalse%2C%22longform_notetweets_inline_media_enabled%22%3Afalse%2C%22responsive_web_enhance_cards_enabled%22%3Afalse%2C%22responsive_web_twitter_blue_verified_badge_is_enabled%22%3Atrue%7D: blocked (404)
CRITICAL:snscrape.base:4 requests to https://twitter.com/i/api/graphql/7jT5GT59P8IFjgxwqnEdQw/SearchTimeline?variables=%7B%22rawQuery%22%3A%22from%3Atim_cook%22%2C%22count%22%3A20%2C%22product%22%3A%22Latest%22%2C%22withDownvotePerspective%22%3Afalse%2C%22withReactionsMetadata%22%3Afalse%2C%22withReactionsPerspective%22%3Afalse%7D&features=%7B%22rweb_lists_timeline_redesign_enabled%22%3Afalse%2C%22blue_business_profile_image_shape_enabled%22%3Afalse%2C%22responsive_web_graphql_exclude_directive_enabled%22%3Atrue%2C%22verified_phone_label_enabled%22%3Afalse%2C%22creator_subscriptions_tweet_preview_api_enabled%22%3Afalse%2C%22responsive_web_graphql_timeline_navigation_enabled%22%3Atrue%2C%22responsive_web_graphql_skip_user_profile_image_extensions_enabled%22%3Afalse%2C%22tweetypie_unmention_optimization_enabled%22%3Atrue%2C%22vibe_api_enabled%22%3Atrue%2C%22responsive_web_edit_tweet_api_enabled%22%3Atrue%2C%22graphql_is_translatable_rweb_tweet_is_translatable_enabled%22%3Atrue%2C%22view_counts_everywhere_api_enabled%22%3Atrue%2C%22longform_notetweets_consumption_enabled%22%3Atrue%2C%22tweet_awards_web_tipping_enabled%22%3Afalse%2C%22freedom_of_speech_not_reach_fetch_enabled%22%3Afalse%2C%22standardized_nudges_misinfo%22%3Atrue%2C%22tweet_with_visibility_results_prefer_gql_limited_actions_policy_enabled%22%3Afalse%2C%22interactive_text_enabled%22%3Atrue%2C%22responsive_web_text_conversations_enabled%22%3Afalse%2C%22longform_notetweets_rich_text_read_enabled%22%3Afalse%2C%22longform_notetweets_inline_media_enabled%22%3Afalse%2C%22responsive_web_enhance_cards_enabled%22%3Afalse%2C%22responsive_web_twitter_blue_verified_badge_is_enabled%22%3Atrue%7D failed, giving up.
CRITICAL:snscrape.base:Errors: blocked (404), blocked (404), blocked (404), blocked (404)
---------------------------------------------------------------------------
ScraperException                          Traceback (most recent call last)
<ipython-input-25-a22aa37f9c43> in <cell line: 9>()
      7 count = 0  # Counter for the number of scraped tweets
      8 tweets_data = []  # List to store the tweets
----> 9 for tweet in sntwitter.TwitterUserScraper(username).get_items():
     10   if count >= int(mycnt):
     11     break

5 frames
/usr/local/lib/python3.10/dist-packages/snscrape/base.py in _request(self, method, url, params, data, headers, timeout, responseOkCallback, allowRedirects, proxies)
    269                         _logger.fatal(msg)
    270                         _logger.fatal(f'Errors: {", ".join(errors)}')
--> 271                         raise ScraperException(msg)
    272                 raise RuntimeError('Reached unreachable code')
    273 

ScraperException: 4 requests to https://twitter.com/i/api/graphql/7jT5GT59P8IFjgxwqnEdQw/SearchTimeline?variables=%7B%22rawQuery%22%3A%22from%3Atim_cook%22%2C%22count%22%3A20%2C%22product%22%3A%22Latest%22%2C%22withDownvotePerspective%22%3Afalse%2C%22withReactionsMetadata%22%3Afalse%2C%22withReactionsPerspective%22%3Afalse%7D&features=%7B%22rweb_lists_timeline_redesign_enabled%22%3Afalse%2C%22blue_business_profile_image_shape_enabled%22%3Afalse%2C%22responsive_web_graphql_exclude_directive_enabled%22%3Atrue%2C%22verified_phone_label_enabled%22%3Afalse%2C%22creator_subscriptions_tweet_preview_api_enabled%22%3Afalse%2C%22responsive_web_graphql_timeline_navigation_enabled%22%3Atrue%2C%22responsive_web_graphql_skip_user_profile_image_extensions_enabled%22%3Afalse%2C%22tweetypie_unmention_optimization_enabled%22%3Atrue%2C%22vibe_api_enabled%22%3Atrue%2C%22responsive_web_edit_tweet_api_enabled%22%3Atrue%2C%22graphql_is_translatable_rweb_tweet_is_translatable_enabled%22%3Atrue%2C%22view_counts_everywhere_api_enabled%22%3Atrue%2C%22longform_notetweets_consumption_enabled%22%3Atrue%2C%22tweet_awards_web_tipping_enabled%22%3Afalse%2C%22freedom_of_speech_not_reach_fetch_enabled%22%3Afalse%2C%22standardized_nudges_misinfo%22%3Atrue%2C%22tweet_with_visibility_results_prefer_gql_limited_actions_policy_enabled%22%3Afalse%2C%22interactive_text_enabled%22%3Atrue%2C%22responsive_web_text_conversations_enabled%22%3Afalse%2C%22longform_notetweets_rich_text_read_enabled%22%3Afalse%2...

解决方案 #

方法1：添加请求头与代理 #

Twitter对无标识请求拦截加剧，给snscrape模拟浏览器请求头，搭配代理IP绕过限制：

import pandas as pd
import snscrape.modules.twitter as sntwitter

# 模拟Chrome浏览器请求头
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36'
}

# 配置代理（替换为可用的代理地址）
proxies = {
    'http': 'http://your-proxy-address:port',
    'https': 'http://your-proxy-address:port'
}

scrapping_link = "https://twitter.com/tim_cook"
mycnt = 10
csv_file = "E:/django project/mywebsite/static/multiple posts/scraped_tweets.csv"   
username = scrapping_link.split('/')[-1]
count = 0
tweets_data = []

# 初始化Scraper并注入请求头和代理
scraper = sntwitter.TwitterUserScraper(username)
scraper._headers = headers
scraper._proxies = proxies

for tweet in scraper.get_items():
    if count >= int(mycnt):
        break
    if tweet.lang == "en" and tweet.content:
        tweets_data.append({"username": username, "text": tweet.content})
        count += 1

df = pd.DataFrame(tweets_data, columns=["username", "text"])
df.to_csv(csv_file, index=False, encoding="utf-8")

方法2：改用TwitterSearchScraper #

通过搜索语法直接限定用户和语言，绕过用户时间线API的限制：

import pandas as pd
import snscrape.modules.twitter as sntwitter

scrapping_link = "https://twitter.com/tim_cook"
mycnt = 10
csv_file = "E:/django project/mywebsite/static/multiple posts/scraped_tweets.csv"   
username = scrapping_link.split('/')[-1]
count = 0
tweets_data = []

# 搜索语法：限定用户+英文推文
search_query = f"from:{username} lang:en"

for tweet in sntwitter.TwitterSearchScraper(search_query).get_items():
    if count >= int(mycnt):
        break
    if tweet.content:
        tweets_data.append({"username": username, "text": tweet.content})
        count += 1

df = pd.DataFrame(tweets_data, columns=["username", "text"])
df.to_csv(csv_file, index=False, encoding="utf-8")

方法3：安装snscrape开发版本 #

稳定版可能未适配Twitter最新API，安装开发分支版本：

pip uninstall -y snscrape
pip install git+https://github.com/JustAnotherArchivist/snscrape.git

内容的提问来源于stack exchange，提问作者Umer