使用requests.get()调用Pinterest接口返回内容不完整的问题

我使用Pinterest的搜索接口URL获取搜索结果，接口原本应返回包含图片数据的字典格式数据，但通过requests.get()请求后，解析得到的data变量和浏览器中显示的内容差异很大，丢失了大量图片数据。尝试过增加timeout=10、更换urllib.request/http.client库，问题依然存在。

接口URL：

https://www.pinterest.com/resource/BaseSearchResource/get/?source_url=%2Fsearch%2Fpins%2F%3Fq%3Dyellow%2520car%2520on%2520T-shirt%26rs%3Dtyped%26term_meta%5B%5D%3Dyellow%257Ctyped%26term_meta%5B%5D%3Dcar%257Ctyped%26term_meta%5B%5D%3Don%257Ctyped%26term_meta%5B%5D%3DT-shirt%257Ctyped&data=%7B%22options%22%3A%20%7B%22isPrefetch%22%3A%20false%2C%20%22auto_correction_disabled%22%3A%20false%2C%20%22query%22%3A%20%22yellow%20car%20on%20T-shirt%22%2C%20%22redux_normalize_feed%22%3A%20true%2C%20%22rs%22%3A%20%22typed%22%2C%20%22scope%22%3A%20%22pins%22%2C%20%22page_size%22%3A%2050%2C%20%22bookmarks%22%3A%20%5Bnull%5D%7D%2C%20%22context%22%3A%20null%7D&_=1729500369393

使用的代码：

import requests
import json

url='https://www.pinterest.com/resource/BaseSearchResource/get/?source_url=%2Fsearch%2Fpins%2F%3Fq%3Dyellow%2520car%2520on%2520T-shirt%26rs%3Dtyped%26term_meta%5B%5D%3Dyellow%257Ctyped%26term_meta%5B%5D%3Dcar%257Ctyped%26term_meta%5B%5D%3Don%257Ctyped%26term_meta%5B%5D%3DT-shirt%257Ctyped&data=%7B%22options%22%3A%20%7B%22isPrefetch%22%3A%20false%2C%20%22auto_correction_disabled%22%3A%20false%2C%20%22query%22%3A%20%22yellow%20car%20on%20T-shirt%22%2C%20%22redux_normalize_feed%22%3A%20true%2C%20%22rs%22%3A%20%22typed%22%2C%20%22scope%22%3A%20%22pins%22%2C%20%22page_size%22%3A%2050%2C%20%22bookmarks%22%3A%20%5Bnull%5D%7D%2C%20%22context%22%3A%20null%7D&_=1729500369393'

response = requests.get(url)

if response.status_code == 200:
    data = json.loads(response.text)

解决方法 #

1. 添加模拟浏览器的请求头 #

Pinterest会检测请求来源，缺少浏览器标识的请求会返回简化版数据。必须添加User-Agent，建议同时带上登录后的Cookie（从浏览器Pinterest请求中复制）和其他关键头：

import requests
import json

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36',
    'Cookie': '替换为你浏览器中Pinterest站点的Cookie值',
    'Accept-Language': 'en-US,en;q=0.9',
    'Referer': 'https://www.pinterest.com/search/pins/?q=yellow%20car%20on%20T-shirt'
}

# 修正URL中的HTML转义符&为&
url = 'https://www.pinterest.com/resource/BaseSearchResource/get/?source_url=%2Fsearch%2Fpins%2F%3Fq%3Dyellow%2520car%2520on%2520T-shirt%26rs%3Dtyped%26term_meta%5B%5D%3Dyellow%257Ctyped%26term_meta%5B%5D%3Dcar%257Ctyped%26term_meta%5B%5D%3Don%257Ctyped%26term_meta%5B%5D%3DT-shirt%257Ctyped&data=%7B%22options%22%3A%20%7B%22isPrefetch%22%3A%20false%2C%20%22auto_correction_disabled%22%3A%20false%2C%20%22query%22%3A%20%22yellow%20car%20on%20T-shirt%22%2C%20%22redux_normalize_feed%22%3A%20true%2C%20%22rs%22%3A%20%22typed%22%2C%20%22scope%22%3A%20%22pins%22%2C%20%22page_size%22%3A%2050%2C%20%22bookmarks%22%3A%20%5Bnull%5D%7D%2C%20%22context%22%3A%20null%7D&_=1729500369393'

response = requests.get(url, headers=headers)

if response.status_code == 200:
    data = json.loads(response.text)
    # 检查图片结果是否存在
    print(data.get('resource_response', {}).get('data', {}).get('results', []))

2. 动态生成时间戳参数_ #

URL中的_是毫秒级时间戳，固定值可能被识别为过期请求，建议每次请求生成当前时间戳：

import time

timestamp = str(int(time.time() * 1000))
url = url.replace('_=1729500369393', f'_={timestamp}')

3. 使用会话保持状态 #

Pinterest依赖会话标识验证请求合法性，用requests.Session()维持会话：

session = requests.Session()
session.headers.update(headers)
response = session.get(url)

4. 验证data参数格式 #

data是URL编码后的JSON，可解码确认格式正确性，再重新编码：

import urllib.parse

# 解码data参数
decoded_data = urllib.parse.unquote('%7B%22options%22%3A%20%7B%22isPrefetch%22%3A%20false%2C%20%22auto_correction_disabled%22%3A%20false%2C%20%22query%22%3A%20%22yellow%20car%20on%20T-shirt%22%2C%20%22redux_normalize_feed%22%3A%20true%2C%20%22rs%22%3A%20%22typed%22%2C%20%22scope%22%3A%20%22pins%22%2C%20%22page_size%22%3A%2050%2C%20%22bookmarks%22%3A%20%5Bnull%5D%7D%2C%20%22context%22%3A%20null%7D')
print(decoded_data)
# 重新编码替换回URL
encoded_data = urllib.parse.quote(decoded_data)
url = url.replace('%7B%22options%22%3A%20%7B%22isPrefetch%22%3A%20false%2C%20%22auto_correction_disabled%22%3A%20false%2C%20%22query%22%3A%20%22yellow%20car%20on%20T-shirt%22%2C%20%22redux_normalize_feed%22%3A%20true%2C%20%22rs%22%3A%20%22typed%22%2C%20%22scope%22%3A%20%22pins%22%2C%20%22page_size%22%3A%2050%2C%20%22bookmarks%22%3A%20%5Bnull%5D%7D%2C%20%22context%22%3A%20null%7D', encoded_data)

内容的提问来源于stack exchange，提问作者Othmane Bazine