终端curl可正常执行GET请求，但Python Requests返回403禁止访问

问题场景 #

终端执行以下curl命令可正常完成GET请求：

curl -kg 'https://website.com/wellview/api/tablename?filter[where][id]=123456' -H "Authorization: Bearer sjfjkdjskdfjlsdflks12313123"

但使用Python Requests库复现请求时，收到403 Forbidden响应：

s = requests.Session()
path=u'https://website.com/wellview/api/tablename?filter[where][id]=123456'
bearer_token="sjfjkdjskdfjlsdflks12313123"
headers = {"Authorization": f"Bearer {bearer_token}"}
r= s.get(path, headers = headers, verify = False)

请求日志显示，Requests自动将URL中的方括号编码为百分形式：

2023-07-21 17:43:32,267 :: DEBUG :: urllib3.connectionpool :: Starting new HTTPS connection (1): website.com:443
2023-07-21 17:43:32,370 :: DEBUG :: urllib3.connectionpool :: https://website.com:443 "GET /wellview/api/tablename?filter%5Bwhere%5D%5Bid%5D=123456 HTTP/1.1" 403 146

即使将筛选条件作为params参数传入，结果依然一致：

base_url = "https://website.com/wellview/api/tablename"
bearer_token="sjfjkdjskdfjlsdflks12313123"
headers = {"Authorization": f"Bearer {bearer_token}"}
params = {'filter[where][id]': '123456'}            
response = requests.get(base_url, headers=headers, params=params,verify=False)

原因分析 #

curl默认不会对URL中的[和]进行百分编码，而Requests库遵循RFC 3986标准，会将这两个字符编码为%5B和%5D。但目标服务器的接口未兼容这种编码格式，导致请求被拒绝返回403。

可行解决方案 #

方案1：手动构造未编码URL并修改预请求对象 #

通过构造Request对象，在发送前手动还原被编码的方括号：

import requests

session = requests.Session()
target_url = 'https://website.com/wellview/api/tablename?filter[where][id]=123456'
bearer_token = "sjfjkdjskdfjlsdflks12313123"
headers = {"Authorization": f"Bearer {bearer_token}"}

# 构造请求对象
req = requests.Request('GET', target_url, headers=headers)
prepped_req = session.prepare_request(req)

# 还原被编码的方括号
prepped_req.url = prepped_req.url.replace('%5B', '[').replace('%5D', ']')

# 发送请求
response = session.send(prepped_req, verify=False)
print(response.status_code)

方案2：使用urllib3直接发送请求 #

绕过Requests的自动编码逻辑，用urllib3直接构造请求：

import urllib3

bearer_token = "sjfjkdjskdfjlsdflks12313123"
headers = {"Authorization": f"Bearer {bearer_token}"}

http = urllib3.PoolManager(verify=False)
response = http.request(
    'GET',
    'https://website.com/wellview/api/tablename?filter[where][id]=123456',
    headers=headers
)
print(response.status)
print(response.data.decode('utf-8'))

方案3：自定义URL编码规则 #

通过urllib.parse手动编码参数，排除对[]的编码：

import requests
from urllib.parse import urlencode

base_url = "https://website.com/wellview/api/tablename"
params = {'filter[where][id]': '123456'}

# 手动编码参数并还原方括号
encoded_params = urlencode(params).replace('%5B', '[').replace('%5D', ']')
full_url = f"{base_url}?{encoded_params}"

bearer_token = "sjfjkdjskdfjlsdflks12313123"
headers = {"Authorization": f"Bearer {bearer_token}"}

response = requests.get(full_url, headers=headers, verify=False)
print(response.status_code)

内容的提问来源于stack exchange，提问作者9879ypxkj