如何使用access_token通过Requests模块登录网站并访问个人页面

我用Requests发送POST请求登录网站，代码如下：

import requests

session = requests.Session()

url = '...'
payload = '{"email":"my_email@gmail.com","password":"12345678*"}'
HEADER = {"content-type": "application/json"}
r = session.post(url=url, data=payload, headers=HEADER)

请求返回<Response [200]>，响应内容：

{
  "data": {
    "access_token": "T6u_VkzP9PkeRTHbPGVmH8gjGUcz8y_XwSMhc",
    "refresh_token": "eyJhbGciOiJI"
  },
  "message": "OK"
}

之后尝试获取个人页面，代码如下：

token = json.loads(r.text)['data']['access_token']
h = {'Authorization': token}
personal_page_url = '....'
r = session.get(url=personal_page_url, headers=h) # 或者尝试用data传参

但结果跳转到了验证码页面，而非个人页面。请问该如何正确使用这个access_token？

补充信息 #

以下是Chrome浏览器Network标签中的完整请求信息：

General

请求URL: https://www.yad2.co.il/personal/posts
请求方法: GET
状态码: 200 OK
远程地址: 141.226.253.90:443
Referrer策略: strict-origin-when-cross-origin

响应头

Cache-Control: max-age=86400
Connection: keep-alive
Content-Encoding: gzip
Content-Type: text/html
Date: Sat, 22 Oct 2022 18:52:31 GMT
Set-Cookie: __uzmc=9226236134145; HttpOnly; path=/; Expires=Sat, 22-Apr-23 18:52:31 GMT ; Max-Age=15724800; SameSite=Lax
Set-Cookie: __uzmd=1666464751; HttpOnly; path=/; Expires=Sat, 22-Apr-23 18:52:31 GMT ; Max-Age=15724800; SameSite=Lax
Set-Cookie: __uzmf=7f60001a351c8f-eaff-4593-8f28-1608d1067d2316653901902961074560738-07340a3d3abf80aa361; HttpOnly; path=/; Expires=Sat, 22-Apr-23 18:52:31 GMT ; Max-Age=15724800; SameSite=Lax
Transfer-Encoding: chunked
Vary: Accept-Encoding
Via: 1.1 a54cda8ccda3480314f451558e4dd062.cloudfront.net (CloudFront)
X-Amz-Cf-Id: AOruLPHS3MNR0VAhyXx5WXqxt0K90H16MJbJONDVqzsoxmxfzubjDQ==
X-Amz-Cf-Pop: FRA60-P3
x-amz-version-id: null
X-Cache: Miss from cloudfront
X-Frame-Options: SAMEORIGIN;
X-Xss-Protection: 1;

请求头

Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.7
Cache-Control: max-age=0
Connection: keep-alive
Cookie: __uzma=9b00ee88-fdcb-4618-a086-926fe1d0e135; __uzmb=1665390190; __uzme=9550; abTestKey=92; y2018-2-cohort=10; leadSaleRentFree=60; use_elastic_search=1; bc.visitor_token=6985233079423275008; y2session=gfiod758btt6mq0t87mfr1rdj7; recommendations-searched-1=21; recommendations-searched-3=0; recommendations-home-category={"categoryId":3,"subCategoryId":0}; y2_cohort_2020=58; canary=never; server_env=production; personalRedirect=1; favorites_userid=fvu4934081; refresh_token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJFbWFpbCI6ImZhbHVra3lAZ21haWwuY29tIiwiVXNlcklEIjo0OTM0MDgxLCJGaXJzdE5hbWUiOiLXqNeR15nXkSIsIkxhc3ROYW1lIjoi16nXmNeZ15nXoNeR16jXkiIsImlzQ2FyVHJhZGVyIjowLCJpc0NhckFjY2Vzc29yeVRyYWRlciI6MCwiaXNUb3VyaXNtVHJhZGVyIjowLCJpc1JlYWxFc3RhdGVNYXJrZXRpbmciOjAsIllhZDJUcmFkZSI6MCwiRW1haWxWZXJpZmllZCI6IjIwMjEtMTEtMjdUMTc6MDY6NDIuMDAwWiIsImlzVHdvV2hlZWxlZFRyYWRlciI6MCwiVGl2IjowLCJpYXQiOjE2NjY0MTgzNDksImV4cCI6MTY3NDE5NDM0OX0.TZi9X0ippZoIJnGP1afdsNAEMwo_ktBEyCQh0jcGD58; __uzmc=3827935822160; __uzmd=1666447956; __uzmf=7f60001a351c8f-eaff-4593-8f28-1608d1067d2316653901902961057766489-7ce2e5538791c6fc358; access_token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJFbWFpbCI6ImZhbHVra3lAZ21haWwuY29tIiwiVXNlcklEIjo0OTM0MDgxLCJGaXJzdE5hbWUiOiLXqNeR15nXkSIsIkxhc3ROYW1lIjoi16nXmNeZ15nXoNeR16jXkiIsImlzQ2FyVHJhZGVyIjowLCJpc0NhckFjY2Vzc29yeVRyYWRlciI6MCwiaXNUb3VyaXNtVHJhZGVyIjowLCJpc1JlYWxFc3RhdGVNYXJrZXRpbmciOjAsIllhZDJUcmFkZSI6MCwiRW1haWxWZXJpZmllZCI6IjIwMjEtMTEtMjdUMTc6MDY6NDIuMDAwWiIsImlzVHdvV2hlZWxlZFRyYWRlciI6MCwiVGl2IjowLCJpYXQiOjE2NjY0NjQ3MTUsImV4cCI6MTY2NjQ2NTYxNX0.W6AEWOKLSPiSbzxKVfTtnQYkkOUIIP-EVAFEwFYBVCs
Host: www.yad2.co.il
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: same-origin
Sec-Fetch-User: ?1
Sec-GPC: 1
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36

解决方案 #

1. 修正access_token的传递方式 #

从浏览器的请求Cookie可以看出，该网站是通过Cookie而非Authorization请求头来验证access_token的。你需要把获取到的access_token和refresh_token添加到session的Cookie中。

2. 模拟浏览器完整请求头 #

网站会校验请求的来源特征，需要模拟浏览器的核心请求头（比如User-Agent、Accept等），避免被识别为非浏览器请求。

修改后的代码示例 #

import requests
import json

session = requests.Session()

# 登录请求，先添加基础浏览器标识头
login_url = '...'
payload = '{"email":"my_email@gmail.com","password":"12345678*"}'
login_headers = {
    "content-type": "application/json",
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36"
}
r = session.post(url=login_url, data=payload, headers=login_headers)

# 解析响应获取token
response_data = json.loads(r.text)
access_token = response_data['data']['access_token']
refresh_token = response_data['data']['refresh_token']

# 将token写入session的Cookie，保持会话一致性
session.cookies.set('access_token', access_token)
session.cookies.set('refresh_token', refresh_token)

# 访问个人页面，模拟完整浏览器请求头
personal_page_url = 'https://www.yad2.co.il/personal/posts'
page_headers = {
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.7",
    "Cache-Control": "max-age=0",
    "Upgrade-Insecure-Requests": "1",
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36"
}

r = session.get(url=personal_page_url, headers=page_headers)
# 检查响应内容是否为个人页面
print(r.text)

3. 额外注意事项 #

• 全程复用同一个session对象，确保会话Cookie的一致性；
• 如果仍触发验证码，建议先访问网站首页，获取浏览器生成的初始Cookie（如__uzma、y2session等）后再执行登录流程，模拟真实用户的访问顺序；
• 部分网站会检测请求的行为模式，尽量放慢请求间隔，避免短时间内连续发送请求。

内容的提问来源于stack exchange，提问作者user979033