使用Curl请求Glassdoor链接返回200，Python Requests返回403问题求助

请求Glassdoor的某个职位URL时，curl请求返回200状态码，但Python Requests请求返回403。具体请求及响应如下：

Curl请求代码及响应 #

curl --head 'https://www.glassdoor.com/Job/scottsdale-az-stock-clerk-jobs-SRCH_IL.0,13_IC1133911_KO14,25.htm?src=GD_JOB_AD&rdserp=true&srs=EI_JOBS&jl=1008874519264&ao=1136043&s=21&guid=0000018ab3ad27ccbc470b1f6b275b17&pos=101&t=ESR&vt=w&uido=E2E7DBC4ACD391CABF382B9AEDAFF99C&ea=1&cs=1_f8a82cfa&cb=1695231584675&jobListingId=1008874519264&jrtk=5-yul1-0-1hapqqa33ir2a800-0383467bbae09c6a' \
-H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36' \
--compressed

响应：

HTTP/2 200 
date: Wed, 20 Sep 2023 17:59:25 GMT
content-type: text/html; charset=UTF-8
cache-control: no-store

Python Requests请求代码及响应 #

import requests

headers = {
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36',
}

response = requests.get(
    'https://www.glassdoor.com/Job/scottsdale-az-stock-clerk-jobs-SRCH_IL.0,13_IC1133911_KO14,25.htm?src=GD_JOB_AD&rdserp=true&srs=EI_JOBS&jl=1008874519264&ao=1136043&s=21&guid=0000018ab3ad27ccbc470b1f6b275b17&pos=101&t=ESR&vt=w&uido=E2E7DBC4ACD391CABF382B9AEDAFF99C&ea=1&cs=1_f8a82cfa&cb=1695231584675&jobListingId=1008874519264&jrtk=5-yul1-0-1hapqqa33ir2a800-0383467bbae09c6a',
    headers=headers,
)
print(response)

响应：

<Response [403]>

1. 请求头不完整：curl默认会携带Accept-Encoding等浏览器常用请求头，而你的Requests代码仅设置了User-Agent，缺少Accept、Accept-Language、Referer等关键标识，Glassdoor的反爬机制会判定这不是正常浏览器请求。
2. 压缩处理差异：curl的--compressed参数明确告知服务器支持gzip/deflate压缩，Requests虽默认支持，但Accept-Encoding头的默认值和curl不一致，触发服务器拦截。
3. 反爬特征识别：Glassdoor可能检测到Requests库的默认请求特征（比如Connection头的默认值），直接返回403拒绝请求。

1. 模拟完整浏览器请求头 #

补充浏览器常用请求头，尽可能和curl的请求头保持一致：

import requests

headers = {
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36',
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8',
    'Accept-Language': 'en-US,en;q=0.5',
    'Accept-Encoding': 'gzip, deflate, br',
    'Referer': 'https://www.glassdoor.com/',
    'Connection': 'keep-alive',
    'Upgrade-Insecure-Requests': '1',
    'TE': 'Trailers'
}

response = requests.get(
    'https://www.glassdoor.com/Job/scottsdale-az-stock-clerk-jobs-SRCH_IL.0,13_IC1133911_KO14,25.htm?src=GD_JOB_AD&rdserp=true&srs=EI_JOBS&jl=1008874519264&ao=1136043&s=21&guid=0000018ab3ad27ccbc470b1f6b275b17&pos=101&t=ESR&vt=w&uido=E2E7DBC4ACD391CABF382B9AEDAFF99C&ea=1&cs=1_f8a82cfa&cb=1695231584675&jobListingId=1008874519264&jrtk=5-yul1-0-1hapqqa33ir2a800-0383467bbae09c6a',
    headers=headers,
    allow_redirects=True,
    verify=True
)
print(response.status_code)

2. 先获取主页Cookie再请求目标URL #

Glassdoor可能依赖会话Cookie验证请求合法性，先请求主页获取Cookie，再发起目标请求：

import requests

session = requests.Session()
headers = {
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36',
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8',
    'Accept-Language': 'en-US,en;q=0.5',
    'Accept-Encoding': 'gzip, deflate, br',
    'Connection': 'keep-alive',
    'Upgrade-Insecure-Requests': '1'
}

# 先请求主页获取Cookie
session.get('https://www.glassdoor.com/', headers=headers)

# 再请求目标URL
response = session.get(
    'https://www.glassdoor.com/Job/scottsdale-az-stock-clerk-jobs-SRCH_IL.0,13_IC1133911_KO14,25.htm?src=GD_JOB_AD&rdserp=true&srs=EI_JOBS&jl=1008874519264&ao=1136043&s=21&guid=0000018ab3ad27ccbc470b1f6b275b17&pos=101&t=ESR&vt=w&uido=E2E7DBC4ACD391CABF382B9AEDAFF99C&ea=1&cs=1_f8a82cfa&cb=1695231584675&jobListingId=1008874519264&jrtk=5-yul1-0-1hapqqa33ir2a800-0383467bbae09c6a',
    headers=headers
)
print(response.status_code)

3. 完全复刻curl请求行为 #

可以将curl命令转换为Python Requests代码，确保所有请求参数（包括头、编码、连接方式）和curl完全一致，最大程度规避反爬检测。

内容的提问来源于stack exchange，提问作者nlblack323