使用Python requests请求网站返回403错误，添加Header仍未解决

clutch.co这类商业目录站反爬机制较严格，只加几个基础请求头大概率无法通过校验，试试下面这些方案：

• 补全完整请求头
  浏览器发送的请求头远不止你用到的几个，把Network面板里Request Headers的所有字段（除Cookie中的个人敏感信息）都复制到HEADERS里，比如Accept、Accept-Encoding、Accept-Language等字段，网站会校验请求头的完整性。
  示例代码：

from bs4 import BeautifulSoup as bs
import requests

url = "https://clutch.co/us/agencies/digital-marketing"
HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
    "Accept-Encoding": "gzip, deflate, br",
    "Accept-Language": "zh-CN,zh;q=0.8,zh-TW;q=0.7,zh-HK;q=0.5,en-US;q=0.3,en;q=0.2",
    "Connection": "keep-alive",
    "Sec-Fetch-Dest": "document",
    "Sec-Fetch-Mode": "navigate",
    "Sec-Fetch-Site": "none",
    "Sec-Fetch-User": "?1",
    "Upgrade-Insecure-Requests": "1"
}
html = requests.get(url, headers=HEADERS)
print("RESULT:", html)

• 带上浏览器的Cookie
  部分网站会通过Cookie验证会话合法性，你可以从浏览器对应请求的Cookie字段复制内容，加到HEADERS里。注意Cookie有有效期，过期后需要重新从浏览器获取。

• 用Session维持会话
  单独的get请求容易被判定为异常请求，用requests.Session()模拟浏览器的会话状态，会自动维持Cookie等会话信息：

from bs4 import BeautifulSoup as bs
import requests

url = "https://clutch.co/us/agencies/digital-marketing"
HEADERS = {
    # 填入完整请求头
}
session = requests.Session()
session.headers.update(HEADERS)
response = session.get(url)
print("RESULT:", response)

• 尝试渲染型爬虫库
  如果网站有JS反爬逻辑，纯静态请求拿不到内容，可以用requests-html或者selenium这类能模拟浏览器渲染的工具，绕过JS检测。比如requests-html的用法：

from requests_html import HTMLSession

session = HTMLSession()
response = session.get(url)
response.html.render()  # 执行页面JS渲染
print("STATUS CODE:", response.status_code)

内容的提问来源于stack exchange，提问作者320V