如何用requests获取目标网站HTML？遇Signal-Not Acceptable问题求助

网站的反爬机制识别出你的请求来自脚本而非真实浏览器，所以返回了拦截响应。不用selenium的话，通过给requests添加合适的请求头就能绕过这个检测。

修改后的代码 #

import requests
from bs4 import BeautifulSoup

url = "https://www.thespruce.com/christmas-village-display-ideas-8407777"
# 添加模拟真实浏览器的请求头
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36"
}
page = requests.get(url, headers=headers) 
soup = BeautifulSoup(page.content, 'lxml')
print(soup.prettify())

进阶优化（若上述代码仍无效） #

可以添加更多完整的浏览器请求头，模拟更真实的请求：

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.5",
    "Accept-Encoding": "gzip, deflate, br",
    "Connection": "keep-alive",
    "Upgrade-Insecure-Requests": "1"
}

内容的提问来源于stack exchange，提问作者Rapid1898