无需Selenium，如何用BeautifulSoup绕过JavaScript检测获取MarketWatch内容？

我用Python写了个脚本，想用requests和BeautifulSoup解析MarketWatch的经济日历页面：

#!/usr/bin/env python3
import requests
from bs4 import BeautifulSoup

def parse_marketwatch_calendar(url):
    #page=requests.get(url).text
    #soup=BeautifulSoup(page, "lxml")
    soup = BeautifulSoup(requests.get(url).text, 'html5lib')
    print(soup)
    title_parent = soup.find("div", class_="column column --primary")
    print(title_parent)
url = "https://www.marketwatch.com/economy-politics/calendar?mod=economy-politics"
parse_marketwatch_calendar(url)

运行后得到的响应是一段要求启用JS和关闭广告拦截的内容，目标元素也找不到：

<html><head><title>marketwatch.com</title><style>#cmsg{animation: A
1.5s;}@keyframes A{0%{opacity:0;}99%{opacity:0;}100%{opacity:1;}}</style></head><body style="margin:0"><p id="cmsg">Please enable JS and disable any ad blocker</p><script data-cfasync="false">var dd={'rt':'c','cid':'AHrlqAAAAAMA0byAgFOkDSoAGKNiPA==','hsh':'D428D51E28968797BC27FB9153435D','t':'bv','s':47891,'e':'a873fd745afc33d92e8f68dbadb4eaa3536c2c677a9785033316a437fb980f49','host':'geo.captcha-delivery.com'}</script><script data-cfasync="false" src="https://ct.captcha-delivery.com/c.js"></script></body></html> None

我不想切换到Selenium，只想用requests或urllib获取实际页面内容，想问光加支持JavaScript的User-Agent行不行？

• 仅替换User-Agent大概率没用：网站的反爬检测不止看User-Agent，它会校验请求是否带有浏览器环境的特征，比如JS执行痕迹、完整的请求头、Cookie等。你收到的是Cloudflare的人机验证响应，说明网站已经识别出你是爬虫。
• 先补全请求头模拟真实浏览器：除了User-Agent，还要添加Accept、Accept-Language、Referer等常见字段，示例：

  headers = {
      'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36',
      'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
      'Accept-Language': 'en-US,en;q=0.5',
      'Referer': 'https://www.marketwatch.com/',
      'DNT': '1',
      'Connection': 'keep-alive',
      'Upgrade-Insecure-Requests': '1'
  }
  response = requests.get(url, headers=headers)
  

• 用Session维护Cookie：网站可能会通过Cookie识别会话，用requests的Session对象先访问首页获取初始Cookie，再请求目标页面：

  session = requests.Session()
  session.headers.update(headers)
  # 先访问首页拿到Cookie
  session.get('https://www.marketwatch.com/')
  # 再请求日历页面
  response = session.get(url)
  

• 用轻量JS执行工具替代Selenium：如果网站必须执行JS才能返回完整内容，可以用requests-html（基于requests，支持JS执行，无需启动浏览器），示例：

  from requests_html import HTMLSession
  
  session = HTMLSession()
  session.headers.update(headers)
  response = session.get(url)
  response.html.render()  # 执行页面JS
  soup = BeautifulSoup(response.html.html, 'html5lib')
  

• 注意合规性：爬取前先查看网站的robots.txt文件，确认允许爬取目标内容，避免违规风险。

内容的提问来源于stack exchange，提问作者murkywaters