为何Python请求带?p=quarterly的URL会被重定向，手动访问正常？

问题根源 #

网站的反爬机制会检测请求的合法性，仅设置User-Agent不足以绕过验证，请求会被重定向到无参数的年度数据页面，导致?p=quarterly参数丢失。

可行解决方案 #

• 补充完整请求头：模拟真实浏览器的请求头，增加Referer、Accept-Language等字段，提升请求合法性：

  headers = {
      'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36',
      'Referer': f'https://stockanalysis.com/stocks/{ticker}/',
      'Accept-Language': 'en-US,en;q=0.9',
      'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8'
  }
  

• 使用Session维持会话：网站会通过Cookie验证会话状态，用requests.Session()自动处理Cookie，先访问个股主页建立会话再请求财务数据：

  session = requests.Session()
  session.headers.update(headers)
  # 先访问个股主页初始化会话
  session.get(f'https://stockanalysis.com/stocks/{ticker}/')
  response = session.get(url)
  

• 手动处理重定向：禁止自动跟随重定向，检查响应状态码，手动将参数拼接至重定向后的URL：

  response = requests.get(url, headers=headers, allow_redirects=False)
  if response.status_code == 302:
      redirected_url = response.headers.get('Location')
      if redirected_url and '?p=quarterly' not in redirected_url:
          redirected_url += '?p=quarterly'
          response = requests.get(redirected_url, headers=headers)
  

• 抓取页面内的季度切换链接：先爬取年度数据页面，找到页面中切换季度数据的按钮链接，直接请求该链接：

  initial_url = f"https://stockanalysis.com/stocks/{ticker}/financials/"
  response = session.get(initial_url)
  soup = BeautifulSoup(response.content, 'html.parser')
  quarterly_link = soup.find('a', href=lambda x: x and 'quarterly' in x)
  if quarterly_link:
      quarterly_url = f"https://stockanalysis.com{quarterly_link['href']}"
      response = session.get(quarterly_url)
  

调整后的完整代码 #

import requests
from bs4 import BeautifulSoup
import pandas as pd

tickers = ["AMZN", "FB", "TSLA"]

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36',
    'Accept-Language': 'en-US,en;q=0.9',
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8'
}

session = requests.Session()
session.headers.update(headers)

for ticker in tickers:
    # 初始化会话
    session.get(f'https://stockanalysis.com/stocks/{ticker}/')
    
    url = f"https://stockanalysis.com/stocks/{ticker}/financials/?p=quarterly"
    print(url)
    response = session.get(url)

    soup = BeautifulSoup(response.content, 'html.parser')
    table = soup.find('table')

    if not table:
        print(f"未找到{ticker}的季度数据表格")
        continue

    df = pd.read_html(str(table))[0]
    columns = df.columns[1:4]
    row = df[df.iloc[:,0].str.contains('Revenue', na=False)]

    if row.empty:
        print(f"未找到{ticker}的营收数据")
        continue

    values = row[columns].values[0]
    print(f"For {ticker}:")
    print(f"Revenue for last quarter ({columns[0]}) is: {values[0]}")
    print(f"Revenue for 2 quarters ago ({columns[1]}) is: {values[1]}")
    print(f"Revenue for 3 quarters ago ({columns[2]}) is: {values[2]}\n")

内容的提问来源于stack exchange，提问作者LoafOfKevs