Requests与AIOHTTP构造HTTP请求的差异及403问题排查

问题描述 #

我正在学习Python异步编程，使用Requests模块发送指定GET请求可返回200状态码，但用AIOHTTP构造相同请求时却得到403状态码。已对比请求头并手动添加Requests自动生成的Host头，但问题仍未解决，需要了解两者构造HTTP请求的差异及进一步排查方向。

测试代码 #

import aiohttp
import asyncio
import requests

url = "https://spectate-web.888sport.com/spectate/sportsbook-req/getTournamentMatches/ice-hockey/united-states-of-america/nhl"

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.79 Safari/537.36', 
    'Accept-Encoding': 'gzip, deflate', 
    'Accept': '*/*', 
    'Connection': 'keep-alive', 
    'Cookie': 'odds_format=DECIMAL; optimizelyEndUserId=oeu1712128176635r0.5640445223678774; _cs_c=0; FPID=FPID2.2.p9QFQEw3wxQgGubYSYQTLEyXHhaPlG6hnxGm5JhFQZs%3D.1712128177; OptanonAlertBoxClosed=2024-04-03T07:09:47.833Z; _gcl_au=1.1.303053898.1712128188; OptanonConsent=isGpcEnabled=0&datestamp=Wed+Apr+03+2024+00%3A09%3A47+GMT-0700+(Pacific+Daylight+Time)&version=202209.2.0&isIABGlobal=false&hosts=&consentId=9f33e415-6ad2-41d4-9ce8-5aa5d4b9c619&interactionCount=1&landingPath=NotLandingPage&groups=C0001%3A1%2CC0002%3A1%2CC0003%3A1%2CC0004%3A1; FPAU=1.2.1880634315.1712128189; __qca=P0-1150796283-1712128188063; blueID=b6302533-2e9a-4ce6-8220-9c82c65df18a; _fbp=fb.1.1712128192008.2093727263; _scid=fd08dc38-6af8-4912-d93b-898a1a94bc17; login_deviceuuid=37d22c5f-2f3e-4dff-a627-879c9dc45870; anon_hash=05c6744a8af2029e8018973278168195; spectate_session=780bc31d-1423-4820-955e-2c1bff1e598a%3Aanon; 888Attribution=1; 888Cookie=lang%3Den%26OSR%3D1927680; 888TestData=%7B%22strategysource%22%3A%22previousvisit%22%2C%22currentvisittype%22%3A%22Unknown%22%2C%22referrer%22%3A%22https%3A%2F%2Fwww.google.com%2F%22%2C%22orig-lp%22%3A%22https%3A%2F%2Fwww.888sport.com%2Fbaseball%2Funited-states-of-america%2Fmajor-league-baseball-t-324139%2F%22%2C%22publisher%22%3A%22SearchEngine%22%2C%22strategy%22%3A%22UnknownStrategy%22%2C%22datecreated%22%3A%222024-05-09T18%3A46%3A37.924Z%22%2C%22expiredat%22%3A%22Thu%2C%2016%20May%202024%2018%3A46%3A00%20GMT%22%2C%22datemodified%22%3A%222024-05-13T23%3A59%3A03.232Z%22%2C%22modifiedcounter%22%3A%224%22%7D; spectate_client_ver=2.71; bbsess=zi4-oo0BBzStlX6jecq76Ohha-x; lang=enu; _cs_cvars=%7B%7D; _gid=GA1.2.388250393.1715644745; _gat_UA-125725186-7=1; FPGSID=1.1715644745.1715644745.G-15CNXYXJ7R.lERb7naix8ZqN0n-RV2TSA; FPLC=os%2FV%2FxiV%2BsUjh9NAS1aCO6Z7OGcyMwfxMEOAXXjQ8fFn2TwmxnMIS3TNIde0PK%2B5Exqd5ou1ztzKH5uqLKIXj9Yt%2Bk7DCeFKK7ni9z%2Bx1LCHs%2FzC6WTKZkn4EoIUjQ%3D%3D; _cs_id=8ce8877b-5380-a03e-ec85-8434a4d28294.1712128178.29.1715644748.1715644744.1708336297.1746292178098.1; _cs_s=2.5.0.1715646548363; _uetsid=c6fa0430118411ef9efc39c590568d56; _uetvid=5ac21390f2fd11ee87406f760e46580c; _ga=GA1.2.957392554.1712128177; _ga_15CNXYXJ7R=GS1.1.1715644744.30.1.1715644756.0.0.1631588786'
}

resp = requests.get(url=url, headers=headers)
print(resp.status_code)
#Output 200

async def main():
    async with aiohttp.ClientSession() as session:
        async with session.get(url, headers=headers) as response:
            print(response.status)

asyncio.run(main())
#Output 403

排查方向 #

• 校验请求头细节差异：
  Requests与aiohttp处理请求头的逻辑存在区别，比如Accept-Encoding字段，Requests会自动适配服务器编码，而aiohttp可能需要更精确的设置；部分服务器会严格校验请求头顺序，两者默认的头字段排序可能不同，这也可能触发403。可以通过抓包工具对比两次请求的原始HTTP报文，确认所有头字段的顺序、大小写、取值是否完全一致。

• 调整Cookie传递方式：
  不要在headers中携带Cookie，改用aiohttp的cookies参数单独传入键值对格式的Cookie。示例：

  cookies = {
      'odds_format': 'DECIMAL',
      'optimizelyEndUserId': 'oeu1712128176635r0.5640445223678774',
      # 拆分所有Cookie键值对...
  }
  async with session.get(url, headers=headers, cookies=cookies) as response:
  

  这种方式能避免aiohttp对Cookie头的解析异常。

• 模拟完整会话流程：
  部分服务器会校验会话的前置请求（比如先访问首页获取初始Cookie），而非直接接受目标接口请求。可以先用aiohttp访问目标域名的首页，再发送目标请求，模拟浏览器的访问顺序。

• 检查SSL/TLS配置：
  Requests基于urllib3，aiohttp基于asyncio的SSL库，两者的SSL默认配置可能不同。可以临时在aiohttp请求中禁用SSL验证（仅测试用）：

  async with session.get(url, headers=headers, ssl=False) as response:
  

  如果返回200，说明是SSL握手问题，需要调整aiohttp的SSL参数（比如指定SSL版本、加密套件）。

• 验证User-Agent是否生效：
  虽然设置了自定义User-Agent，但aiohttp可能存在覆盖逻辑。可以通过抓包或在请求中打印实际发送的头信息，确认User-Agent是否与Requests完全一致。

内容的提问来源于stack exchange，提问作者Kevin O