使用aiohttp发GET请求报Expecting value错误，为何requests.get可正常返回JSON？

问题现象 #

调用https://data.similarweb.com/api/v1/data?domain=httpbin.org接口时，aiohttp抛出Expecting value: line 1 column 1 (char 0)错误，但使用requests.get可正常获取JSON数据；经校验，两者请求头仅X-Amzn-Trace-Id值存在差异。

问题原因 #

该错误本质是响应内容无法被解析为JSON，可能是以下原因导致：

• aiohttp的自动编码/JSON解析逻辑与requests存在差异，导致无法正确识别响应编码
• 接口对请求头的隐性校验（如缺失Accept、Accept-Encoding等requests默认携带的头），返回了非JSON内容（如空串、HTML错误页）

解决方案 #

1. 先排查响应内容 #

修改代码先打印响应状态码和内容，明确错误根源：

import aiohttp
import asyncio
import nest_asyncio

nest_asyncio.apply()

async def fetch(session, url):
    try:
        async with session.get(url, headers=headers) as response:
            print(f"状态码: {response.status}")
            content = await response.text()
            print(f"响应内容片段: {content[:500]}")
            return await response.json(content_type=None)
    except Exception as e:
        print(f"请求错误: {str(e)}")

async def fetch_all(urls):
    async with aiohttp.ClientSession() as session:
        tasks = [fetch(session, url) for url in urls]
        return await asyncio.gather(*tasks, return_exceptions=True)

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36',
}

urls = ['https://data.similarweb.com/api/v1/data?domain=httpbin.org']
responses = asyncio.run(fetch_all(urls))

2. 修正请求逻辑 #

如果排查后确认是请求头或解析逻辑问题，使用以下优化代码：

import aiohttp
import asyncio
import nest_asyncio
import json

nest_asyncio.apply()

async def fetch(session, url):
    try:
        async with session.get(url, headers=headers) as response:
            response.raise_for_status()  # 捕获HTTP错误状态码
            # 手动处理编码，避免自动解析偏差
            raw_content = await response.read()
            text_content = raw_content.decode('utf-8')
            return json.loads(text_content)
    except Exception as e:
        print(f"请求错误: {str(e)}")
        # 错误时打印完整响应内容排查
        try:
            error_content = await response.text()
            print(f"错误响应内容: {error_content}")
        except:
            pass
        return None

async def fetch_all(urls):
    async with aiohttp.ClientSession() as session:
        tasks = [fetch(session, url) for url in urls]
        return await asyncio.gather(*tasks, return_exceptions=True)

# 补充requests默认携带的请求头，贴近标准浏览器请求
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36',
    'Accept': '*/*',
    'Accept-Encoding': 'gzip, deflate, br'
}

urls = ['https://data.similarweb.com/api/v1/data?domain=httpbin.org']
responses = asyncio.run(fetch_all(urls))
print(responses)

关键优化点 #

• 添加response.raise_for_status()，主动抛出4xx/5xx HTTP错误，避免解析错误内容
• 手动读取二进制内容再解码，绕过aiohttp自动编码检测的潜在问题
• 补充Accept、Accept-Encoding等头，让aiohttp请求更贴近requests的默认行为，规避接口隐性校验

内容的提问来源于stack exchange，提问作者Neret