Heroku上Redis随机出现aioredis.ConnectionError问题求助

错误现象 #

随机触发以下错误：

aioredis.exceptions.ConnectionError: Connection closed by server.

同时伴随二次异常：

RuntimeError: await wasn't used with future

完整报错栈 #

2023-03-25T23:34:34.116795+00:00 app[web.1]: There was an exception checking if the field exists in the hashmap: await wasn't used with future
2023-03-25T23:34:34.117607+00:00 app[web.1]: [2023-03-25 23:34:34,117] ERROR in app: Exception on /thelio_bot [POST]
2023-03-25T23:34:34.117608+00:00 app[web.1]: Traceback (most recent call last):
2023-03-25T23:34:34.117608+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.10/site-packages/aioredis/connection.py", line 1422, in get_connection
2023-03-25T23:34:34.117609+00:00 app[web.1]: if await connection.can_read():
2023-03-25T23:34:34.117609+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.10/site-packages/aioredis/connection.py", line 893, in can_read
2023-03-25T23:34:34.117609+00:00 app[web.1]: return await self._parser.can_read(timeout)
2023-03-25T23:34:34.117609+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.10/site-packages/aioredis/connection.py", line 479, in can_read
2023-03-25T23:34:34.117610+00:00 app[web.1]: raise ConnectionError(SERVER_CLOSED_CONNECTION_ERROR)
2023-03-25T23:34:34.117610+00:00 app[web.1]: aioredis.exceptions.ConnectionError: Connection closed by server.
2023-03-25T23:34:34.117611+00:00 app[web.1]: 
2023-03-25T23:34:34.117611+00:00 app[web.1]: During handling of the above exception, another exception occurred:
2023-03-25T23:34:34.117611+00:00 app[web.1]: 
2023-03-25T23:34:34.117612+00:00 app[web.1]: Traceback (most recent call last):
2023-03-25T23:34:34.117612+00:00 app[web.1]: File "/app/thelioapp/redis_factory.py", line 300, in field_exists
2023-03-25T23:34:34.117612+00:00 app[web.1]: hash_exists = await redis_conn.exists(self.redis_key)
2023-03-25T23:34:34.117613+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.10/site-packages/aioredis/client.py", line 1082, in execute_command
2023-03-25T23:34:34.117613+00:00 app[web.1]: conn = self.connection or await pool.get_connection(command_name, **options)
2023-03-25T23:34:34.117614+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.10/site-packages/aioredis/connection.py", line 1425, in get_connection
2023-03-25T23:34:34.117614+00:00 app[web.1]: await connection.disconnect()
2023-03-25T23:34:34.117614+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.10/site-packages/aioredis/connection.py", line 806, in disconnect
2023-03-25T23:34:34.117620+00:00 app[web.1]: await self._writer.wait_closed()  # type: ignore[union-attr]
2023-03-25T23:34:34.117620+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.10/asyncio/streams.py", line 343, in wait_closed
2023-03-25T23:34:34.117620+00:00 app[web.1]: await self._protocol._get_close_waiter(self)
2023-03-25T23:34:34.117620+00:00 app[web.1]: RuntimeError: await wasn't used with future

相关代码 #

触发错误的业务代码 #

async def update_hashmap(self):
    redis_conn = await self.get_redis_conn()

    try:
        await redis_conn.hset(self.redis_key, self.field, self.value)
    except Exception as e:
        print(f"There was an exception updating/adding to the hashmap: {e}")
        raise Exception

Redis连接池与客户端获取代码 #

executed_redis_pool = False
main_redis_pool = None
def create_redis_pool():
    global executed_redis_pool
    global main_redis_pool
    if not executed_redis_pool:
        executed_redis_pool=True
        main_redis_pool = aioredis.ConnectionPool.from_url(os.environ["REDIS_URI_PROD"])
    else:
        print("Redis pool already executed")

create_redis_pool()


async def get_redis_conn():
    redis_conn = await aioredis.Redis(connection_pool=main_redis_pool, health_check_interval=30, db=0, decode_responses=True)
    return redis_conn

问题分析 #

1. 连接池初始化错误：aioredis v2.x版本中，ConnectionPool.from_url是异步方法，同步调用会返回一个coroutine对象而非实际的连接池。后续使用这个无效的连接池创建客户端时，会导致底层异步操作异常，也就是RuntimeError: await wasn't used with future的根源。
2. 客户端重复创建：每次调用get_redis_conn都新建一个Redis客户端实例，没有复用连接池的优势，同时增加了连接管理的开销，容易触发Heroku Redis的连接限制或超时断开。
3. 连接断开未处理：Heroku Redis会自动断开长时间闲置的连接，虽然设置了health_check_interval=30，但由于连接池本身无效，健康检查无法正常工作，导致使用已断开的连接时触发ConnectionError。

解决方案 #

1. 正确异步初始化连接池 #

修改连接池初始化逻辑，确保异步执行：

import os
import aioredis

main_redis_pool = None

async def create_redis_pool():
    global main_redis_pool
    if main_redis_pool is None:
        main_redis_pool = await aioredis.ConnectionPool.from_url(
            os.environ["REDIS_URI_PROD"],
            health_check_interval=30,
            decode_responses=True
        )
    return main_redis_pool

2. 复用Redis客户端实例 #

避免每次创建新客户端，初始化一个全局客户端：

main_redis_client = None

async def get_redis_client():
    global main_redis_client
    if main_redis_client is None:
        pool = await create_redis_pool()
        main_redis_client = aioredis.Redis(connection_pool=pool)
    return main_redis_client

3. 修改业务代码使用复用的客户端 #

async def update_hashmap(self):
    redis_client = await self.get_redis_client()

    try:
        await redis_client.hset(self.redis_key, self.field, self.value)
    except aioredis.exceptions.ConnectionError:
        # 遇到连接错误时，重置客户端并重试一次
        global main_redis_client
        main_redis_client = None
        redis_client = await self.get_redis_client()
        await redis_client.hset(self.redis_key, self.field, self.value)
    except Exception as e:
        print(f"There was an exception updating/adding to the hashmap: {e}")
        raise

4. 应用启动时初始化Redis连接 #

在ASGI应用启动时（比如FastAPI的startup事件、Starlette的on_startup）调用create_redis_pool，确保连接池在应用开始处理请求前初始化完成：

# 以FastAPI为例
from fastapi import FastAPI

app = FastAPI()

@app.on_event("startup")
async def startup_event():
    await create_redis_pool()

补充说明 #

• 确保使用的aioredis版本是稳定的v2.x版本，避免版本兼容问题。
• Heroku Redis的连接字符串需要正确配置，包含密码、主机、端口等信息，可通过heroku config:get REDIS_URI_PROD确认。
• 增加日志记录，方便后续排查连接相关问题。

内容的提问来源于stack exchange，提问作者3awny