添加SQLAlchemy+oracledb后Gunicorn Worker频繁终止问题求助

我的FastAPI应用此前长期运行正常，近期新增SQLAlchemy连接Oracle数据库后，本地运行一切正常，但在Docker+K8s环境下用Gunicorn部署时出现异常：部分Worker持续被终止，部分仍可处理请求，甚至整个镜像会被杀死重启。正常版本与问题版本的唯一差异是新增了SQLAlchemy和oracledb库，Gunicorn配置及Dockerfile均未改动。

Dockerfile #

FROM python:3.10

EXPOSE 8000

WORKDIR /app
RUN mkdir -p /app
COPY . /app

RUN pip install poetry
RUN poetry config virtualenvs.create false
RUN poetry install

COPY certs/ /usr/local/share/ca-certificates
RUN update-ca-certificates

ENV REQUESTS_CA_BUNDLE "/etc/ssl/certs/ca-certificates.crt"

CMD [ "gunicorn", "wsgi:app", "--bind", "0.0.0.0:8000", "-w 4", "--access-logfile", "-", "--log-level", "debug", "-k uvicorn.workers.UvicornWorker", "-t 90"]

SQLAlchemy Engine 创建代码 #

from sqlalchemy import Engine, create_engine

import config


def get_engine() -> Engine:
    connection_url = f"oracle+oracledb://{config.DB_USERNAME}:{config.DB_PASSWORD}@{config.DB_HOST}:{config.DB_PORT}/?service_name={config.DB_SERVICE_NAME}"

    engine = create_engine(connection_url)
    return engine

Gunicorn 异常日志 #

[2023-02-13 00:02:27 +0000] [1] [WARNING] Worker with pid 203 was terminated due to signal 9
[2023-02-13 00:02:27 +0000] [207] [INFO] Booting worker with pid: 207
[2023-02-13 00:02:35 +0000] [205] [INFO] Started server process [205]
[2023-02-13 00:02:35 +0000] [205] [INFO] Waiting for application startup.
[2023-02-13 00:02:35 +0000] [205] [INFO] Application startup complete.
some-ip:some-port - "GET /status HTTP/1.1" 200
some-ip:some-port - "GET /status HTTP/1.1" 200
[2023-02-13 00:02:38 +0000] [1] [WARNING] Worker with pid 197 was terminated due to signal 9
[2023-02-13 00:02:38 +0000] [209] [INFO] Booting worker with pid: 209
[2023-02-13 00:02:45 +0000] [207] [INFO] Started server process [207]
[2023-02-13 00:02:45 +0000] [207] [INFO] Waiting for application startup.
[2023-02-13 00:02:45 +0000] [207] [INFO] Application startup complete.
some-ip:some-port - "GET /status HTTP/1.1" 200
some-ip:some-port - "GET /status HTTP/1.1" 200
[2023-02-13 00:02:47 +0000] [1] [WARNING] Worker with pid 205 was terminated due to signal 9
[2023-02-13 00:02:47 +0000] [212] [INFO] Booting worker with pid: 212
[2023-02-13 00:02:55 +0000] [209] [INFO] Started server process [209]
[2023-02-13 00:02:55 +0000] [209] [INFO] Waiting for application startup.
[2023-02-13 00:02:55 +0000] [209] [INFO] Application startup complete.
some-ip:some-port - "GET /status HTTP/1.1" 200
some-ip:some-port - "GET /status HTTP/1.1" 200
[2023-02-13 00:02:57 +0000] [1] [WARNING] Worker with pid 207 was terminated due to signal 9
[2023-02-13 00:02:57 +0000] [214] [INFO] Booting worker with pid: 214
[2023-02-13 00:03:04 +0000] [212] [INFO] Started server process [212]
[2023-02-13 00:03:04 +0000] [212] [INFO] Waiting for application startup.
[2023-02-13 00:03:04 +0000] [212] [INFO] Application startup complete.
some-ip:some-port - "GET /status HTTP/1.1" 200
[2023-02-13 00:03:06 +0000] [1] [WARNING] Worker with pid 209 was terminated due to signal 9
[2023-02-13 00:03:06 +0000] [217] [INFO] Booting worker with pid: 217
[2023-02-13 00:03:14 +0000] [214] [INFO] Started server process [214]
[2023-02-13 00:03:14 +0000] [214] [INFO] Waiting for application startup.
[2023-02-13 00:03:14 +0000] [214] [INFO] Application startup complete.
some-ip:some-port- "GET /status HTTP/1.1" 200
some-ip:some-port - "GET /status HTTP/1.1" 200
[2023-02-13 00:03:17 +0000] [1] [WARNING] Worker with pid 212 was terminated due to signal 9
[2023-02-13 00:03:17 +0000] [219] [INFO] Booting worker with pid: 219
[2023-02-13 00:03:25 +0000] [217] [INFO] Started server process [217]
[2023-02-13 00:03:25 +0000] [217] [INFO] Waiting for application startup.
[2023-02-13 00:03:25 +0000] [217] [INFO] Application startup complete.
some-ip:some-port - "GET /status HTTP/1.1" 200
some-ip:some-port - "GET /status HTTP/1.1" 200

此前Docker镜像会因未就绪在约1分钟后被杀死，将Gunicorn超时从默认30秒改为90秒后有所改善：部分Worker可运行处理请求，但仍有Worker持续被终止。已将日志级别调至debug，未获得有效信息。

排查思路与解决方案 #

1. 优先排查OOM（内存不足）问题 #

日志中Worker被终止的信号是signal 9，这是Linux的强制终止信号，最常见原因是进程占用内存超过容器/节点限制被OOM Killer杀死：

• 检查K8s Pod的资源限制（resources.limits.memory），确认是否过小，可临时调高内存限制测试；
• 在容器内运行top或ps aux查看进程内存占用，观察Worker进程是否在启动或运行中内存飙升；
• 查看节点上的dmesg日志，确认是否有OOM Killer触发记录。

2. 修正oracledb的初始化时机 #

Gunicorn多Worker模式下，若Engine在Worker启动前（主进程）被初始化，会导致多Worker共享同一个数据库连接对象，引发资源冲突或内存泄漏：

• 避免在模块级别直接调用get_engine()，改为在接口请求时延迟初始化，或使用FastAPI的依赖注入（Depends）在每个Worker中独立创建：

  from sqlalchemy import Engine, create_engine
  from fastapi import Depends
  import config
  
  def get_engine() -> Engine:
      connection_url = f"oracle+oracledb://{config.DB_USERNAME}:{config.DB_PASSWORD}@{config.DB_HOST}:{config.DB_PORT}/?service_name={config.DB_SERVICE_NAME}"
      return create_engine(connection_url)
  
  # 在接口中通过Depends获取独立的Engine实例
  @app.get("/db-data")
  def get_db_data(engine: Engine = Depends(get_engine)):
      with engine.connect() as conn:
          # 执行数据库操作
          pass
  

3. 调整Gunicorn Worker配置 #

• 减少Worker数量：当前设置-w 4，若容器内存有限，过多Worker会导致内存竞争，可尝试改为-w 2测试；
• 启用Worker自动重启：添加--max-requests 1000和--max-requests-jitter 100，让Worker处理一定请求后自动重启，避免内存泄漏累积；
• 延长优雅退出时间：添加--graceful-timeout 30，给Worker足够时间完成当前请求并释放资源。

4. 排查oracledb环境依赖问题 #

• 切换oracledb模式：显式指定使用Thin模式，避免Thick模式对Oracle客户端库的依赖：

  engine = create_engine(connection_url, thick_mode=False)
  

• 验证数据库连接参数：确认K8s环境中DB_HOST、DB_PORT、DB_SERVICE_NAME等配置正确，检查网络连通性（如防火墙、Service配置）。

5. 增强日志排查能力 #

• 开启SQLAlchemy引擎日志：记录数据库连接的创建、销毁过程，便于定位异常：

  import logging
  logging.basicConfig()
  logging.getLogger('sqlalchemy.engine').setLevel(logging.INFO)
  

• 重定向Worker错误日志：修改Gunicorn启动命令，添加--error-logfile -将Worker错误输出到控制台。

内容的提问来源于stack exchange，提问作者Cysio