Django+Gunicorn(gevent)数据库持久连接无法复用问题分析求助

Let’s break down why you’re seeing excessive PostgreSQL connections even with CONN_MAX_AGE configured, and how to fix this issue.

The Core Conflict: Thread-Local Storage vs. Coroutines #

First, let’s unpack the key systems at play and where they clash:

1. Gunicorn Gevent Worker Behavior #

Gunicorn’s gevent workers use coroutines (via monkey-patching Python’s standard library) to handle multiple requests concurrently in a single thread. Unlike OS-managed threads, coroutines share the same thread’s resources and are scheduled internally by gevent. This means hundreds of coroutines can run in one worker thread, all sharing the exact same thread ID.

2. Django’s Connection Management #

Django stores database connections in ConnectionHandler()._connections, a threading.local() object. This object isolates data by thread ID—so each thread gets its own dedicated set of connections. When CONN_MAX_AGE is set, Django reuses unexpired connections within the same thread to avoid the overhead of creating new ones.

3. The Root Problem #

threading.local() has no awareness of coroutines. Since all gevent coroutines in a worker share the same thread ID, they all access the same threading.local() storage. Here’s what happens in practice:

• Coroutine A creates a connection and stores it in the thread-local space.
• Before Coroutine A finishes (or before its connection is eligible for reuse), Coroutine B is scheduled.
• Coroutine B tries to fetch a connection from the thread-local storage, but either:
  • It sees Coroutine A’s connection is still locked/in use by the database driver, so it creates a new one.
  • It reuses the connection, but concurrent coroutine access leads to race conditions or connection state leaks between requests.

Over time, each new coroutine ends up spawning its own connection, completely bypassing CONN_MAX_AGE and causing your PostgreSQL connection count to skyrocket.

Fixes and Recommendations #

Here are actionable steps to resolve the issue:

1. Disable Persistent Connections (Quick Fix) #

Set CONN_MAX_AGE = 0 in your Django settings. This tells Django to close the database connection immediately after each request. While this adds minor overhead from creating new connections per request, it eliminates the connection leakage caused by coroutines sharing thread-local storage.

2. Use a Gevent-Compatible Connection Pool #

Switch to a connection pool library built for gevent, like django-db-geventpool. This library replaces Django’s thread-local storage with gevent.local.local, which isolates connections per coroutine instead of per thread. It maintains proper connection reuse within each coroutine, keeping your PostgreSQL connection count under control.

To implement this:

• Install the package: pip install django-db-geventpool
• Update your database settings to use the gevent pool:

  DATABASES = {
      'default': {
          'ENGINE': 'django.db.backends.postgresql',
          'NAME': 'your_db_name',
          'USER': 'your_db_user',
          'PASSWORD': 'your_db_pass',
          'HOST': 'your_db_host',
          'PORT': '5432',
          'CONN_MAX_AGE': 6,
          'OPTIONS': {
              'MAX_CONNS': 20,  # Adjust based on your PostgreSQL max connections limit
          },
          'POOL_CLASS': 'django_db_geventpool.pool.GeventPool',
      }
  }
  

3. Tune Gunicorn Gevent Worker Settings #

Limit the number of concurrent coroutines per worker with the --worker-connections flag (default is 1000). Reducing this number (e.g., to 100-200) prevents a single worker from spawning too many connections. You can also adjust the number of worker processes, but ensure the total possible connections (workers × worker_connections) stays below PostgreSQL’s max_connections setting.

Example Gunicorn command:

gunicorn --worker-class=gevent --workers=4 --worker-connections=100 your_project.wsgi:application

4. Switch to Sync Workers (If Possible) #

If your traffic doesn’t demand the high concurrency of gevent workers, switch back to Gunicorn’s default synchronous workers (omit the --worker-class flag). Sync workers use one thread per request, so Django’s thread-local connection reuse works as intended, and CONN_MAX_AGE will correctly pool connections.

内容的提问来源于stack exchange，提问作者Jayden Foo