如何在asyncio中结合阻塞函数？附Django ORM场景选型疑问

Nice question! Mixing synchronous and asynchronous code can feel tricky at first, but breaking down the tradeoffs and aligning with your project's needs makes it easier to decide. Let’s walk through your two options and the general rules to guide your choice.

Let’s start by unpacking when each approach makes sense, with practical examples.

方案1：整体异步架构 + loop.run_in_executor #

This is the way to go if your project leans heavily into asynchronous logic—think using an async framework like FastAPI, or having lots of async IO tasks (like multiple API calls, message queue operations) alongside your Django ORM work.

How it works #

You offload the blocking Django ORM queries to a thread/process pool using loop.run_in_executor, which keeps your async event loop free to handle other tasks instead of getting stuck waiting for database responses.

Example code snippet #

import asyncio
import django
from django.conf import settings

# Manually initialize Django (required for non-Django projects)
settings.configure(
    DEBUG=True,
    DATABASES={
        'default': {
            'ENGINE': 'django.db.backends.sqlite3',
            'NAME': 'mydb.sqlite3',
        }
    },
    INSTALLED_APPS=['myapp']  # Replace with your app name
)
django.setup()

from myapp.models import MyModel

async def send_to_queue(obj):
    # Async message queue logic (e.g., using aio-pika or asyncio-redis)
    await queue.publish(obj.serialize())

async def main():
    loop = asyncio.get_running_loop()
    # Run blocking ORM query in a thread pool
    queryset = await loop.run_in_executor(None, MyModel.objects.all)
    
    # Process and send objects asynchronously
    tasks = [send_to_queue(obj) for obj in queryset]
    await asyncio.gather(*tasks)

asyncio.run(main())

Key considerations #

• Thread pool size: Don’t oversize your thread pool—too many threads can lead to resource contention. Start with a small number (matching your database connection pool size) and adjust based on load.
• Django initialization: Make sure you call django.setup() before using any ORM features.

方案2：整体同步架构 + asyncio.run() #

This is simpler if your project is mostly synchronous (most of your code uses blocking operations like Django ORM, and the async message queue is just a small part of the workflow).

How it works #

You keep your core logic synchronous, and use asyncio.run() to spin up a temporary event loop whenever you need to run async tasks (like sending to the message queue). This lets you leverage async IO without rewriting your entire codebase.

Example code snippet #

import asyncio
import django
from django.conf import settings

# Initialize Django
settings.configure(...)  # Same config as above
django.setup()

from myapp.models import MyModel

async def send_to_queue(obj):
    await queue.publish(obj.serialize())

def main():
    # Run blocking ORM query (sync code)
    queryset = MyModel.objects.all()
    
    # Bundle async tasks and run them with asyncio.run()
    tasks = [send_to_queue(obj) for obj in queryset]
    asyncio.run(asyncio.gather(*tasks))

if __name__ == "__main__":
    main()

Key considerations #

• No nested loops: asyncio.run() creates and closes an event loop each time you call it—you can’t nest it inside another async function or call it multiple times in a way that overlaps.
• Simplicity first: This approach keeps your code easy to read and debug, which is great if your team isn’t deeply familiar with async patterns.

Here’s a framework to decide which approach fits your project:

1. Follow your project’s core direction

   • If you plan to add more async features (async APIs, async third-party integrations) down the line, go with the async architecture + run_in_executor—it avoids costly refactoring later.
   • If your project will stay mostly sync (heavy ORM use, few async tasks), stick with sync architecture + asyncio.run() for simplicity.

2. Evaluate the ratio of sync vs async work

   • If async IO tasks (message queue, HTTP calls) make up most of your workload, async architecture will give you better throughput and resource utilization.
   • If blocking sync tasks (ORM, CPU-heavy work) are the majority, async won’t provide much benefit—thread pool overhead can even slow things down. Sync architecture is better here.

3. Consider team expertise

   • Async code has a steeper learning curve (event loops, coroutines, concurrency bugs). If your team is new to async, the sync approach will be easier to maintain.
   • If your team is comfortable with async patterns, the async architecture gives you more flexibility for future growth.

• Django ORM thread safety: Django handles database connections per thread, so using run_in_executor is safe—each thread gets its own connection. Just make sure your database connection pool is sized appropriately.
• Avoid blocking the event loop: If you go the async route, never call blocking functions directly in your async code—always offload them to run_in_executor.
• Process vs thread pools: For CPU-heavy tasks (not just ORM queries), use a process pool instead of a thread pool (since Python’s GIL limits thread performance for CPU work).

内容的提问来源于stack exchange，提问作者ospider