Django 2.0.1中独立任务并行执行的问题求助

Hey, I’ve dealt with this exact problem before—Windows uses the spawn start method for multiprocessing by default, which means every new process re-imports your entire module from scratch. When that happens, the child process tries to import your Django models before Django’s app registry is fully loaded, hence the AppRegistryNotReady error. Here’s how to fix it:

1. Initialize Django in Child Processes #

The key is to manually set up the Django environment inside your worker tasks, before importing any models or accessing Django components.

Add this initialization code directly at the start of your task functions:

def process_file_task(file_path):
    # Set up Django environment first
    import os
    import django
    os.environ.setdefault("DJANGO_SETTINGS_MODULE", "your_project_name.settings")
    django.setup()

    # Now it's safe to import models and run your task logic
    from uploads.core.models import Document
    # Your actual task code here: analyze file, write results, etc.
    # ...
    return task_result

2. Avoid Module-Level Model Imports #

If your views.py has a module-level import like from uploads.core.models import Document, the child process will hit this import before Django is set up. Move those imports inside your view functions or worker tasks instead.

3. Call Multiprocessing Correctly in Views #

Make sure you create and start your processes inside your view function (not at the module level). For example:

from django.shortcuts import render
from multiprocessing import Pool

def upload_and_process(request):
    if request.method == 'POST' and request.FILES['file']:
        # Handle file upload first, save to disk or get file path
        uploaded_file = request.FILES['file']
        file_path = f"/path/to/save/{uploaded_file.name}"
        with open(file_path, 'wb+') as destination:
            for chunk in uploaded_file.chunks():
                destination.write(chunk)
        
        # Run parallel tasks
        with Pool(processes=2) as pool:
            # Assume you have two independent tasks to run
            results = pool.map(process_file_task, [file_path, file_path])
        
        # Merge results from parallel tasks
        merged_result = combine_results(results)
        
        # Render results to user
        return render(request, 'results.html', {'result': merged_result})
    return render(request, 'upload.html')

4. Important Notes #

• Windows Spawn Behavior: Unlike Unix systems (which use fork), Windows spawn starts a fresh Python interpreter, so you can’t rely on shared state from the parent process. All task data must be picklable to pass between processes.
• Why Celery Isn’t an Option: You mentioned needing real-time results, so waiting for multiprocessing tasks to finish synchronously makes sense—Celery is designed for async background jobs where you don’t wait immediately for results.

This setup will ensure each child process properly initializes Django before touching any models, eliminating the AppRegistryNotReady error while letting you run independent tasks in parallel.

内容的提问来源于stack exchange，提问作者Sunil Vikram