Django StreamingHttpResponse本地正常生产环境无法流式输出求助

本地开发时，StreamingHttpResponse可正常流式返回数据，但部署到生产环境（Nginx反向代理Uvicorn）后，出现两种异常：部分接口一次性返回所有数据，数据量较大的接口完全无法返回数据。已尝试多种Nginx配置（如proxy_buffering off），问题仍未解决。

代码与配置详情 #

stream.py #

import json
from django.http import StreamingHttpResponse
from decimal import Decimal

class DecimalEncoder(json.JSONEncoder):
    def default(self, obj):        
        if isinstance(obj, Decimal):
            return str(obj)  # or float(obj)
        return super(DecimalEncoder, self).default(obj)

class StreamResponseMixin:   
    def stream_response(self, queryset, serializer_class, gen_message_func=None):
        def generate_stream():
            for item in queryset:
                serializer = serializer_class(item)
                yield self.gen_message(serializer.data, gen_message_func)

        response = StreamingHttpResponse(generate_stream(), content_type='text/event-stream')
        response['X-Accel-Buffering'] = 'no'  # Disable buffering in nginx
        response['Cache-Control'] = 'no-cache'  # Ensure clients don't cache the data
        response['Transfer-Encoding'] = 'chunked'
        return response

    def gen_message(self, data, gen_message_func=None):
        if gen_message_func:
            return gen_message_func(data)
        return '{}
'.format(json.dumps(data, cls=DecimalEncoder)) # Default behavior if no custom message generator is provided

Views.py #

class PackageViewSet(viewsets.ModelViewSet, StreamResponseMixin):
    queryset = Package.objects.all().order_by('-created_at')

    def get_serializer_class(self):
        if self.action in ['list', 'modified']:
            return PackageListSerializer
        return PackageDetailSerializer

    def get_queryset(self):
        user = self.request.user
        package_queryset = Package.objects.all().order_by('-created_at')

        package_queryset = package_queryset.select_related('customer', 'creator').prefetch_related(
            Prefetch('products', queryset=Product.objects.all()),
        )

        try:
            customer = Customer.objects.get(user=user)
            package_queryset = package_queryset.filter(customer=customer)
        except ObjectDoesNotExist:
            pass

        return package_queryset

    @action(detail=False, methods=['GET'], url_path='stream-packages')
    def get_stream(self, *args, **kwargs):
        packages = self.get_queryset().iterator(chunk_size=50)
        return self.stream_response(queryset=packages, serializer_class=PackageListSerializer)

nginx.conf #

server {
        listen 443 ssl;
        server_name abc.api.ai;

        ssl_certificate /etc/letsencrypt/live/abc.api.ai/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/abc.api.ai/privkey.pem;

        location / {
            if ($internal_ips = 0) {
                return 403;
            }
            proxy_pass http://abc-backend-staging:8000;
            include proxy.conf;
        }
        location /api/packages/stream-packages/ {
            proxy_pass http://abc-backend-staging:8000;
            include proxy.conf;
            proxy_http_version 1.1;
            proxy_set_header Connection "";
            chunked_transfer_encoding on;
            proxy_buffering off;
            proxy_request_buffering off;
        }
    }

proxy.conf #

proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto "https";

entrypoint.sh #

exec poetry run uvicorn myapplication.asgi:application --host 0.0.0.0 --port 8000

解决方案 #

1. 修复SSE消息格式错误 #

Server-Sent Events（SSE）要求每条消息以两个换行符结尾，否则客户端无法实时识别并处理消息。修改gen_message方法的返回值：

return '{}
\n\n'.format(json.dumps(data, cls=DecimalEncoder))

2. 禁用Uvicorn输出缓冲 #

Uvicorn默认会缓冲输出，需启动时添加--no-buffer参数强制实时发送每个数据块，同时补充代理相关配置确保反向代理兼容性：
修改entrypoint.sh：

exec poetry run uvicorn myapplication.asgi:application --host 0.0.0.0 --port 8000 --no-buffer --proxy-headers --forwarded-allow-ips='*'

3. 优化Nginx配置，确保路径优先匹配 #

当前Nginx的location /可能优先匹配请求（若URL末尾无斜杠），需调整流式接口的location规则，确保优先匹配，同时补充缓存禁用配置：

location ^~ /api/packages/stream-packages/ {
    proxy_pass http://abc-backend-staging:8000;
    include proxy.conf;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    chunked_transfer_encoding on;
    proxy_buffering off;
    proxy_request_buffering off;
    proxy_cache off;
    proxy_set_header Cache-Control no-cache;
}

^~标记会让Nginx优先匹配该路径，避免被location /拦截。

4. 排除流式接口的压缩中间件 #

若Django启用了GZipMiddleware等压缩中间件，会缓冲整个响应导致流式失效，需自定义中间件排除流式接口：

from django.http import StreamingHttpResponse
from django.middleware.gzip import GZipMiddleware

class ConditionalGZipMiddleware(GZipMiddleware):
    def process_response(self, request, response):
        # 对流式响应或指定路径跳过压缩
        if isinstance(response, StreamingHttpResponse) or request.path.startswith('/api/packages/stream-packages/'):
            return response
        return super().process_response(request, response)

在settings.py中替换原GZipMiddleware为自定义的ConditionalGZipMiddleware。

5. 优化数据库查询的流式特性 #

prefetch_related会一次性加载所有关联数据，可能打破流式效果。可暂时注释prefetch_related测试，若恢复正常，再优化关联查询：

• 改用select_related（仅针对外键关联）
• 在序列化器中延迟加载关联数据（需注意避免N+1查询问题）
• 调整iterator(chunk_size)的大小，根据数据量设置合理值

内容的提问来源于stack exchange，提问作者Lean Learner