Django+JQuery Ajax批量上传大量DICOM文件失败求助

嘿，我看到你在上传14000个DICOM文件时遇到了网站卡顿的问题，小批量（200个）正常运行，这其实是典型的大批量文件一次性上传导致的前端内存溢出+后端同步IO阻塞问题。咱们一步步来拆解解决：

一、前端优化：分片分批上传 #

一次性把14000个文件塞进FormData会瞬间占用大量浏览器内存，导致页面卡顿甚至崩溃。我们改成分批上传，比如每次传100个文件，降低单请求的资源压力：

修改你的JQuery代码，替换原表单提交逻辑：

$(document).ready(function(){
    $("#my-file").on('change',function(e){
        $("#file-wrap p").text('Now click on Upload button');
    });

    // 定义分批上传参数
    const BATCH_SIZE = 100; // 每批上传100个文件
    let totalFiles = 0;
    let uploadedFiles = 0;
    let fileBatches = [];
    let directoriesMap = {};

    $("#my-form").on('submit',function(e){
        e.preventDefault();
        const files = document.querySelector("#my-file").files;
        totalFiles = files.length;
        uploadedFiles = 0;

        // 先收集所有文件的相对路径映射
        for (const file of files) {
            directoriesMap[file.name] = file.webkitRelativePath;
        }

        // 将文件拆分成批次
        fileBatches = [];
        for (let i = 0; i < files.length; i += BATCH_SIZE) {
            fileBatches.push(Array.from(files).slice(i, i + BATCH_SIZE));
        }

        // 递归上传每一批
        uploadNextBatch();
    });

    function uploadNextBatch() {
        if (fileBatches.length === 0) {
            // 所有批次上传完成
            $('#my-form')[0].reset();
            $("#file-wrap p").html('Drag and drop file here');
            $('.myprogress').text('100%');
            $('.myprogress').css('width', '100%');
            alert('All files uploaded successfully!');
            return;
        }

        const currentBatch = fileBatches.shift();
        const formData = new FormData();

        // 添加当前批次的文件
        currentBatch.forEach(file => {
            formData.append('file', file);
        });

        // 添加目录映射和CSRF token
        formData.append('directories', JSON.stringify(directoriesMap));
        formData.append('csrfmiddlewaretoken', '{{ csrf_token }}');

        $.ajax({
            headers: {
                "X-CSRFToken": '{{ csrf_token }}'
            },
            type: 'POST',
            url: "{% url 'fileupload:upload_scan' %}",
            data: formData,
            cache: false,
            contentType: false,
            processData: false,
            timeout: 30000, // 添加超时设置
            xhr: function () {
                const xhr = new window.XMLHttpRequest();
                xhr.upload.addEventListener("progress", function (e) {
                    if (e.lengthComputable) {
                        // 计算全局上传进度
                        const batchProgress = e.loaded / e.total;
                        const globalPercent = Math.round((uploadedFiles + currentBatch.length * batchProgress) / totalFiles * 100);
                        $('.myprogress').text(globalPercent + '%');
                        $('.myprogress').css('width', globalPercent + '%');
                    }
                }, false);
                return xhr;
            },
            success: function(getResult) {
                uploadedFiles += currentBatch.length;
                // 继续上传下一批
                uploadNextBatch();
            },
            error: function(xhr, status, error) {
                console.error('Batch upload failed:', error);
                alert(`Failed to upload batch. Error: ${error}`);
            }
        });
    }
});

二、后端优化：异步处理+IO操作精简 #

后端同步处理每个文件的写入会长时间阻塞请求，导致网站资源被占用卡顿。我们用Django异步视图处理文件写入，同时优化目录创建逻辑：

修改views.py：

import asyncio
import os
import json
from django.http import HttpResponse
from django.views import View
from django.conf import settings
from django.shortcuts import render
from .forms import create_scan_form # 请确保表单路径正确

async def handle_uploaded_file_async(f, file_path):
    # 合并目录判断与创建，简化IO操作
    os.makedirs(file_path, exist_ok=True)
    full_path = os.path.join(file_path, f.name)
    with open(full_path, 'wb+') as destination:
        for chunk in f.chunks():
            destination.write(chunk)
    await asyncio.sleep(0) # 主动让出事件循环，避免阻塞

class UploadScanView(View):
    async def get(self, request):
        form = create_scan_form()
        return render(request, 'fileupload/basic_upload/scan_upload.html', {'form': form})

    async def post(self, request):
        form = create_scan_form(request.POST)
        if form.is_valid():
            scan = form.save(commit=False)
            scan.project_id = form.cleaned_data.get('project_id')
            scan.save() # 先保存scan对象，避免异步操作丢失数据
            
            directory_name = request.POST.get('directories')
            json_to_dictionary = json.loads(directory_name)
            
            # 收集所有文件任务，控制并发数避免IO过载
            tasks = []
            for upload_file in request.FILES.getlist('file'):
                relative_path = json_to_dictionary[upload_file.name]
                file_path = os.path.join(settings.MEDIA_ROOT, os.path.dirname(relative_path))
                tasks.append(handle_uploaded_file_async(upload_file, file_path))
            
            # 限制并发数为20，平衡上传速度与系统压力
            semaphore = asyncio.Semaphore(20)
            async def bounded_task(task):
                async with semaphore:
                    await task
            
            await asyncio.gather(*[bounded_task(t) for t in tasks])
        
        return HttpResponse("Batch uploaded successfully") # 返回极简响应，减少渲染耗时

同步URL配置 #

更新urls.py指向新的类视图：

# urls.py
from django.urls import path
from .views import UploadScanView

app_name = 'fileupload'
urlpatterns = [
    path('upload_scan/', UploadScanView.as_view(), name='upload_scan'),
]

三、额外配置建议 #

• 服务器调整：增大Nginx/Apache的client_max_body_size，避免大批次上传被拒绝；
• Django配置：调整FILE_UPLOAD_MAX_MEMORY_SIZE，让大文件自动写入临时文件，避免内存溢出；
• 重试机制：可以给前端失败的批次添加自动重试逻辑，提升上传成功率。

这样修改后，前端分批上传不会占用过多内存，后端异步处理文件写入不会阻塞请求，网站就不会再卡顿了。

内容的提问来源于stack exchange，提问作者reasm001