使用Ktor发送大文件触发OOM错误，求分块上传解决方案

我按照Ktor官方文档的方式发送本地90MB文件，代码如下：

val client = HttpClient(CIO)

val response: HttpResponse = client.post("http://localhost:8080/upload") {
    setBody(MultiPartFormDataContent(
        formData {
            append("description", "Ktor logo")
            append("image", File("ktor_logo.png").readBytes(), Headers.build {
                append(HttpHeaders.ContentType, "image/png")
                append(HttpHeaders.ContentDisposition, "filename=\"ktor_logo.png\"")
            })
        },
        boundary = "WebAppBoundary"
    )
    )
    onUpload { bytesSentTotal, contentLength ->
        println("Sent $bytesSentTotal bytes from $contentLength")
    }
}

运行后触发OutOfMemoryError错误，错误栈如下：

FATAL EXCEPTION: DefaultDispatcher-worker-2
    PID: 8152
    java.lang.OutOfMemoryError: Failed to allocate a 4112 byte allocation with 1424 free bytes and 1424B until OOM, max allowed footprint 100663296, growth limit 100663296
        at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:54)
        at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:49)
        at java.nio.ByteBuffer.allocate(ByteBuffer.java:281)
        at io.ktor.utils.io.bits.DefaultAllocator.alloc-gFv-Zug(MemoryFactoryJvm.kt:41)
        at io.ktor.utils.io.core.DefaultBufferPool.produceInstance(BufferFactory.kt:58)
        at io.ktor.utils.io.core.DefaultBufferPool.produceInstance(BufferFactory.kt:51)
        at io.ktor.utils.io.pool.DefaultPool.borrow(DefaultPool.kt:51)
        at io.ktor.utils.io.core.internal.ChunkBuffer$Companion$Pool$1.borrow(ChunkBuffer.kt:133)
        at io.ktor.utils.io.core.internal.ChunkBuffer$Companion$Pool$1.borrow(ChunkBuffer.kt:128)
        at io.ktor.utils.io.core.Output.appendNewChunk(Output.kt:102)
        at io.ktor.utils.io.core.Output.prepareWriteHead(Output.kt:354)
        at io.ktor.utils.io.core.internal.UnsafeKt.prepareWriteHead(Unsafe.kt:57)
        at io.ktor.utils.io.ByteBufferChannel.readRemainingSuspend(ByteBufferChannel.kt:3689)
        at io.ktor.utils.io.ByteBufferChannel.access$readRemainingSuspend(ByteBufferChannel.kt:24)
        at io.ktor.utils.io.ByteBufferChannel$readRemainingSuspend$1.invokeSuspend(Unknown Source:16)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
        at kotlinx.coroutines.internal.LimitedDispatcher.run(LimitedDispatcher.kt:42)
        at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:95)
        at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:570)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:677)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:664)
        Suppressed: kotlinx.coroutines.DiagnosticCoroutineContextException: [StandaloneCoroutine{Cancelling}@29cc6a1, Dispatchers.IO]

我认为问题出在readBytes()方法，尝试使用appendInput方式上传，代码如下：

appendInput(
        "MY_KEY",
        Headers.build {
            headers.forEach { h ->
                this.append(h.first, h.second) // Content-Disposition and Content-Type here
            }
        },
        file.length()
    ){
        file.inputStream().asInput()
    }

仍出现相同错误，我的HttpClient配置如下：

HttpClient(OkHttp) {
    install(ContentNegotiation) {
        json(Json {
            prettyPrint = true
            isLenient = true
            ignoreUnknownKeys = true
        })
    }

    install(Resources)
    install(Logging) {
        logger = Logger.DEFAULT
        level = LogLevel.BODY
    }    

    engine {
        config {
            followRedirects(true)
            connectTimeout(100_000, TimeUnit.MILLISECONDS)
            readTimeout(100_000, TimeUnit.MILLISECONDS)
        }
    }
}

解决方案 #

核心问题分析 #

OOM的本质是一次性将整个大文件加载到内存，不管是readBytes()直接读全量字节，还是你尝试的appendInput写法，都因为Logging插件的LogLevel.BODY设置导致Ktor把整个请求体读入内存做日志输出，直接触发内存溢出。

正确的流式上传实现 #

使用Ktor内置的append方法重载，直接传入File对象，底层会自动处理流式分块上传，无需手动操作输入流：

val client = HttpClient(OkHttp) {
    install(ContentNegotiation) {
        json(Json {
            prettyPrint = true
            isLenient = true
            ignoreUnknownKeys = true
        })
    }

    install(Resources)
    install(Logging) {
        logger = Logger.DEFAULT
        // 关键：避免打印请求体，防止全量加载到内存
        level = LogLevel.HEADERS
        // 或者更精细控制：仅排除multipart类型的请求体日志
        filter { request ->
            !request.contentType().toString().startsWith("multipart/form-data")
        }
    }    

    engine {
        config {
            followRedirects(true)
            connectTimeout(100_000, TimeUnit.MILLISECONDS)
            readTimeout(100_000, TimeUnit.MILLISECONDS)
        }
    }
}

val response: HttpResponse = client.post("http://localhost:8080/upload") {
    setBody(MultiPartFormDataContent(formData {
        append("description", "Ktor logo")
        // 直接传入File对象，自动流式分块
        append(
            key = "image",
            file = File("ktor_logo.png"),
            headers = Headers.build {
                append(HttpHeaders.ContentType, "image/png")
                append(HttpHeaders.ContentDisposition, "filename=\"ktor_logo.png\"")
            }
        )
    }))
    onUpload { bytesSentTotal, contentLength ->
        println("已发送 $bytesSentTotal / $contentLength 字节")
    }
}

额外注意事项 #

• 确保使用支持流式上传的HttpClient引擎：OkHttp和CIO都支持，避免使用已废弃的Apache引擎。
• 服务器端也需要配置支持分块接收，避免服务器端出现OOM。

内容的提问来源于stack exchange，提问作者SemiCacke