批量下载3000-6000个小文件时应用因堆内存压力被杀死的解决方法

问题背景 #

需要从3000-6000个URL下载小文件（最大仅25KB），现有批量下载逻辑因动态创建大量对象实例导致堆内存压力剧增，执行若干次下载后GC日志出现“waiting for allocated...”，随后应用被系统以Signal 9信号杀死，需保留UI进度更新功能。

核心问题分析 #

1. 并发过载：原代码通过forEach直接启动所有下载任务，瞬间创建数千个协程、HTTP连接、流对象，内存占用暴增
2. 对象频繁创建：下载过程中每次读取缓冲区都调用ds.copy()生成新的DownloadState实例，小文件每秒可能生成上百个对象，GC压力陡增
3. UI更新冗余：小文件体积小，频繁更新进度无实际意义，反而加重主线程负担

修复方案 #

方案一：优化现有代码（无需第三方库） #

1. 限制并发数

通过协程的limitedParallelism控制同时运行的下载任务数，避免瞬间占满系统资源

2. 减少对象创建

复用同一个DownloadState实例，仅更新字段而非每次复制；降低进度更新频率，比如每下载1KB或达到文件总大小才更新

3. 复用UI更新逻辑

提取重复的UI状态更新代码，减少冗余对象创建

修改后的下载函数：

fun download(
    dest: File,
    url: String,
    filename: String,
    onPreExecute: () -> Unit = {},
    onPostExecute: (Any) -> Unit = {},
    onProgress: (DownloadState) -> Unit = {},
): Job {
    return appScope.executeAsyncTask(
        onPreExecute = onPreExecute,
        onPostExecute = onPostExecute,
        onProgress = onProgress,
        doInBackground = { reportProgress: suspend (progress: DownloadState) -> Unit ->
            try {
                val httpUrl = URL(url)
                val connection = httpUrl.openConnection()
                connection.connect()
                val totalBytes = connection.contentLengthLong
                httpUrl.openStream().use { ins ->
                    BufferedInputStream(ins).use { bis ->
                        FileOutputStream(File(dest, filename)).use { fos ->
                            var bytesRead: Int
                            var totalBytesRead = 0L
                            val buffer = ByteArray(4096)
                            // 复用DownloadState实例，避免频繁copy
                            val ds = DownloadState(
                                downloading = true,
                                fileSizeBytes = totalBytes,
                                bytesDownloaded = 0L,
                                progress = 0f
                            )
                            // 进度更新阈值：每下载1KB更新一次（小文件可适当调大）
                            val progressUpdateThreshold = 1024L
                            var lastReportedBytes = 0L
                            
                            reportProgress(ds) // 发送初始下载状态
                            
                            while (bis.read(buffer, 0, buffer.size).also { bytesRead = it } != -1) {
                                totalBytesRead += bytesRead
                                fos.write(buffer, 0, bytesRead)
                                
                                // 达到阈值或下载完成才更新进度
                                if (totalBytesRead - lastReportedBytes >= progressUpdateThreshold || totalBytesRead == totalBytes) {
                                    ds.bytesDownloaded = totalBytesRead
                                    ds.progress = totalBytesRead / totalBytes.toFloat()
                                    reportProgress(ds)
                                    lastReportedBytes = totalBytesRead
                                }
                            }
                            
                            // 更新下载完成状态
                            ds.downloading = false
                            reportProgress(ds)
                        }
                    }
                }
            } catch (e: Exception) {
                e.printStackTrace()
                Log.e(TAG, e.message.toString())
                reportProgress(DownloadState(error = e, downloading = false))
                throw e
            }
        }
    )
}

修改后的ViewModel调用代码：

private fun download(directory: File) {
    viewModelScope.launch {
        try {
            val audios = listOf(/* Audio(...) */)

            if (isExternalStorageWritable()) {
                // 限制并发数：根据设备性能调整，推荐10-20个
                coroutineScope {
                    audios.forEachIndexed { i, audio ->
                        val count = i + 1
                        val id = audio.id
                        val done = count == audios.size

                        val filename = audio.getFilename(id)
                        val destDir = File(directory, audio.getDirectory())

                        if (!destDir.exists()) destDir.mkdirs()

                        val targetFile = File(destDir, filename)
                        if (targetFile.exists()) {
                            updateUIState(count, done)
                            return@forEachIndexed
                        }

                        // 用limitedParallelism控制并发任务数
                        launch(Dispatchers.IO.limitedParallelism(15)) {
                            downloadClient.download(
                                dest = destDir,
                                filename = filename,
                                url = audio.url,
                                onPostExecute = {
                                    updateUIState(count, done)
                                }
                            ).join()
                        }
                    }
                }
            }
        } catch (e: Exception) {
            _uiState.update {
                it.copy(
                    syncCompleted = true,
                    audioDownloadErred = true,
                )
            }
        }
    }
}

// 提取UI更新逻辑，避免重复代码
private fun updateUIState(count: Int, done: Boolean) {
    _uiState.update {
        it.copy(
            audioCount = count,
            syncCompleted = done,
            audioDownloadDone = done,
        )
    }
}

方案二：使用成熟下载库（推荐OkHttp+协程） #

OkHttp自带连接池优化，可复用HTTP连接，减少TCP握手开销和对象创建，配合协程更高效管理并发。

添加依赖（Gradle）：

implementation 'com.squareup.okhttp3:okhttp:4.12.0'

下载工具类实现：

class OkHttpDownloader(private val okHttpClient: OkHttpClient) {
    suspend fun download(
        destDir: File,
        filename: String,
        url: String,
        onProgress: (DownloadState) -> Unit
    ) {
        val targetFile = File(destDir, filename)
        if (targetFile.exists()) return

        val request = Request.Builder().url(url).build()
        okHttpClient.newCall(request).execute().use { response ->
            if (!response.isSuccessful) throw IOException("Unexpected code $response")

            response.body?.let { body ->
                val totalBytes = body.contentLength()
                val ds = DownloadState(
                    downloading = true,
                    fileSizeBytes = totalBytes,
                    bytesDownloaded = 0L,
                    progress = 0f
                )
                onProgress(ds)

                val progressThreshold = 1024L
                var lastReported = 0L
                var totalRead = 0L

                body.byteStream().use { inputStream ->
                    BufferedInputStream(inputStream).use { bis ->
                        FileOutputStream(targetFile).use { fos ->
                            val buffer = ByteArray(4096)
                            var bytesRead: Int
                            while (bis.read(buffer).also { bytesRead = it } != -1) {
                                totalRead += bytesRead
                                fos.write(buffer, 0, bytesRead)

                                if (totalRead - lastReported >= progressThreshold || totalRead == totalBytes) {
                                    ds.bytesDownloaded = totalRead
                                    ds.progress = totalRead / totalBytes.toFloat()
                                    onProgress(ds)
                                    lastReported = totalRead
                                }
                            }
                        }
                    }
                }

                ds.downloading = false
                onProgress(ds)
            } ?: throw IOException("Response body is null")
        }
    }
}

ViewModel调用代码：

// 初始化OkHttp客户端，配置连接池
private val okHttpClient = OkHttpClient.Builder()
    .connectionPool(ConnectionPool(20, 5, TimeUnit.MINUTES)) // 调整连接池大小
    .build()
private val downloader = OkHttpDownloader(okHttpClient)

private fun download(directory: File) {
    viewModelScope.launch {
        try {
            val audios = listOf(/* Audio(...) */)

            if (isExternalStorageWritable()) {
                coroutineScope {
                    audios.forEachIndexed { i, audio ->
                        val count = i + 1
                        val done = count == audios.size
                        val destDir = File(directory, audio.getDirectory())
                        val filename = audio.getFilename(audio.id)
                        val targetFile = File(destDir, filename)

                        if (!destDir.exists()) destDir.mkdirs()
                        if (targetFile.exists()) {
                            updateUIState(count, done)
                            return@forEachIndexed
                        }

                        launch(Dispatchers.IO.limitedParallelism(15)) {
                            runCatching {
                                downloader.download(
                                    destDir = destDir,
                                    filename = filename,
                                    url = audio.url,
                                    onProgress = { state ->
                                        // 下载完成后更新UI状态
                                        if (!state.downloading) {
                                            updateUIState(count, done)
                                        }
                                    }
                                )
                            }.onFailure { e ->
                                Log.e(TAG, "Download failed: ${e.message}")
                                _uiState.update { it.copy(audioDownloadErred = true) }
                            }
                        }
                    }
                }
            }
        } catch (e: Exception) {
            _uiState.update {
                it.copy(
                    syncCompleted = true,
                    audioDownloadErred = true,
                )
            }
        }
    }
}

private fun updateUIState(count: Int, done: Boolean) {
    _uiState.update {
        it.copy(
            audioCount = count,
            syncCompleted = done,
            audioDownloadDone = done,
        )
    }
}

关键优化点总结 #

• 并发控制：通过limitedParallelism限制同时运行的任务数，避免资源耗尽
• 对象复用：减少DownloadState实例创建，复用同一个对象更新字段
• 进度节流：降低进度更新频率，减少GC和UI刷新压力
• 连接池优化：使用OkHttp连接池复用HTTP连接，减少TCP握手开销和对象创建

内容的提问来源于stack exchange，提问作者codejockie