如何实现30MB JSON文件的流式读取与返回，规避内存峰值？

我需要读取一个30MB的JSON文件，然后以JSON格式流式返回结果。找到的代码会序列化整个大对象，导致内存峰值。怎么修改才能实现从JSON文件流式读取部分数据，再流式返回给调用应用？

初始代码（存在内存峰值问题） #

HttpResponseMessage response = new HttpResponseMessage(HttpStatusCode.OK);
response.Content = new PushStreamContent((stream, content, context) =>
{
     using (StreamWriter sw = new StreamWriter(stream, Encoding.UTF8))
     using (JsonTextWriter jtw = new JsonTextWriter(sw))
     {
         JsonSerializer ser = new JsonSerializer();
         ser.Serialize(jtw, hugeObject); // 不要序列化整个对象
         // 需要改为：读取文件的一部分并流式返回，直到整个文件处理完毕
     }
 }, "application/json");

编辑说明：hugeObject是JSON数组格式 #

[
    {
        "agent": "name1",
        "entityCode": "010002",
        "state": "AL",
        "memberType": "U",
        "producerCode": "572714"
    },
    {
        "agent": "name2",
        "entityCode": "010002",
        "state": "AL",
        "memberType": "U",
        "producerCode": "572714"
    }
]

更新说明：当前实现的代码（会生成大字符串，需改为流式处理） #

[HttpGet, Route("/api/GetAllData/{callingApp}")]
public IActionResult GetAllData(
 [RegularExpression(@"^[a-zA-Z0-9_-]*$"), Required, MaxLength(50)] string callingApp)
{
    string fileName = _config.GetSection("AllAgentsPath").Value;
    StreamReader reader = new StreamReader(fileName);

    var buffer = new char[4096];
    int numberRead;
    var stringBuilder = new StringBuilder();
    while ((numberRead = reader.ReadBlock(buffer, 0, buffer.Length)) > 0)
    {
        stringBuilder.Append(buffer[..numberRead]);
    }

    return Ok(stringBuilder.ToString());
}

解决方案1：直接返回文件流（最简单，无需修改JSON内容） #

如果只是要原封不动返回JSON文件，用FileStreamResult即可自动实现流式传输，不会把整个文件加载到内存：

[HttpGet, Route("/api/GetAllData/{callingApp}")]
public IActionResult GetAllData(
 [RegularExpression(@"^[a-zA-Z0-9_-]*$"), Required, MaxLength(50)] string callingApp)
{
    string fileName = _config.GetSection("AllAgentsPath").Value;
    // 以只读、异步方式打开文件流，指定缓冲区大小
    var fileStream = new FileStream(fileName, FileMode.Open, FileAccess.Read, FileShare.Read, bufferSize: 4096, useAsync: true);
    // 返回流式结果，指定JSON内容类型
    return File(fileStream, "application/json");
}

解决方案2：处理JSON内容时的流式读写（需修改条目再返回） #

如果需要对JSON数组的每个条目做修改，用Json.NET的流式API边读边写，避免加载整个对象到内存：

[HttpGet, Route("/api/GetAllData/{callingApp}")]
public async Task<IActionResult> GetAllData(
 [RegularExpression(@"^[a-zA-Z0-9_-]*$"), Required, MaxLength(50)] string callingApp)
{
    string fileName = _config.GetSection("AllAgentsPath").Value;

    var response = new HttpResponseMessage(HttpStatusCode.OK);
    response.Content = new PushStreamContent(async (outputStream, context, transportContext) =>
    {
        using var fileStream = new FileStream(fileName, FileMode.Open, FileAccess.Read, FileShare.Read, 4096, true);
        using var streamReader = new StreamReader(fileStream, Encoding.UTF8);
        using var jsonReader = new JsonTextReader(streamReader);
        using var streamWriter = new StreamWriter(outputStream, Encoding.UTF8);
        using var jsonWriter = new JsonTextWriter(streamWriter);

        var serializer = new JsonSerializer();

        // 处理JSON数组
        if (await jsonReader.ReadAsync() && jsonReader.TokenType == JsonToken.StartArray)
        {
            await jsonWriter.WriteStartArrayAsync();

            // 逐个读取数组中的对象
            while (await jsonReader.ReadAsync() && jsonReader.TokenType != JsonToken.EndArray)
            {
                // 读取单个对象（仅加载当前条目到内存）
                var item = serializer.Deserialize<JObject>(jsonReader);

                // 这里可对条目做修改，比如添加字段
                // item["processed"] = true;

                // 流式写入修改后的对象
                serializer.Serialize(jsonWriter, item);
            }

            await jsonWriter.WriteEndArrayAsync();
        }

        await streamWriter.FlushAsync();
    }, "application/json");

    return new HttpResponseMessageResult(response);
}

// 辅助类：将HttpResponseMessage转换为IActionResult
public class HttpResponseMessageResult : IActionResult
{
    private readonly HttpResponseMessage _responseMessage;

    public HttpResponseMessageResult(HttpResponseMessage responseMessage)
    {
        _responseMessage = responseMessage;
    }

    public async Task ExecuteResultAsync(ActionContext context)
    {
        var response = context.HttpContext.Response;
        response.StatusCode = (int)_responseMessage.StatusCode;

        foreach (var header in _responseMessage.Headers)
        {
            response.Headers.TryAdd(header.Key, new Microsoft.Extensions.Primitives.StringValues(header.Value.ToArray()));
        }

        foreach (var header in _responseMessage.Content.Headers)
        {
            response.ContentType = header.Value.ToString();
        }

        await using var stream = await _responseMessage.Content.ReadAsStreamAsync();
        await stream.CopyToAsync(response.Body);
        await response.Body.FlushAsync();
    }
}

关键注意点 #

• 始终使用流式API（FileStream、JsonTextReader/JsonTextWriter），避免将整个文件或对象加载到内存
• 无需修改JSON内容时，优先用FileStreamResult，实现最简单
• 需要处理内容时，采用边读边写的方式，仅加载单个JSON条目到内存

内容的提问来源于stack exchange，提问作者WorkJ