多流合并序列化优化：小文件备份场景替代JsonTextWriter方案咨询

Great question—dealing with thousands of tiny files is a classic performance pain point, and ditching JSON (which adds unnecessary text overhead, especially for binary data) is absolutely the right call. Here are three high-performance approaches tailored to your scenario, ordered by flexibility/performance tradeoff:

1. Custom Binary Format (极致性能 & 最小体积) #

This is the fastest and most space-efficient option because you eliminate all overhead from generic serialization libraries. You define a simple, compact structure for each file entry, so you’re only writing exactly what you need.

Structure Design #

I’d recommend a straightforward binary layout for each entry:

• 4-byte integer: Length of the filename (UTF-8 encoded)
• Variable bytes: Filename (UTF-8)
• 4-byte integer: Length of the file data
• Variable bytes: File data (raw binary)
  Plus a leading 4-byte integer to store the total number of files (makes parsing easier later).

Code Example #

using System.IO;
using System.Text;

// Writing the backup
using (var outputStream = new FileStream("backup.bin", FileMode.Create))
using (var writer = new BinaryWriter(outputStream, Encoding.UTF8, leaveOpen: false))
{
    // Write total file count first
    writer.Write(models.Count);

    foreach (var model in models)
    {
        // Write filename
        var nameBytes = Encoding.UTF8.GetBytes(model.Name);
        writer.Write(nameBytes.Length);
        writer.Write(nameBytes);

        // Write file data
        writer.Write((int)model.Data.Length);
        model.Data.CopyTo(outputStream);
        // Reset stream position in case it's reused later
        model.Data.Position = 0;
    }
}

// Reading the backup (for reference)
using (var inputStream = new FileStream("backup.bin", FileMode.Open))
using (var reader = new BinaryReader(inputStream, Encoding.UTF8))
{
    int fileCount = reader.ReadInt32();
    for (int i = 0; i < fileCount; i++)
    {
        int nameLength = reader.ReadInt32();
        string name = Encoding.UTF8.GetString(reader.ReadBytes(nameLength));
        
        int dataLength = reader.ReadInt32();
        byte[] data = reader.ReadBytes(dataLength);
        Stream dataStream = new MemoryStream(data);
        
        // Recreate your Model instance here
        var restoredModel = new Model { Name = name, Data = dataStream };
    }
}

Pros & Cons #

• ✅ Zero overhead, fastest I/O and smallest output size
• ✅ Full control over the format
• ❌ No built-in tooling to view the backup (you’ll need your own parser)
• ❌ Manual maintenance of serialization/deserialization logic

2. ZipArchive with No Compression (Standard Format & Ease of Use) #

If you want a standard, tool-compatible format (so you can open the backup with any zip utility) without the overhead of compression (which is useless for tiny files—sometimes even increases size), this is a great middle ground. .NET’s built-in ZipArchive is optimized for streaming, so it won’t bog down your performance.

Code Example #

using System.IO;
using System.IO.Compression;

using (var outputStream = new FileStream("backup.zip", FileMode.Create))
using (var archive = new ZipArchive(outputStream, ZipArchiveMode.Create))
{
    foreach (var model in models)
    {
        // Create an entry with the file's name, disable compression
        var entry = archive.CreateEntry(model.Name, CompressionLevel.NoCompression);
        
        using (var entryStream = entry.Open())
        {
            model.Data.CopyTo(entryStream);
            // Reset stream position for reuse
            model.Data.Position = 0;
        }
    }
}

Pros & Cons #

• ✅ Standard zip format—use any zip tool to inspect contents
• ✅ Minimal code, no custom logic needed
• ✅ Streaming support keeps memory usage low
• ❌ Slight overhead compared to custom binary (zip headers add a few bytes per entry)
• ❌ Compression is counterproductive here, so make sure to disable it

3. Binary Serialization with MessagePack (Mature Library & Low Maintenance) #

If you prefer using a battle-tested binary serialization library instead of rolling your own, MessagePack is a fantastic choice—it’s way faster than JSON and produces compact output. Since you can’t modify the Model class, you can create a custom formatter to only serialize the Name and Data properties you care about.

Setup #

First, install the MessagePack NuGet package:

Install-Package MessagePack

Code Example #

using MessagePack;
using System.IO;

// Configure custom resolver to handle the Model class
var options = MessagePackSerializerOptions.Standard.WithResolver(new ModelResolver());

// Write the backup
using (var outputStream = new FileStream("backup.msgpack", FileMode.Create))
{
    MessagePackSerializer.Serialize(outputStream, models, options);
}

// Custom resolver and formatter
public class ModelResolver : IMessagePackFormatterResolver
{
    public static readonly ModelResolver Instance = new ModelResolver();

    public IMessagePackFormatter<T> GetFormatter<T>()
    {
        if (typeof(T) == typeof(Model))
        {
            return (IMessagePackFormatter<T>)new ModelFormatter();
        }
        // Fall back to standard resolver for other types
        return StandardResolver.Instance.GetFormatter<T>();
    }

    private class ModelFormatter : IMessagePackFormatter<Model>
    {
        public void Serialize(ref MessagePackWriter writer, Model value, MessagePackSerializerOptions options)
        {
            // Write a map with 2 entries: Name and Data
            writer.WriteMapHeader(2);
            writer.Write("Name");
            writer.Write(value.Name);
            
            writer.Write("Data");
            // Copy stream to memory and write as byte array
            using (var ms = new MemoryStream())
            {
                value.Data.CopyTo(ms);
                writer.Write(ms.ToArray());
                value.Data.Position = 0;
            }
        }

        public Model Deserialize(ref MessagePackReader reader, MessagePackSerializerOptions options)
        {
            reader.ReadMapHeader();
            var model = new Model();
            
            while (reader.ReadString() is string key)
            {
                switch (key)
                {
                    case "Name":
                        model.Name = reader.ReadString();
                        break;
                    case "Data":
                        var bytes = reader.ReadBytes();
                        model.Data = new MemoryStream(bytes);
                        break;
                }
            }
            return model;
        }
    }
}

Pros & Cons #

• ✅ Mature, optimized library with great performance
• ✅ No manual binary parsing logic
• ✅ Compact output (close to custom binary)
• ❌ Slight overhead compared to custom binary
• ❌ Requires adding a third-party dependency

Final Recommendation #

• Go with Custom Binary Format if you need absolute top performance and smallest backup size, and don’t need to inspect the backup with external tools.
• Choose ZipArchive (no compression) if you want a standard format that’s easy to work with, and can tolerate minimal overhead.
• Pick MessagePack if you want the convenience of a serialization library without rolling your own binary logic.

内容的提问来源于stack exchange，提问作者Oleg Golovkov