寻求高效保存多维NumPy数组的自描述格式方案

我正在寻找一种高效保存多维数组的自描述格式（如JSON）。存储格式可分为自描述格式（例如JSON、YAML、CSV）和非自描述格式（例如pickle、protobuf、HDF5）。

首先想到的是JSON，因此我对比了JSON与pickle、npy的dump/load耗时，结果显示标准库的JSON效率极低：

pickle dump: 0.011399507522583008
pickle load: 0.01577591896057129
npy dump: 0.006514072418212891
npy load: 0.004297971725463867
json dump: 4.027008533477783
json load: 0.6741242408752441

测试代码如下：

import json
import time
import pickle
import uuid
import numpy as np

# 测试数据：100张图像序列
image_seq = np.random.randint(0, high=255, size=(100, 3, 224, 224), dtype=np.uint8)


# pickle 性能测试
ts = time.time()
pickle_file = "/tmp/{}.pickle".format(str(uuid.uuid4()))
with open(pickle_file, "wb") as f:
    pickle.dump(image_seq, f)
print("pickle dump:", time.time() - ts)

with open(pickle_file, "rb") as f:
    pickle.load(f)
print("pickle load:", time.time() - ts)


# npy 性能测试
npy_file = "/tmp/{}.npy".format(str(uuid.uuid4()))
ts = time.time()
np.save(npy_file, image_seq)
print("npy dump:", time.time() - ts)

ts = time.time()
np.load(npy_file)
print("npy load:", time.time() - ts)


# JSON 性能测试
ts = time.time()
json_file = "/tmp/{}.json".format(str(uuid.uuid4()))
with open(json_file, "w") as f:
    json.dump(image_seq.flatten().tolist(), f)
print("json dump:", time.time() - ts)

ts = time.time()
with open(json_file, "r") as f:
    json.load(f)
print("json load:", time.time() - ts)

是否有人了解其他可替代的自描述格式或更高效的序列化/反序列化方法？

推荐方案 #

1. MsgPack

MsgPack是二进制类JSON格式，兼具自描述特性和接近二进制格式的性能，适配多维数组场景。通过msgpack-numpy扩展可直接支持numpy数组序列化，无需手动转列表。

安装依赖：

pip install msgpack msgpack-numpy

示例代码：

import msgpack
import msgpack_numpy as m
import numpy as np
import time

# 补丁让msgpack支持numpy数组
m.patch()

image_seq = np.random.randint(0, high=255, size=(100, 3, 224, 224), dtype=np.uint8)

# 序列化
ts = time.time()
with open("/tmp/data.msgpack", "wb") as f:
    msgpack.dump(image_seq, f)
print("msgpack dump:", time.time() - ts)

# 反序列化
ts = time.time()
with open("/tmp/data.msgpack", "rb") as f:
    loaded_data = msgpack.load(f)
print("msgpack load:", time.time() - ts)

2. ujson

ujson是标准JSON库的高性能替代，序列化/反序列化速度远快于原生json，同时保持JSON的自描述特性。虽需将numpy数组转为列表，但整体效率远超标准json。

安装依赖：

pip install ujson

示例代码：

import ujson
import numpy as np
import time

image_seq = np.random.randint(0, high=255, size=(100, 3, 224, 224), dtype=np.uint8)

# 序列化
ts = time.time()
with open("/tmp/data.json", "w") as f:
    ujson.dump(image_seq.tolist(), f)
print("ujson dump:", time.time() - ts)

# 反序列化
ts = time.time()
with open("/tmp/data.json", "r") as f:
    loaded_data = np.array(ujson.load(f)).reshape(100, 3, 224, 224)
print("ujson load:", time.time() - ts)

3. BSON

BSON是MongoDB使用的二进制JSON格式，支持二进制数据存储，适合多维数组这类二进制密集型数据，同时保持自描述性。

安装依赖：

pip install bson

示例代码：

import bson
import numpy as np
import time

image_seq = np.random.randint(0, high=255, size=(100, 3, 224, 224), dtype=np.uint8)

# 序列化
ts = time.time()
with open("/tmp/data.bson", "wb") as f:
    bson.dump({"data": image_seq.tobytes(), "shape": image_seq.shape, "dtype": str(image_seq.dtype)}, f)
print("bson dump:", time.time() - ts)

# 反序列化
ts = time.time()
with open("/tmp/data.bson", "rb") as f:
    loaded_dict = bson.load(f)
    loaded_data = np.frombuffer(loaded_dict["data"], dtype=loaded_dict["dtype"]).reshape(loaded_dict["shape"])
print("bson load:", time.time() - ts)

4. YAML（高效实现）

如果偏好可读性强的文本格式，可使用PyYAML的CSafeLoader/CSafeDumper实现，性能比原生YAML提升显著，同时保持自描述特性。不过文本格式性能仍不如二进制类JSON格式。

安装依赖：

pip install pyyaml

示例代码：

import yaml
import numpy as np
import time

image_seq = np.random.randint(0, high=255, size=(100, 3, 224, 224), dtype=np.uint8)

# 序列化
ts = time.time()
with open("/tmp/data.yaml", "w") as f:
    yaml.dump({"data": image_seq.tolist(), "shape": image_seq.shape, "dtype": str(image_seq.dtype)}, f, Dumper=yaml.CSafeDumper)
print("yaml dump:", time.time() - ts)

# 反序列化
ts = time.time()
with open("/tmp/data.yaml", "r") as f:
    loaded_dict = yaml.load(f, Loader=yaml.CSafeLoader)
    loaded_data = np.array(loaded_dict["data"], dtype=loaded_dict["dtype"]).reshape(loaded_dict["shape"])
print("yaml load:", time.time() - ts)

内容的提问来源于stack exchange，提问作者HiroIshida