无需安装Caffe，在Python中创建LMDB的其他方法

Great question! You're totally right—installing Caffe just for that Datum protobuf is overkill. Here are some clean, framework-agnostic ways to create LMDB databases in Python without relying on Caffe:

1. Use Python's built-in pickle for serialization #

Pickle is the easiest drop-in replacement since it’s part of Python’s standard library. It lets you serialize almost any Python object (numpy arrays, tuples of data/labels, etc.) into bytes, which you can write directly to LMDB.

Pros: No extra dependencies, works with any Python object.
Cons: Pickle files are Python-specific, so other languages might have trouble reading them. Also, not the most space-efficient for large datasets.

Here’s a quick example:

import lmdb
import pickle
import numpy as np

# Initialize LMDB environment (adjust map_size based on your dataset size)
env = lmdb.open('my_lmdb', map_size=1024*1024*1024)  # 1GB map size

# Write data to LMDB
with env.begin(write=True) as txn:
    for idx in range(100):
        # Sample data: 3-channel image tensor + label
        data = np.random.rand(3, 224, 224)
        label = idx % 10
        # Serialize the data-label tuple
        serialized = pickle.dumps((data, label))
        # Write to LMDB with a padded string key (ensures consistent key length)
        txn.put(f'{idx:08d}'.encode(), serialized)

# Read data back from LMDB
with env.begin() as txn:
    cursor = txn.cursor()
    for key, value in cursor:
        data, label = pickle.loads(value)
        print(f'Key: {key.decode()}, Label: {label}, Data shape: {data.shape}')

env.close()

2. Define your own lightweight Protobuf schema #

If you want a language-agnostic, efficient serialization format (like Caffe’s Datum but without the Caffe dependency), you can create your own minimal Protobuf schema. This is a great middle ground between pickle and heavyweight framework-specific tools.

First, create a custom_datum.proto file:

syntax = "proto3";

message CustomDatum {
  optional int32 channels = 1;
  optional int32 height = 2;
  optional int32 width = 3;
  optional bytes data = 4;  # Raw byte array (e.g., numpy array converted to bytes)
  optional int32 label = 5;
}

Generate the Python code using the protobuf compiler (install first with pip install protobuf):

protoc --python_out=. custom_datum.proto

Now use it in your LMDB code:

import lmdb
import numpy as np
from custom_datum_pb2 import CustomDatum

env = lmdb.open('custom_lmdb', map_size=1024*1024*1024)

# Write data
with env.begin(write=True) as txn:
    for idx in range(100):
        data = np.random.randint(0, 255, (3, 224, 224), dtype=np.uint8)
        label = idx % 10
        
        datum = CustomDatum()
        datum.channels = 3
        datum.height = 224
        datum.width = 224
        datum.data = data.tobytes()
        datum.label = label
        
        txn.put(f'{idx:08d}'.encode(), datum.SerializeToString())

# Read data
with env.begin() as txn:
    cursor = txn.cursor()
    for key, value in cursor:
        datum = CustomDatum()
        datum.ParseFromString(value)
        data = np.frombuffer(datum.data, dtype=np.uint8).reshape(datum.channels, datum.height, datum.width)
        print(f'Key: {key.decode()}, Label: {datum.label}, Data shape: {data.shape}')

env.close()

3. Store raw NumPy bytes (for numerical data) #

If you’re working exclusively with numpy arrays, you can skip serialization libraries altogether and write the raw bytes of the array directly to LMDB. This is super fast and space-efficient.

import lmdb
import numpy as np

env = lmdb.open('numpy_lmdb', map_size=1024*1024*1024)

# Write data: store array bytes and label as separate entries
with env.begin(write=True) as txn:
    for idx in range(100):
        data = np.random.rand(3, 224, 224).astype(np.float32)
        label = idx % 10
        # Store data and label with paired keys
        txn.put(f'data_{idx:08d}'.encode(), data.tobytes())
        txn.put(f'label_{idx:08d}'.encode(), np.int32(label).tobytes())

# Read data
with env.begin() as txn:
    for idx in range(100):
        data_bytes = txn.get(f'data_{idx:08d}'.encode())
        label_bytes = txn.get(f'label_{idx:08d}'.encode())
        
        data = np.frombuffer(data_bytes, dtype=np.float32).reshape(3, 224, 224)
        label = np.frombuffer(label_bytes, dtype=np.int32)[0]
        print(f'Idx: {idx}, Label: {label}, Data shape: {data.shape}')

env.close()

4. Store compressed image bytes (for image datasets) #

If your data is images, save space by storing them in compressed formats (JPEG/PNG) instead of raw tensors. Use PIL/Pillow to handle encoding and decoding.

import lmdb
from PIL import Image
import numpy as np

env = lmdb.open('image_lmdb', map_size=1024*1024*1024)

# Write compressed images
with env.begin(write=True) as txn:
    for idx in range(100):
        # Create a sample RGB image (or load from disk with Image.open())
        img_array = np.random.randint(0, 255, (224, 224, 3), dtype=np.uint8)
        img = Image.fromarray(img_array)
        label = idx % 10
        
        # Encode image to JPEG bytes (adjust quality as needed)
        img_bytes = img.tobytes(format='JPEG', quality=90)
        
        # Store image and label
        txn.put(f'img_{idx:08d}'.encode(), img_bytes)
        txn.put(f'label_{idx:08d}'.encode(), np.int32(label).tobytes())

# Read and decode images
with env.begin() as txn:
    for idx in range(100):
        img_bytes = txn.get(f'img_{idx:08d}'.encode())
        label_bytes = txn.get(f'label_{idx:08d}'.encode())
        
        img = Image.frombytes('RGB', (224, 224), img_bytes, 'JPEG')
        label = np.frombuffer(label_bytes, dtype=np.int32)[0]
        print(f'Idx: {idx}, Label: {label}, Image size: {img.size}')

env.close()

内容的提问来源于stack exchange，提问作者Sean Saito