OpenCV图像帧与时间戳优化存储后无法转回字典问题求助

Your original approach of converting frame dictionaries to strings fails because the string representation of a numpy array doesn’t preserve the binary data needed to reconstruct the array. Let’s fix this with efficient, reversible serialization methods that keep your data intact while saving space.

Why Your Current Method Doesn’t Work #

When you do str(frame_dict).encode('utf-8'), you’re creating a human-readable text string of the dictionary, not a serialized version of the actual data. The numpy array becomes a text description (like array([[[0.,0.,0.], ...]])), which can’t be converted back to a functional numpy array reliably. This approach is a dead end—let’s use proper serialization instead.

Solution 1: Use numpy.savez_compressed (Most Space-Efficient) #

Numpy’s savez_compressed is optimized for storing arrays and compresses data automatically. Since all your frames have the same dimensions, we can stack them into a single 4D array (number of frames × height × width × channels) along with a separate array for timestamps.

Saving Code #

import cv2
import numpy as np

video_capture = cv2.VideoCapture('/tmp/abc.mp4')
success, frame = video_capture.read()
fps = video_capture.get(5)
frame_number = 0
success = True

frames_list = []
timestamps_list = []

while success:
    timestamp = 1000 * float(frame_number) / fps
    frames_list.append(frame)
    timestamps_list.append(timestamp)
    success, frame = video_capture.read()
    frame_number += 1

# Convert lists to numpy arrays (stack frames into 4D array)
frames_array = np.stack(frames_list)
timestamps_array = np.array(timestamps_list)

# Save with compression
np.savez_compressed('/tmp/frame_data.npz', frames=frames_array, timestamps=timestamps_array)

Loading Code #

import numpy as np

# Load the compressed data
data = np.load('/tmp/frame_data.npz')
frames = data['frames']  # Shape: (num_frames, height, width, channels)
timestamps = data['timestamps']  # Shape: (num_frames,)

# Access individual frames and timestamps
for idx in range(len(timestamps)):
    current_frame = frames[idx]
    current_timestamp = timestamps[idx]
    # Your processing logic here

This method will drastically reduce file size (far smaller than your original 2.2GB) and maintain full fidelity of your frames and timestamps.

Solution 2: Pickle with Gzip Compression (Keep Dict Structure) #

If you prefer to retain the dictionary structure (each entry has timestamp and frame_matrix), use pickle with gzip compression to save space while preserving the data structure.

Saving Code #

import cv2
import pickle
import gzip

video_capture = cv2.VideoCapture('/tmp/abc.mp4')
success, frame = video_capture.read()
fps = video_capture.get(5)
frame_number = 0
success = True
frames_arr = []

while success:
    timestamp = 1000 * float(frame_number) / fps
    frame_dict = {'timestamp': timestamp, 'frame_matrix': frame}
    frames_arr.append(frame_dict)
    success, frame = video_capture.read()
    frame_number += 1

# Save with gzip compression
with gzip.open('/tmp/frame_data.pkl.gz', 'wb') as bf:
    pickle.dump(frames_arr, bf)

Loading Code #

import pickle
import gzip

with gzip.open('/tmp/frame_data.pkl.gz', 'rb') as f:
    frames_data = pickle.load(f)

for frame_dict in frames_data:
    frame = frame_dict['frame_matrix']
    timestamp = frame_dict['timestamp']
    # Your processing logic here

This approach keeps your original data structure intact while compressing the file to a manageable size.

Solution 3: Use Joblib (Optimized for Numpy Data) #

Joblib is designed for efficient serialization of numpy arrays and large data structures. It often outperforms pickle for numpy-heavy data.

Saving Code #

import cv2
from joblib import dump

video_capture = cv2.VideoCapture('/tmp/abc.mp4')
success, frame = video_capture.read()
fps = video_capture.get(5)
frame_number = 0
success = True
frames_arr = []

while success:
    timestamp = 1000 * float(frame_number) / fps
    frame_dict = {'timestamp': timestamp, 'frame_matrix': frame}
    frames_arr.append(frame_dict)
    success, frame = video_capture.read()
    frame_number += 1

# Save with zlib compression
dump(frames_arr, '/tmp/frame_data.joblib', compress='zlib')

Loading Code #

from joblib import load

frames_data = load('/tmp/frame_data.joblib')

for frame_dict in frames_data:
    frame = frame_dict['frame_matrix']
    timestamp = frame_dict['timestamp']
    # Your processing logic here

Which Solution Should You Choose? #

• Use numpy.savez_compressed if you want the smallest file size and fastest access to frames.
• Use pickle + gzip if you need to keep the dictionary structure and want a simple, familiar approach.
• Use joblib if you’re working with large numpy arrays and want optimized serialization/deserialization speed.

All these methods will let you properly load your data back into usable numpy arrays and timestamps, unlike your original string conversion approach.

内容的提问来源于stack exchange，提问作者Avoid