询问可限制内存占用并磁盘缓存大字典的Python现有库

Absolutely, there are several mature Python libraries that do exactly what you're looking for—handling large dictionaries by automatically offloading data to disk when a memory threshold is hit, while maintaining a dict-like interface so you don't have to rewrite your core logic. Here are the most popular and reliable options:

diskcache #

This is one of the most straightforward and feature-rich options. It acts as a drop-in replacement for regular dictionaries, with built-in support for limiting memory usage and automatically persisting excess data to disk.

• Key features:
  • Configurable memory size limit (size_limit parameter)
  • Supports TTL, compression, and atomic operations
  • Works with most Python objects (thanks to built-in serialization)
• Quick example:

from diskcache import Cache

# Initialize cache with 1GB memory limit; overflow goes to disk
cache = Cache(size_limit=1024 * 1024 * 1024)  # 1GB

# Use just like a normal dict
cache["huge_dataset"] = your_large_dictionary
value = cache["huge_dataset"]  # Automatically loads from disk if needed

zict #

Zict is designed specifically for creating layered dictionaries—perfect for your use case of combining in-memory caching with disk persistence. It lets you chain storage backends, so you can have a fast in-memory LRU cache that spills over to disk when full.

• Key features:
  • Flexible layered storage (combine memory, disk, even cloud storage)
  • Strict dict-compatible API
  • Lightweight and focused on performance
• Quick example:

from zict import LRU, File

# Create a 1GB in-memory LRU cache
memory_layer = LRU(1024 * 1024 * 1024)
# Create a disk-backed layer (stores data in ./disk_cache directory)
disk_layer = File("./disk_cache")
# Combine them: use memory first, spill to disk when full
combined_cache = LRU(1024 * 1024 * 1024, memory_layer, disk_layer)

# Normal dict operations work here
combined_cache["big_key"] = massive_data
del combined_cache["big_key"]  # Works just like a dict

joblib (for cached function outputs or large objects) #

While joblib is best known for caching function results, it’s also great for efficiently serializing and storing large Python objects (including dictionaries) to disk. You can use its Memory class to wrap operations that generate large dictionaries, or directly use dump/load for manual control.

• Key features:
  • Optimized for numerical data (like NumPy arrays) which are common in large dictionaries
  • Compression support to save disk space
• Quick example for caching a function that returns a large dict:

from joblib import Memory

# Create a memory cache that stores results on disk
memory = Memory(location="./joblib_cache", verbose=0)

@memory.cache
def generate_large_dict():
    # Your code to create the huge dictionary here
    return {f"key_{i}": i * 1000 for i in range(1_000_000)}

# First run generates and saves to disk; subsequent runs load from disk
large_dict = generate_large_dict()

cachetools (with custom disk extension) #

Cachetools provides in-memory caching utilities (like LRU caches), and while it doesn’t have built-in disk support, you can extend it with a disk backend fairly easily. This is a good option if you want full control over the caching logic.

• Quick example using cachetools + disk storage:

from cachetools import LRUCache
import pickle
import os

class DiskBackedLRU(LRUCache):
    def __init__(self, maxsize, disk_dir="./cachetools_disk"):
        super().__init__(maxsize)
        self.disk_dir = disk_dir
        os.makedirs(disk_dir, exist_ok=True)
    
    def __getitem__(self, key):
        try:
            return super().__getitem__(key)
        except KeyError:
            # Load from disk if not in memory
            with open(os.path.join(self.disk_dir, str(key)), "rb") as f:
                return pickle.load(f)
    
    def __setitem__(self, key, value):
        try:
            super().__setitem__(key, value)
        except ValueError:
            # If memory is full, save to disk instead
            with open(os.path.join(self.disk_dir, str(key)), "wb") as f:
                pickle.dump(value, f)

# Use with 1GB memory limit
cache = DiskBackedLRU(maxsize=1024 * 1024 * 1024)
cache["large_data"] = your_big_dict

• Serialization: Make sure your dictionary values are serializable (most built-in types are, but custom classes may need __pickle__ methods or use alternative serializers like msgpack with some libraries).
• Performance: Disk I/O is slower than memory, so these libraries work best when you’re accessing a subset of the data frequently (the hot data stays in memory, cold data goes to disk).
• Cleanup: Don’t forget to clear disk caches when you’re done, or use libraries that support automatic cleanup (like diskcache’s evict method or TTL settings).

内容的提问来源于stack exchange，提问作者madan kandula