Python项目多JSON文件替代单文件初始化的实现及最佳实践咨询

Hey there! Splitting a monolithic JSON into nested files is a great way to keep your data organized—let's cover how to implement this smoothly and share some best practices for managing static data in Python projects.

Implementation: Load Nested JSON Files as a Single Dictionary #

The core idea is to write a helper function that recursively traverses your nested database/ directory, loads each JSON file, and assembles their contents into a single nested dictionary. This way, you can access the data exactly like you did with the original single data.json file.

Here's a practical implementation you can drop into your project:

import os
import json

def load_nested_static_data(base_dir):
    """Recursively load all JSON files in a directory into a nested dictionary."""
    assembled_data = {}
    
    # Walk through every file in the base directory
    for root, _, files in os.walk(base_dir):
        for file in files:
            if not file.endswith(".json"):
                continue  # Skip non-JSON files
            
            # Calculate the relative path from the base directory to the current file's folder
            relative_folder_path = os.path.relpath(root, base_dir)
            # Split the relative path into nested keys (e.g., "B/C" becomes ["B", "C"])
            nested_keys = relative_folder_path.split(os.sep) if relative_folder_path != "." else []
            # Get the filename without the .json extension as the final key
            data_key = os.path.splitext(file)[0]
            
            # Traverse or build the nested dictionary structure
            current_dict_level = assembled_data
            for key in nested_keys:
                if key not in current_dict_level:
                    current_dict_level[key] = {}
                current_dict_level = current_dict_level[key]
            
            # Load the JSON file content and assign it to the final key
            full_file_path = os.path.join(root, file)
            with open(full_file_path, "r", encoding="utf-8") as f:
                current_dict_level[data_key] = json.load(f)
    
    return assembled_data

# ------------------------------
# Usage in your lambda_function.py
# ------------------------------
# Get the absolute path to your database directory (no hardcoding!)
DATABASE_DIR = os.path.join(os.path.dirname(__file__), "database")
# Load the data ONCE when the module initializes (since it's static)
APP_STATIC_DATA = load_nested_static_data(DATABASE_DIR)

# Now you can access data just like you did with the single JSON file:
# print(APP_STATIC_DATA["A"]["a"])
# print(APP_STATIC_DATA["B"]["C"]["b"])

How this works: #

• The function uses os.walk() to scan every file in your database/ directory and its subfolders.
• It converts the folder structure into nested dictionary keys (e.g., database/B/C/a.json maps to APP_STATIC_DATA["B"]["C"]["a"]).
• It loads each JSON file's content and places it in the correct position in the nested dictionary.

Best Practices for Static Data Management in Python #

Since your data is static (only used for initialization, no changes at runtime), here are some tips to keep your project maintainable:

• Cache the loaded data: Load the data once when your module starts (like in the example above) instead of reading files every time you need data. This avoids unnecessary I/O and speeds up your code.
• Use relative paths: Never hardcode absolute paths (e.g., /home/user/project/database). Use os.path.dirname(__file__) to dynamically get the current script's directory, then build the path to your database folder. This works across different environments and deployment setups.
• Align directory structure with data logic: Organize your folders and files to mirror your application's domain logic. For example, if A/ contains user-related data and B/ contains product data, this makes it easy for other developers to find and modify specific data.
• Validate data structure: Add a validation step after loading the data to ensure it matches your expected schema. You can use libraries like pydantic to define data models and catch errors early (e.g., missing keys, incorrect data types).
• Include data in version control: Commit all your JSON files to Git (or your version control system). This lets you track changes, roll back mistakes, and collaborate with others effectively.
• Consider alternative formats (optional): If your data has complex comments or nested structures, YAML might be more readable than JSON. Python's pyyaml library can load YAML files just as easily as JSON, and YAML supports comments which can help document your static data.

内容的提问来源于stack exchange，提问作者Matthieu Veron