嵌套字典转pandas DataFrame：处理含空值数组的数据问题

Got it, let's break this down properly. The main hurdle here is dealing with those inner arrays that are full of None values—we need to make sure those don't break the DataFrame structure, while keeping outer_key* as the index and key1/key2/key3 as fixed columns.

Here's a robust solution that handles both normal and all-None cases:

1. Import Required Libraries #

First, grab pandas and numpy (we'll use numpy's nan for consistent missing values):

import pandas as pd
import numpy as np

2. Prepare Your Raw Data #

Let's assume your data looks something like this (adjust to match your actual dataset):

raw_data = {
    "outer_key1": [{"key1": 1, "key2": 2, "key3": 3}, {"key1": 4, "key2": 5, "key3": 6}, {"key1": 7, "key2": 8, "key3": 9}],
    "outer_key2": [{"key1": 10, "key2": 11, "key3": 12}, {"key1": 13, "key2": 14, "key3": 15}, {"key1": 16, "key2": 17, "key3": 18}],
    "outer_key3": [None, None, None]
}

3. Process the Data & Build the DataFrame #

The core idea is to:

• Expand each outer key's inner array into individual rows
• Replace any None entries with a dictionary that has key1/key2/key3 set to NaN (so pandas recognizes the columns properly)
• Repeat the outer key as the index for each row in its inner array

You can choose between a concise one-liner or a more explicit loop for clarity:

Option A: Concise List Comprehension #

# Expand data and handle None values
expanded_rows = [
    item if item is not None else {"key1": np.nan, "key2": np.nan, "key3": np.nan}
    for outer_key, inner_arr in raw_data.items()
    for item in inner_arr
]

# Create index values (repeat each outer key 3 times, since inner arrays are fixed length)
index_vals = [outer_key for outer_key, inner_arr in raw_data.items() for _ in inner_arr]

# Build the DataFrame
df = pd.DataFrame(expanded_rows, index=index_vals)
# Optional: Name the index for better readability
df.index.name = "outer_key"

Option B: Explicit Loop (Easier to Debug) #

If you want to track each step clearly, use this version:

expanded_rows = []
index_vals = []

for outer_key, inner_arr in raw_data.items():
    for item in inner_arr:
        # Handle None by creating a placeholder dict with NaNs
        if item is None:
            expanded_rows.append({"key1": np.nan, "key2": np.nan, "key3": np.nan})
        else:
            expanded_rows.append(item)
        # Add the outer key to the index list
        index_vals.append(outer_key)

df = pd.DataFrame(expanded_rows, index=index_vals)
df.index.name = "outer_key"

4. Check the Result #

Running either option will give you a consistent DataFrame that looks like this:

key1  key2  key3
outer_key                  
outer_key1   1.0   2.0   3.0
outer_key1   4.0   5.0   6.0
outer_key1   7.0   8.0   9.0
outer_key2  10.0  11.0  12.0
outer_key2  13.0  14.0  15.0
outer_key2  16.0  17.0  18.0
outer_key3   NaN   NaN   NaN
outer_key3   NaN   NaN   NaN
outer_key3   NaN   NaN   NaN

This works because we're enforcing a uniform structure—even when the inner array has None, we replace each None with a dictionary containing all three required keys. This prevents pandas from throwing errors or dropping columns when it encounters missing entries.

内容的提问来源于stack exchange，提问作者ba_ul