如何将含嵌套字典的Python日志数据转成同级键值同行的CSV

Got it, let's solve this problem of converting your nested log dictionaries into a CSV where all sibling keys (even those inside warning/error nested dicts) show up in the same row—including handling entries that don't have those nested structures at all.

Step 1: Collect All Possible Columns #

First, we need to gather every possible key from both the top-level dictionaries and the nested ones. This ensures our CSV has a column for every data point, even if some entries don't have that data.

Step 2: Flatten Each Log Entry #

For each entry in your dlog list, we'll merge the top-level key-value pairs with the nested ones from warning and error. If an entry doesn't have a warning or error dict, we'll fill those columns with a default value (like 0 for counts, or an empty string if you prefer).

Full Code Example #

import csv

# Your sample log data (filled in the error section for completeness)
dlog = [
    {
        'agentName': 'agent 1',
        'date': '2018-03-26',
        'fileName': 'log_2018-3-26.log',
        'warning': {
            'Street': 618,
            'Suite': 470,
            'TargetID': 558,
            'Error loading page frame': 27,
            'writeOverride': 51,
            'State': 53,
            'Zip': 52,
            'PhoneNumber': 5
        },
        'error': {
            'Locations error handling': 12
        }
    },
    {
        'agentName': 'agent 2',
        'date': '2018-03-26',
        'fileName': 'log_2018-3-26.log'
        # No warning/error nested dicts here
    }
]

def flatten_entry(entry):
    # Start with a copy of top-level key-value pairs
    flattened = entry.copy()
    
    # Merge warning dict if present, else fill missing columns with 0
    if 'warning' in flattened:
        flattened.update(flattened.pop('warning'))
    else:
        for col in warning_cols:
            flattened[col] = 0
    
    # Merge error dict if present, else fill missing columns with 0
    if 'error' in flattened:
        flattened.update(flattened.pop('error'))
    else:
        for col in error_cols:
            flattened[col] = 0
    
    return flattened

# First, scan all entries to collect all possible columns
top_level_cols = set()
warning_cols = set()
error_cols = set()

for entry in dlog:
    top_level_cols.update(entry.keys() - {'warning', 'error'})
    if 'warning' in entry:
        warning_cols.update(entry['warning'].keys())
    if 'error' in entry:
        error_cols.update(entry['error'].keys())

# Combine and sort columns for a clean CSV structure
all_cols = sorted(top_level_cols) + sorted(warning_cols) + sorted(error_cols)

# Flatten all log entries
flattened_entries = [flatten_entry(entry) for entry in dlog]

# Write to CSV file
with open('log_output.csv', 'w', newline='', encoding='utf-8') as csvfile:
    writer = csv.DictWriter(csvfile, fieldnames=all_cols)
    writer.writeheader()
    writer.writerows(flattened_entries)

Key Details #

• Dynamic Column Collection: We scan every entry to capture all possible keys from nested dicts—this means if future logs have new warning/error types, the code will automatically include those columns.
• Safe Flattening: Using dict.copy() and pop() ensures we don't modify the original log data while merging nested values into the top-level dict.
• Missing Data Handling: Entries without warning or error get those columns filled with 0 (swap this to '' if empty strings make more sense for your use case).
• Sorted Columns: Sorting the columns makes the CSV easier to scan, but you can skip this step if you want to preserve the order of key appearance.

Output CSV Preview #

The resulting file will have consistent rows, even for entries without nested data:

agentName,date,fileName,Error loading page frame,PhoneNumber,State,Street,Suite,TargetID,Zip,Locations error handling
agent 1,2018-03-26,log_2018-3-26.log,27,5,53,618,470,558,52,12
agent 2,2018-03-26,log_2018-3-26.log,0,0,0,0,0,0,0,0

内容的提问来源于stack exchange，提问作者Derrick Brewer