深度嵌套数据结构扁平化的性能优化与递归限制问题解决方案咨询

I’m working on a Python project where I need to process a nested data structure. The structure consists of lists and dictionaries, and the nesting level can vary from a few levels to potentially hundreds. I need to flatten this data structure into a single list while preserving the values. However, I am facing performance issues when dealing with deep nesting.

Here is the simplified data structure I’m working with:

data = {
    "name": "John",
    "contacts": [
        {
            "type": "email",
            "value": "john@example.com",
        },
        {
            "type": "phone",
            "value": [
                {
                    "country": "US",
                    "number": "123-456-7890"
                },
                {
                    "country": "UK",
                    "number": "987-654-3210"
                }
            ]
        }
    ],
    "address": {
        "city": "New York",
        "postal_code": "10001",
        "coordinates": [
            {
                "lat": 40.7128,
                "lon": -74.0060
            }
        ]
    }
}

I need to create a function that will flatten this structure such that all values are extracted into a single list. The output for the above input would look something like:

["John", "email", "john@example.com", "phone", "123-456-7890", "US", "987-654-3210", "UK", "New York", "10001", 40.7128, -74.0060]

I’ve tried using recursion, but I’m running into issues with handling very deep structures. Here is my initial attempt:

def flatten(data):
    flat_list = []
    
    if isinstance(data, dict):
        for key, value in data.items():
            flat_list.extend(flatten(value))
    elif isinstance(data, list):
        for item in data:
            flat_list.extend(flatten(item))
    else:
        flat_list.append(data)
    
    return flat_list

flattened_data = flatten(data)
print(flattened_data)

This works fine for small and medium-sized structures, but when the nesting gets deeper (hundreds of levels deep), I run into recursion depth issues and performance bottlenecks.

What I’ve Tried: #

• Increasing the recursion limit with sys.setrecursionlimit() but it only marginally helps and doesn’t fully address the performance concerns.
• Optimizing the recursive function by converting it to an iterative approach, but I’m unsure how to manage the recursion manually for deeply nested structures.

Questions: #

1. How can I improve the recursion or refactor this code to handle much deeper structures efficiently?
2. Is there an iterative way to flatten this data structure without running into recursion depth limitations?
3. Are there any known libraries or patterns that can handle very deep and complex data structures like this more efficiently

The structure is dynamic and may not always follow the same pattern (dictionaries may not always contain the same keys, lists may not always contain the same types of data), so the function should be as generic as possible.

备注：内容来源于stack exchange，提问作者ahmad