使用jq streaming过滤嵌套列表并保留JSON全局结构

If you're dealing with a massive JSON file and can't load the whole thing into memory, streaming parsing is the way to go. Let's walk through how to filter out elements in the filter_this array where keep is "false", while keeping every other part of the structure exactly as it was.

Step 1: Grab the Right Tool #

We’ll use Python’s ijson library—it’s built specifically for streaming JSON parsing, so it processes the file in chunks instead of loading everything at once. First, install it:

pip install ijson

Step 2: Full Working Script #

Here’s a script that handles the streaming filter while preserving your original JSON structure:

import ijson
import json

def filter_large_json(input_path, output_path):
    with open(input_path, 'rb') as infile, open(output_path, 'w') as outfile:
        # Track our position in the JSON structure
        inside_filter_array = False
        first_top_level_key = True

        # Start writing the root object
        outfile.write('{')

        for prefix, event, value in ijson.parse(infile):
            # Handle top-level keys like "keep_untouched" and "filter_this"
            if prefix == '' and event == 'map_key':
                if not first_top_level_key:
                    outfile.write(',')
                first_top_level_key = False
                # Write the key to output
                outfile.write(f'"{value}"')
                outfile.write(':')

                if value == 'filter_this':
                    # We're entering the array we need to filter
                    inside_filter_array = True
                    outfile.write('[')
                    first_array_element = True
                else:
                    # For other keys, copy their entire value as-is
                    sub_value = ijson.items(infile, prefix).__next__()
                    json.dump(sub_value, outfile)
            
            # Handle objects inside the filter_this array
            elif inside_filter_array and event == 'start_map':
                # Parse the full object from the stream
                obj = ijson.items(infile, prefix).__next__()
                # Only keep it if keep isn't "false"
                if obj.get('keep') != 'false':
                    if not first_array_element:
                        outfile.write(',')
                    first_array_element = False
                    json.dump(obj, outfile)
            
            # Close the filter_this array when we reach its end
            elif inside_filter_array and event == 'end_array':
                inside_filter_array = False
                outfile.write(']')

        # Close the root object
        outfile.write('}')

# Test with your sample input
if __name__ == '__main__':
    sample_input = '''{ "keep_untouched": { "keep_this": [ "this", "list" ] }, "filter_this": [ {"keep" : "true"}, { "keep": "true", "extra": "keeper" } , { "keep": "false", "extra": "non-keeper" } ] }'''
    with open('sample_input.json', 'w') as f:
        f.write(sample_input)
    
    filter_large_json('sample_input.json', 'sample_output.json')
    
    # Print the cleaned result
    with open('sample_output.json', 'r') as f:
        print(json.dumps(json.load(f), indent=2))

Step 3: How This Works #

• Memory Efficiency: ijson.parse() reads the input file in small chunks, so even multi-GB files won’t hog your RAM.
• Structure Preservation: We manually write the JSON’s braces and brackets to ensure every part of the file except the filtered elements stays identical to the original.
• Targeted Filtering: When we detect we’re inside the filter_this array, we check each object’s keep value—only write it to the output if it’s not "false".

Step 4: Sample Output #

Running the script with your example input will produce this cleaned JSON:

{
  "keep_untouched": {
    "keep_this": [
      "this",
      "list"
    ]
  },
  "filter_this": [
    {
      "keep": "true"
    },
    {
      "keep": "true",
      "extra": "keeper"
    }
  ]
}

This approach is perfect for large files where loading everything into memory isn’t an option, and it keeps your JSON structure intact exactly as you need.

内容的提问来源于stack exchange，提问作者Pete C