如何用Ansible或Python规范化Terraform输出的扁平数据结构？

Let's tackle this problem with both Ansible (since your team prefers it) and Python (for a more streamlined approach) solutions.

Ansible Solution (Fixing Your Existing Workflow) #

Your current Ansible tasks are already grouping the devices correctly—you just need to clean up those verbose key names. Here's how to adjust your playbook to strip the ebs_block_device.<ID>. prefix from each key:

- name: "Get input from file"
  set_fact:
    device_data: "{{ lookup('file', file_path) | from_json | dict2items }}"

- name: "Extract list of volume id numbers created by terraform"
  vars:
    ebs_regex: "ebs_block_device\\.(\\d*)\\.device_name"
  set_fact:
    volume_id_list: "{{ device_data | selectattr('key', 'match', ebs_regex) | map(attribute='key') | map('regex_replace', ebs_regex, '\\1') | list }}"

- name: "Organize volumes into list with cleaned keys"
  vars:
    query_text: "[?contains(key, '{{ item }}')]"
    single_volume_raw: "{{ device_data | to_json | from_json | json_query(query_text) }}"
    # Clean each key by removing the prefix and ID segment
    single_volume_cleaned: >-
      {{ single_volume_raw | items2dict 
         | dict2items 
         | map('combine', {'key': item.key | regex_replace('^ebs_block_device\\.\\d+\\.', '')}) 
         | items2dict }}
  set_fact:
    volume_data: "{{ volume_data | default([]) + [single_volume_cleaned] }}"
  loop: "{{ volume_id_list }}"

- name: "Finalize structured output"
  set_fact:
    final_output: {"devices": "{{ volume_data }}"}

- debug: var=final_output

How This Works: #

1. After fetching the raw single-device data, we convert it to a list of key-value pairs (dict2items).
2. For each key-value pair, we use regex_replace to strip off the ebs_block_device.<random-number>. prefix, leaving only the attribute name (like device_name).
3. We convert the cleaned key-value pairs back to a dictionary and add it to our volume list.
4. Finally, we wrap the list in a devices key to match your desired output.

Python Solution (For a More Streamlined Approach) #

If you want a simpler, more direct solution, Python can handle this normalization in just a few lines. This avoids Ansible's filter complexity and is easy to adapt:

import json

def normalize_ebs_devices(input_json):
    device_map = {}
    # Iterate over each flat key-value pair
    for full_key, value in input_json.items():
        # Split the key into components: [prefix, device_id, attribute]
        parts = full_key.split('.')
        if len(parts) != 3:
            continue  # Skip any unexpected keys
        
        _, device_id, attribute = parts
        # Initialize a dict for this device if it doesn't exist
        if device_id not in device_map:
            device_map[device_id] = {}
        # Assign the value to the correct attribute
        device_map[device_id][attribute] = value
    
    # Convert the device map values to a list and wrap in the desired structure
    return {"devices": list(device_map.values())}

# Example usage (replace with your input source)
if __name__ == "__main__":
    with open("terraform_output.json", "r") as f:
        input_data = json.load(f)
    
    normalized_data = normalize_ebs_devices(input_data)
    print(json.dumps(normalized_data, indent=2))

Why This Works: #

• We use a dictionary to group all attributes by their unique device ID (the numeric segment in the flat keys).
• For each key, we split out the attribute name and assign it to the corresponding device entry.
• Finally, we convert the grouped device entries into a list and wrap it in the devices key to match your expected output.

Which Should You Choose? #

• Stick with the Ansible solution if you need to integrate this into your existing Ansible workflows or if your team is more comfortable with Ansible syntax.
• Go with the Python solution if you want a faster, less verbose approach that's easy to test and modify independently.

内容的提问来源于stack exchange，提问作者Aiken