使用jq/yq合并对象时，如何对子数组进行唯一值合并？

我需要合并.environments.default和.environments.dev两个对象，规则是dev的值覆盖default对应值，default补充dev未定义的内容。用.environments.default * .environments.dev时，多数对象合并正常，但env和secrets下的数组会被dev完全覆盖。我希望这类数组能按唯一标识字段合并（比如env按name，secrets按secretID/secretUUID），而且代码里不能硬编码"env"或"secrets"这些键名。

源数据 #

{
"environments": {
    "default": {
      "config": {
        "security": {
          "user": 0,
          "group": 0
        },
        "resources": {
          "limits": {
            "memory": "256Mi",
            "cpu": "500m"
          }
        },
        "env": [
          {
            "name": "DEFAULT_ENV",
            "value": "DEFAULT_ENV_VALUE"
          },
          {
            "name": "BASE_URL",
            "value": "DEFAULT_BASE_URL"
          }
        ],
        "secrets": [
          {
            "secretID": "567",
            "mountPath": "/default/dir/to/567/path"
          },
          {
            "secretID": "123",
            "mountPath": "/default/dir/to/123/path"
          },
          {
            "secretUUID": "789",
            "mountPath": "/default/dir/to/789/path"
          }
        ]
      }
    },
    "dev": {
      "config": {
        "replicas": 1,
        "resources": {
          "limits": {
            "cpu": "5000m"
          }
        },
        "env": [
          {
            "name": "BASE_URL",
            "value": "DEV_BASE_URL"
          },
          {
            "name": "DEV_ENV_1",
            "value": "DEV_ENV_1"
          }
        ],
        "secrets": [
          {
            "secretID": "456",
            "mountPath": "/dev/dir/to/456/path"
          },
          {
            "secretID": "123",
            "mountPath": "/dev/dir/to/456/path"
          }
        ]
      }
    }
  }
}

当前输出（使用.environments.default * .environments.dev） #

{
  "config": {
    "security": {
      "user": 0,
      "group": 0
    },
    "resources": {
      "limits": {
        "memory": "256Mi",
        "cpu": "5000m"
      }
    },
    "env": [
      {
        "name": "BASE_URL",
        "value": "DEV_BASE_URL"
      },
      {
        "name": "DEV_ENV_1",
        "value": "DEV_ENV_1"
      }
    ],
    "secrets": [
      {
        "secretID": "456",
        "mountPath": "/dev/dir/to/456/path"
      },
      {
        "secretID": "123",
        "mountPath": "/dev/dir/to/456/path"
      }
    ],
    "replicas": 1
  }
}

期望输出 #

{
  "config": {
    "security": {
      "user": 0,
      "group": 0
    },
    "resources": {
      "limits": {
        "memory": "256Mi",
        "cpu": "5000m"
      }
    },
    "env": [
      {
        "name": "BASE_URL",
        "value": "DEV_BASE_URL"
      },
      {
        "name": "DEV_ENV_1",
        "value": "DEV_ENV_1"
      },
      {
        "name": "DEFAULT_ENV",
        "value": "DEFAULT_ENV_VALUE"
      }
    ],
    "secrets": [
      {
        "secretID": "456",
        "mountPath": "/dev/dir/to/456/path"
      },
      {
        "secretID": "123",
        "mountPath": "/dev/dir/to/456/path"
      },
      {
        "secretID": "567",
        "mountPath": "/default/dir/to/567/path"
      },
      {
        "secretUUID": "789",
        "mountPath": "/default/dir/to/789/path"
      }
    ],
    "replicas": 1
  }
}

1. 使用jq实现 #

核心是自定义递归合并逻辑，遇到数组时自动合并并按对象的唯一键去重（保留dev的条目优先）。

执行以下jq命令：

def merge_with_unique_arrays($base; $override):
  def deep_merge($a; $b):
    if $a | type == "object" and $b | type == "object" then
      ($a + $b) | map_values(
        if . == $a[keys[]] and $b[keys[]] | type == "array" then
          ($b[keys[]] + ($a[keys[]] | map(select(
            . as $item | $b[keys[]] | any(. == $item) | not
          ))))
        else
          deep_merge($a[keys[]]; $b[keys[]])
        end
      )
    elif $b != null then $b
    else $a
    end;
  deep_merge($base; $override);

merge_with_unique_arrays(.environments.default; .environments.dev)

逻辑说明 #

• 递归遍历对象，常规字段按dev覆盖default的规则合并。
• 遇到数组时，先将dev的数组放在前面，再添加default中未在dev数组出现过的条目，自动实现按对象内容去重，无需硬编码数组键名。

2. 使用yq实现（适用于YAML文件） #

如果配置是YAML格式，yq（v4版本以上）的语法更简洁，同样支持自动识别数组并去重：

假设配置文件为config.yaml，执行命令：

yq eval '.environments.dev * .environments.default | (.. | select(type == "array") | unique_by(keys[]))' config.yaml

逻辑说明 #

• .environments.dev * .environments.default先完成基础对象合并，dev字段覆盖default。
• (.. | select(type == "array") | unique_by(keys[]))递归遍历所有数组，按数组中对象的唯一键（自动识别每个对象的标识字段，如env的name、secrets的secretID）去重，保留最后出现的条目（即dev的覆盖default的）。

内容的提问来源于stack exchange，提问作者Chris F