使用jq将多文本文件转换为结构化JSON数组的技术问题

需求 #

存在多个包含多组键值对的文件，需要将其转换为结构化JSON数组。

文件内容示例 #

# cat content/1.yaml
time: "2020-09-14T22:33:40Z"
id: ed1d4321
name: One
description: 'Here is number "one"
  this is good'

# cat content/2yaml
time: "2021-09-14T22:33:40Z"
id: eg134841
name: Two
description: 'Here is number "two"
  best of all'
newkey: value

已尝试步骤 #

先将文件合并为包含文件名的内容块：

# for file in $(ls content/*yaml); do echo filename: $file; cat $file; done
filename: content/1.yaml
time: "2020-09-14T22:33:40Z"
id: ed1d4321
name: One
description: 'Here is number "one"
  this is good'
filename: content/2yaml
time: "2021-09-14T22:33:40Z"
id: eg134841
name: Two
description: 'Here is number "two"
  best of all'
newkey: value

随后尝试用jq命令转换：

# for file in $(ls content/*yaml); do echo filename: $file; cat $file; done | jq -Rn '[inputs|split(": ")] | map({(.[0]): .[1]})'
[...（输出内容略）]

待解决问题 #

• 未按文件名分组为独立的JSON对象；
• time字段保留了转义引号，需去除引号保留原始值；
• description字段的多行值未合并为单一值，且需去除单引号。

解决方案 #

使用以下jq命令可同时解决三个问题：

for file in content/*yaml; do echo "filename: $file"; cat "$file"; done | jq -Rn '
  [split("\nfilename: ")[] | select(length>0)]
  | map(
      "filename: " + .
      | split("\n")
      | reduce .[] as $line ({};
          if $line | test("^\\s") then
            last_key as $k | .[$k] += "\n" + ($line | ltrimstr("  "))
          else
            ($line | split(": "; 2)) as [$k, $v]
            | if $v then
                $v |= gsub("^[\'\"](.*)[\'\"]$"; "\1")
                | .[$k] = $v
                | .last_key = $k
              else
                .
              end
          end
        )
      | del(.last_key)
    )
'

命令逻辑说明 #

1. 按文件分组：通过split("\nfilename: ")将输入分割为单个文件的内容块，每个块对应一个JSON对象。
2. 处理多行值：识别以空格开头的行（多行值的延续），将其追加到上一个字段，并去除缩进空格。
3. 清理引号：用gsub去掉字段值首尾的单引号或双引号，保留原始内容。

最终输出示例 #

[
  {
    "filename": "content/1.yaml",
    "time": "2020-09-14T22:33:40Z",
    "id": "ed1d4321",
    "name": "One",
    "description": "Here is number \"one\"\nthis is good"
  },
  {
    "filename": "content/2yaml",
    "time": "2021-09-14T22:33:40Z",
    "id": "eg134841",
    "name": "Two",
    "description": "Here is number \"two\"\nbest of all",
    "newkey": "value"
  }
]

内容的提问来源于stack exchange，提问作者André Letterer