如何批量转换Bash脚本中Awk提取的马里奥制造关卡ID

我正在编写基础Bash脚本，用于搜索《超级马里奥制造1》的courses.jsonl关卡数据库文件，将搜索到的关卡ID通过给定的Python脚本mmid.py转换为游戏内实际ID。当前代码可实现搜索、提取单个ID并完成转换，但无法批量处理搜索到的所有ID。我仅有基础Bash知识，想了解是否用for循环即可实现，还是需要更复杂的方案。

现有Bash代码：

#!/bin/sh
id=$(grep $1 /file/path/courses.jsonl | awk -F ',' '{print $1}' | awk -F ':' '{print $2}')

python3 /file/path/mmid.py $id

mmid.py代码（已修改为通过命令行参数传入ID）：

import struct
import hashlib
import hmac
import sys
idno = int(sys.argv[1])
key = hashlib.md5(b"9f2b4678").digest()
data = struct.pack("<Q", idno)
checksum = hmac.HMAC(key, data, 'md5').digest()
checksum = checksum[3:1:-1].hex().upper()
idstring = str.upper(hex(idno))[2:]
for y in range(8 - len(idstring)):
    idstring = '0' + str(idstring)
code = str(checksum) + '0000' + str(idstring)
print(code)

courses.jsonl单条数据示例（需提取的ID为首个字段）：

{"id":69558108,"retrieval_date":"2021-06-05T08:04:59.154776Z","url":"https://d2sno3mhmk1ekx.cloudfront.net/10.WUP_AMAJ_datastore/ds/1/data/00069558108-00001?Expires=1625472299&Signature=BqxLU18d~4rV4FeHNN-Sm2dbKqKyLhS5qX2liHPqr3MN9Y0Fgt5GCYCXrseFt-tinj4LVjrtMIWZW4G1ijj4FSHQyXN9OoW3PfyZbSuatl9~aNHLPtCCBfMDm7HmF~Njry1pvFjvmeMTJf4ZlUTrKIJF00gfA~2Mu2K1qjIR4yqSQCyxsxG61J~KG3CQMVJUKnbdGupeHN5-nigDqZ3EbT3-odEmjJsq04wb8K98iNMVHKINg8MZwuWHtEtECGWWqvhAvkdRDj6ZZQxAPW1fwxkLJBY4PSRW28A20T-lcq0CDDeU4yFUzeeE6GqFV0xvwj2IOFrlWVOo3McmE21HqA__&Key-Pair-Id=APKAJUYKVK3BE6ZPNZBQ","stars":1,"course_name":"hello kitty saves xmas by Lario™","creator":{"pid":1745383521,"nnid":"realdealLario","mii_data":"QlBGQwAAAAEAAAAAAAAAAAAAAAAAAQAAAwAAQMHCaSxE5iEQ2lRGSOgTQXuGIgAAiANMAGEAcgBpAG8AIiEAAAAAAAAAACNiKABCAQPHRBgLM0cQDg8RsCoAACIAUkhQAABlAAAAcgBlAGEAawAAAAAAAAAAAO1hAAAAAAAAAAAAAAAAAAAAAAAAAAQ="},"upload_time":"2021-03-30T22:59:39Z","user_plays":2,"clears":1,"total_attempts":5,"failures":4,"world_record":{"best_time_player":{"pid":1747253384,"nnid":"ilovesnowandcold","mii_data":"QlBGQwAAAAEAAAAAAAAAAAAAAAAAAQAAAwAAEGpkmkWgRBCggeC0lA/Q2aSlewAAghVLAGUAaQB0AGgAAAAAAAAAAAAAAAsVAAAsApJuRBgnNEcYhhIOaA8AACnQUUhQSwBlAGkAdABoAAAAAAAAAAAAAAAAAJLUAAAAAAAAAAAAAAAAAAAAAAAAAAE="},"first_complete_player":{"pid":1747253384,"nnid":"ilovesnowandcold","mii_data":"QlBGQwAAAAEAAAAAAAAAAAAAAAAAAQAAAwAAEGpkmkWgRBCggeC0lA/Q2aSlewAAghVLAGUAaQB0AGgAAAAAAAAAAAAAAAsVAAAsApJuRBgnNEcYhhIOaA8AACnQUUhQSwBlAGkAdABoAAAAAAAAAAAAAAAAAJLUAAAAAAAAAAAAAAAAAAAAAAAAAAE="},"best_time_ms":13245,"created_time":"2021-03-30T23:52:40Z","updated_time":"2021-03-30T23:52:40Z"}}

注：该courses.jsonl文件大小超过16GB。

方法1：Bash For循环实现批量处理 #

用for循环遍历所有提取到的ID，逐个调用Python脚本转换，适合小范围搜索场景：

#!/bin/sh
# 给搜索关键词加双引号，避免特殊字符/空格导致错误
for id in $(grep "$1" /file/path/courses.jsonl | awk -F ',' '{print $1}' | awk -F ':' '{print $2}'); do
    python3 /file/path/mmid.py "$id"
done

局限性 #

每处理一个ID就启动一次Python进程，在16GB大文件下会显著拖慢处理速度。

方法2：批量输入优化（推荐） #

修改Python脚本支持从标准输入读取所有ID，仅启动一次进程处理，大幅提升效率：

步骤1：更新mmid.py #

import struct
import hashlib
import hmac
import sys

def convert_id(idno):
    key = hashlib.md5(b"9f2b4678").digest()
    data = struct.pack("<Q", idno)
    checksum = hmac.HMAC(key, data, 'md5').digest()
    checksum = checksum[3:1:-1].hex().upper()
    # 用zfill简化补零逻辑
    idstring = str.upper(hex(idno))[2:].zfill(8)
    return f"{checksum}0000{idstring}"

# 读取标准输入的每一行ID并转换
for line in sys.stdin:
    line = line.strip()
    if line:
        try:
            print(convert_id(int(line)))
        except ValueError:
            print(f"无效ID: {line}", file=sys.stderr)

步骤2：更新Bash脚本 #

#!/bin/sh
grep "$1" /file/path/courses.jsonl | awk -F ',' '{print $1}' | awk -F ':' '{print $2}' | python3 /file/path/mmid.py

方法3：Awk一站式提取+批量处理（最优） #

用Awk直接完成搜索和ID提取，减少管道步骤，进一步优化大文件处理速度：

#!/bin/sh
# Awk内置搜索和分割，无需额外调用grep
awk -v keyword="$1" 'index($0, keyword) {split($1, id_part, ":"); print id_part[2]}' /file/path/courses.jsonl | python3 /file/path/mmid.py

内容的提问来源于stack exchange，提问作者Jonah Alexander