如何在PySpark中用正则表达式替换Map类型列的部分键名？

解决方案思路 #

要修改Map类型列的键名，可通过以下三步实现：

• 将Map转换为键值对结构体数组（借助map_entries函数）
• 遍历数组，对每个键执行两次字符串替换：把living_costs替换为lc，把[和]替换为_（用transform+regexp_replace组合实现）
• 将处理后的键值对数组转回Map类型（使用map_from_entries）

完整代码示例 #

from pyspark.sql import SparkSession
from pyspark.sql.functions import map_entries, transform, regexp_replace, map_from_entries

# 初始化SparkSession
spark = SparkSession.builder.appName("MapKeyRename").getOrCreate()

# 创建测试DataFrame
data = [
    (1, 2019, {"living_costs[1]":"","living_costs[2]":"","living_costs[3]":"","living_costs[4]":""}, "2019-09-04T02:32:00.990+0000"),
    (2, 2020, {"living_costs[1]":"","living_costs[2]":"","living_costs[3]":"","living_costs[4]":""}, "2020-09-04T02:32:00.990+0000"),
    (3, 2021, {"living_costs[1]":"","living_costs[2]":"","living_costs[3]":"","living_costs[4]":""}, "2021-09-04T02:32:00.990+0000"),
    (4, 2022, {"living_costs[1]":"","living_costs[2]":"","living_costs[3]":"","living_costs[4]":""}, "2022-09-04T02:32:00.990+0000")
]

df = spark.createDataFrame(data, ["id", "year", "data", "timestamp"])

# 处理data列的键名
processed_df = df.withColumn(
    "data",
    map_from_entries(
        transform(
            map_entries("data"),
            lambda entry: (
                regexp_replace(
                    regexp_replace(entry.key, "living_costs", "lc"),
                    r"[\[\]]", "_"
                ),
                entry.value
            )
        )
    )
)

# 展示结果
processed_df.show(truncate=False)

代码细节解释 #

• map_entries("data")：把data列的Map结构拆成包含key和value字段的结构体数组，方便逐个处理键名
• transform(...)：遍历数组中的每个键值对，嵌套两次regexp_replace完成键名的规则替换
• map_from_entries(...)：将处理后的键值对数组重新组合成Map类型，覆盖原data列

最终输出 #

+---+----+----------------------------------------------------------------+-------------------------------+
|id |year|data                                                            |timestamp                      |
+---+----+----------------------------------------------------------------+-------------------------------+
|1  |2019|{lc_1_ -> , lc_2_ -> , lc_3_ -> , lc_4_ -> }                    |2019-09-04T02:32:00.990+0000   |
|2  |2020|{lc_1_ -> , lc_2_ -> , lc_3_ -> , lc_4_ -> }                    |2020-09-04T02:32:00.990+0000   |
|3  |2021|{lc_1_ -> , lc_2_ -> , lc_3_ -> , lc_4_ -> }                    |2021-09-04T02:32:00.990+0000   |
|4  |2022|{lc_1_ -> , lc_2_ -> , lc_3_ -> , lc_4_ -> }                    |2022-09-04T02:32:00.990+0000   |
+---+----+----------------------------------------------------------------+-------------------------------+

内容的提问来源于stack exchange，提问作者user8178045