如何将Spark DataFrame数组类型列转为字符串且保留元素结构

问题背景 #

需要将Spark DataFrame中的array类型列转换为字符串类型，同时保留该列数据的元素字段名与结构。

DataFrame Schema #

root
 |-- accountId: string (nullable = true)
 |-- documents: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- accountId: string (nullable = true)
 |    |    |-- agreementId: string (nullable = true)
 |    |    |-- createdBy: string (nullable = true)
 |    |    |-- createdDate: string (nullable = true)
 |    |    |-- id: string (nullable = true)
 |    |    |-- obligations: array (nullable = true)
 |    |    |    |-- element: string (containsNull = true)
 |    |    |-- resourceVersion: long (nullable = true)
 |    |    |-- updatedBy: string (nullable = true)
 |    |    |-- updatedDate: string (nullable = true)

示例数据 #

{
    "accountId":"1",
    "documents":{
        "list":[{
            "element":{
                "accountId":"1",
                "agreementId":"1.2",
                "createdDate":"2022-10-06T19:33:42.539646Z",
                "externalId":"16",
                "id":"123",
                "name":"test1.docx",
                "obligations":{},
                "resourceVersion":1,
                "updatedDate":"2022-10-06T19:33:42.680233Z"
            }
        }]
    }
}
{
    "accountId":"2",
    "documents":{
        "list":[{
            "element":{
                "accountId":"2",
                "agreementId":"2.2",
                "createdDate":"2022-10-06T19:33:42.539646Z",
                "externalId":"18",
                "id":"123",
                "name":"test2.docx",
                "obligations":{},
                "resourceVersion":1,
                "updatedDate":"2022-10-06T19:33:42.680233Z"
            }
        }]
    }
}

当前代码及问题 #

使用直接强制转换的方式：

df_string = df.select([col(c).cast("string") for c in df.columns])

问题：转换后documents列的结构体字段名丢失，变成无键值的元组形式：

{
    "accountId":"1",
    "documents":[{"1","1.2","2022-10-06T19:33:42.539646Z","16","123","test1.docx","",1,"2022-10-06T19:33:42.680233Z"}]
}
{
    "accountId":"2",
    "documents":[{"2","2.2","2022-10-06T19:33:42.539646Z","18","123","test2.docx","","1","2022-10-06T19:33:42.680233Z"}]
}

期望结果 #

转换后documents列保留完整字段名与结构：

{
    "accountId":"1",
    "documents":[{"accountId":"1","agreementId":"1.2","createdDate":"2022-10-06T19:33:42.539646Z","externalId":"16","id":"123","name":"test1.docx","obligations":"","resourceVersion":"1","updatedDate":"2022-10-06T19:33:42.680233Z"}]
}
{
    "accountId":"2",
    "documents":[{"accountId":"2","agreementId":"2.2","createdDate":"2022-10-06T19:33:42.539646Z","externalId":"18","id":"123","name":"test2.docx","obligations":"","resourceVersion":"1","updatedDate":"2022-10-06T19:33:42.680233Z"}]
}

解决方案 #

直接cast("string")会丢失复杂类型的结构信息，应使用to_json函数将数组列序列化为JSON格式的字符串，完整保留字段名与层级结构。

方式1：仅转换指定数组列 #

from pyspark.sql.functions import to_json, col

# 将documents列转为JSON字符串，其他列保持原类型（如需转字符串可额外cast）
df_string = df.withColumn("documents", to_json(col("documents")))

方式2：自动识别所有数组列转换 #

如果DataFrame中有多个数组列，可遍历列类型自动处理：

from pyspark.sql.functions import to_json, col

converted_columns = []
for col_name in df.columns:
    col_type = df.schema[col_name].dataType.typeName()
    if col_type == "array":
        # 数组列用to_json转字符串
        converted_columns.append(to_json(col(col_name)).alias(col_name))
    else:
        # 非数组列直接cast为字符串
        converted_columns.append(col(col_name).cast("string").alias(col_name))

df_string = df.select(converted_columns)

说明：to_json函数会将array、struct等复杂类型转换为标准JSON格式的字符串，完全保留字段名与数据结构，符合期望结果。

内容的提问来源于stack exchange，提问作者bda