PySpark调用from_json传入schema报ArrayType无法识别解析错误

开发者阿泽

2026-9-2

PySpark动态schema场景下from_json解析报错问题

问题背景

本问题为PySpark空数组处理相关问题的跟进咨询，首先感谢@abiratis此前的解答。我们在AWS Glue作业中落地对应方案时，与原方案存在场景差异：当前场景无预定义静态schema，因此新增colSchema列存储some-array字段每一项的schema信息，样例数据如下：

+------------------------+-----------------------------------------------------------------------------------------------------------------------+
|some-array              |colSchema                                                                                                              |
+------------------------+-----------------------------------------------------------------------------------------------------------------------+
|[{f1a, f2a}, {f1b, f2b}]|ArrayType(StructType(List(StructField(array-field-1,StringType,true),StructField(array-field-2,StringType,true))),true)|
+------------------------+-----------------------------------------------------------------------------------------------------------------------+

复现代码

我们使用from_json函数做格式转换，核心代码如下：

final_df.select(from_json(col('some-array'), 'ArrayType(StructType(List(StructField(array-field-1,StringType,true),
StructField(array-field-2,StringType,true))),true)' {'allowUnquotedFieldNames':True}).alias('json1')).show(3, False)

报错信息

代码执行时抛出如下异常：

AnalysisException: Cannot parse the schema in JSON format: Unrecognized token 'ArrayType': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
 at [Source: (String)"ArrayType(StructType(List(StructField(array-field-1,StringType,true),StructField(array-field-2,StringType,true))),true)"; line: 1, column: 10]
Failed fallback parsing: Cannot parse the data type: 
mismatched input 'StructType' expecting INTEGER_VALUE(line 1, pos 10)

== SQL ==
ArrayType(StructType(List(StructField(array-field-1,StringType,true),StructField(array-field-2,StringType,true))),true)
----------^^^