PySpark为DataFrame StructType字段添加注释报错如何解决

你传入的第四个参数不符合StructField的参数要求：StructField的第四个参数为metadata，必须传入键值对格式的字典对象，而非直接传入注释字符串。你直接传入val1这类字符串时，Spark无法将其解析为符合规范的元数据结构，因此抛出JSON转换失败的异常。同样，StructType.add()方法也不能直接传入注释字符串作为第四个参数，需遵循元数据格式要求。

要为字段添加注释，需要将注释内容放在metadata参数的comment键对应的值中，两种定义Schema的方式修改后均可正常运行：

import pyspark
from pyspark.sql import SparkSession
from pyspark.sql.types import StructType,StructField, StringType, IntegerType

spark = SparkSession.builder.master("local[1]") \
                    .appName('SparkByExamples.com') \
                    .getOrCreate()

data = [("James","","Smith","36636","M",3000),
    ("Michael","Rose","","40288","M",4000),
    ("Robert","","Williams","42114","M",4000),
    ("Maria","Anne","Jones","39192","F",4000),
    ("Jen","Mary","Brown","","F",-1)
  ]

# 方式1：直接通过StructType构造Schema
schema = StructType([ 
    StructField("firstname",StringType(),True, {"comment": "val1"}), 
    StructField("middlename",StringType(),True, {"comment": "val2"}), 
    StructField("lastname",StringType(),True, {"comment": "val3"}), 
    StructField("id", StringType(), True, {"comment": "val4"}), 
    StructField("gender", StringType(), True, {"comment": "val5"}), 
    StructField("salary", IntegerType(), True, {"comment": "val6"}) 
  ])

# 方式2：通过add方法构造Schema
# schema= StructType()\
#     .add("firstname",StringType(),True, {"comment": "val1"})\
#     .add("middlename",StringType(),True, {"comment": "val2"})\
#     .add("lastname",StringType(),True, {"comment": "val3"})\
#     .add("id", StringType(), True, {"comment": "val4"})\
#     .add("gender", StringType(), True, {"comment": "val5"})\
#     .add("salary", IntegerType(), True, {"comment": "val6"})
          
df = spark.createDataFrame(data=data,schema=schema)
df.printSchema()
df.show(truncate=False)

运行上述代码后，df.printSchema()就会输出你期望的带字段注释的结果。

内容的提问来源于stack exchange，提问作者IamDataEngineer