PySpark中按名称选择数组类型列报错‘列不存在’的求助

我在PySpark中选择DataFrame里的结构体数组类型列documents时，遇到AnalysisException: Column 'documents' does not exist错误，但该列实际存在且有数据，能通过索引选中，却不能通过名称选中，求解决方法。

DataFrame Schema #

root
 |-- accountId: string (nullable = true)
 |-- documents: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- accountId: string (nullable = true)
 |    |    |-- agreementId: string (nullable = true)
 |    |    |-- createdBy: string (nullable = true)
 |    |    |-- createdDate: string (nullable = true)
 |    |    |-- documentType: string (nullable = true)
 |    |    |-- externalId: string (nullable = true)
 |    |    |-- externalSource: string (nullable = true)
 |    |    |-- id: string (nullable = true)
 |    |    |-- name: string (nullable = true)
 |    |    |-- obligations: array (nullable = true)
 |    |    |    |-- element: struct (containsNull = true)
 |    |    |    |    |-- accountId: string (nullable = true)
 |    |    |    |    |-- agreementId: string (nullable = true)
 |    |    |    |    |-- createdBy: string (nullable = true)
 |    |    |    |    |-- createdDate: string (nullable = true)
 |    |    |    |    |-- description: string (nullable = true)
 |    |    |    |    |-- documentId: string (nullable = true)
 |    |    |    |    |-- dueDate: string (nullable = true)
 |    |    |    |    |-- externalId: string (nullable = true)
 |    |    |    |    |-- id: string (nullable = true)
 |    |    |    |    |-- name: string (nullable = true)
 |    |    |    |    |-- partyId: string (nullable = true)
 |    |    |    |    |-- reminderPeriodUnit: long (nullable = true)
 |    |    |    |    |-- resourceVersion: long (nullable = true)
 |    |    |    |    |-- status: long (nullable = true)
 |    |    |    |    |-- updatedBy: string (nullable = true)
 |    |    |    |    |-- updatedDate: string (nullable = true)
 |    |    |-- resourceVersion: long (nullable = true)
 |    |    |-- updatedBy: string (nullable = true)
 |    |    |-- updatedDate: string (nullable = true)
 |-- effectiveDate: string (nullable = true)
 |-- parties: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- agreementContacts: array (nullable = true)
 |    |    |    |-- element: struct (containsNull = true)
 |    |    |    |    |-- contactId: string (nullable = true)
 |    |    |    |    |-- isPrimary: boolean (nullable = true)
 |    |    |    |    |-- role: string (nullable = true)
 |    |    |-- partyId: string (nullable = true)
 |    |    |-- role: string (nullable = true)
 |-- updatedBy: string (nullable = true)
 |-- updatedDate: string (nullable = true)

可行操作：通过索引选中列 #

df.select(df.columns[2]).show(truncate=False)

执行结果：

+--------------------+
|           documents|
+--------------------+
|[{d73db7ba-5329-4...|
|[{d73db7ba-5329-4...|
|[{d73db7ba-5329-4...|
|[{d73db7ba-5329-4...|
|[{d73db7ba-5329-4...|
+--------------------+

报错操作：按名称选中列 #

df.select("documents").show(truncate=False)

报错信息：

AnalysisException: Column 'documents' does not exist.

解决方案 #

1. 检查列名是否含隐形字符 #

列名可能存在空格、制表符等不可见字符，导致表面名称与实际不匹配。验证方式：

for col in df.columns:
    print(f"列名: '{col}', 长度: {len(col)}")

若发现长度异常，重命名列修复：

df = df.withColumnRenamed(df.columns[2], "documents")

2. 使用Column对象选择列 #

直接通过df["列名"]或df.列名引用列，绕过字符串匹配问题：

df.select(df["documents"]).show(truncate=False)
# 或
df.select(df.documents).show(truncate=False)

3. 检查元数据异常 #

偶尔DataFrame元数据会出现混乱，尝试重新读取数据源或重建DataFrame：

# 根据实际数据源调整读取方式
df = spark.read.json("your_data_source_path")

4. 排查嵌套列冲突 #

若DataFrame内存在同名嵌套列，可能导致字符串选择时混淆。通过df.columns确认顶层列的准确名称，或使用完整路径选择（若为嵌套列）。

内容的提问来源于stack exchange，提问作者bda