PySpark SQL数组索引越界错误排查：AWS MWAA+EMR Serverless场景

在AWS MWAA+EMR Serverless环境执行PySpark SQL插入查询时，触发Index 1 out of bounds for length 1数组索引越界错误，但查询语句中无显式数组引用。

失败的查询语句 #

INSERT INTO TABLE dev.aggregates
  PARTITION (p_date='2024-03-03')
  BY NAME
  SELECT '13x1' AS dimensions
  FROM dev.other_aggregates
  LIMIT 3

可成功执行的简化语句 #

移除查询中的FROM dev.other_aggregates LIMIT 3部分后，语句能正常执行：

INSERT INTO TABLE dev.aggregates
  PARTITION (p_date='2024-03-03')
  BY NAME
  SELECT '13x1' AS dimensions

目标表dev.aggregates的建表定义 #

CREATE EXTERNAL TABLE `dev.aggregates`(
  `dimensions` string COMMENT '')
PARTITIONED BY ( 
  `p_date` varchar(11) COMMENT '')
ROW FORMAT SERDE 
  'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' 
STORED AS INPUTFORMAT 
  'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat' 
OUTPUTFORMAT 
  'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'
LOCATION
  's3://airflow-dev/tables/aggregates/parquet'
TBLPROPERTIES (
  'parquet.compression'='SNAPPY',
  'projection.enabled'='true',
  'projection.p_date.format'='yyyy-MM-dd',
  'projection.p_date.interval'='1',
  'projection.p_date.interval.unit'='DAYS',
  'projection.p_date.range'='2023-12-01,NOW',
  'projection.p_date.type'='date',
  'storage.location.template'='s3://airflow-dev/tables/aggregates/parquet/p_date=${p_date}')

部分堆栈跟踪信息 #

File "/tmp/spark-9ef48e87-4e57-4225-a373-6579254c3f89/spark_job_runner.py", line 66, in <module>
    run_spark_commands(spark, sql_commands)
File "/tmp/spark-9ef48e87-4e57-4225-a373-6579254c3f89/spark_job_runner.py", line 41, in run_spark_commands
    spark_runner.sql(command_string)
File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/session.py", line 1631, in sql
File "/usr/lib/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py", line 1322, in __call__
File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/errors/exceptions/captured.py", line 179, in deco
File "/usr/lib/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/protocol.py", line 326, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o101.sql.
: java.lang.IndexOutOfBoundsException: Index 1 out of bounds for length 1
    at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64)
    at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70)
    at java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:266)
    at java.base/java.util.Objects.checkIndex(Objects.java:361)
    at java.base/java.util.ArrayList.get(ArrayList.java:427)
    at org.apache.hadoop.hive.ql.metadata.Table.createSpec(Table.java:881)
    at org.apache.hadoop.hive.ql.metadata.Table.createSpec(Table.java:873)
    at org.apache.hadoop.hive.ql.metadata.Partition.getSpec(Partition.java:416)
    at org.apache.spark.sql.hive.client.HiveClientImpl$.fromHivePartition(HiveClientImpl.scala:1245)
    at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$getPartitions$4(HiveClientImpl.scala:832)
    at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
    at scala.collection.Iterator.foreach(Iterator.scala:943)
...

排查线索 #

• 检查源表dev.other_aggregates的分区配置：错误发生在Spark读取源表分区元数据阶段，可能源表的分区列定义与实际分区目录结构不匹配，或是分区目录命名不符合Hive规范。
• 验证源表的TBLPROPERTIES：源表若启用了分区投影但配置有误，会导致Spark解析分区时访问不存在的索引。对比目标表的投影配置，检查源表参数是否正确。
• 确认Spark/Hive版本兼容性：堆栈涉及Hive的Table.createSpec方法，可能是Spark与Hive元数据版本不兼容导致的解析逻辑错误，需确认MWAA与EMR Serverless的版本匹配。
• 排查源表分区数据：执行SHOW PARTITIONS dev.other_aggregates查看源表分区是否正常，若存在损坏的分区记录或异常分区数量，可能触发元数据读取异常。
• 测试源表读取：执行SELECT * FROM dev.other_aggregates LIMIT 3，确认错误是否出在读取源表阶段而非插入阶段。
• 检查目标表分区权限：涉及源表读取时，Spark可能触发目标表分区元数据的额外校验，需确认S3目录权限及EMR Serverless执行角色的读写权限是否完整。

内容的提问来源于stack exchange，提问作者SomeDude