AWS Glue Notebook中spark.sql.extensions配置未生效求助

环境配置 #

%idle_timeout 2880
%glue_version 4.0
%worker_type G.1X
%number_of_workers 2

from pyspark import SparkContext, SparkConf
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
from pyspark.sql import SparkSession

conf = SparkConf()
conf.setAppName("MyAppName")
conf.set('spark.sql.extensions', 'org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions')
conf.set('spark.sql.defaultCatalog', 'glue_catalog')
conf.set('spark.sql.catalog.glue_catalog', 'org.apache.iceberg.spark.SparkCatalog')
conf.set('spark.sql.catalog.glue_catalog.catalog-impl', 'org.apache.iceberg.aws.glue.GlueCatalog')
conf.set('spark.sql.catalog.glue_catalog.io-impl', 'org.apache.iceberg.aws.s3.S3FileIO')
conf.set('spark.sql.iceberg.handle-timestamp-without-timezone', 'true')
conf.set('spark.sql.catalog.glue_catalog.warehouse', 'S3_PATH_TO_WAREHOUSE')
conf.set('spark.jars', 'S3_PATH_TO_JARS/bundle-2.23.5.jar,S3_PATH_TO_JARS/iceberg-spark-runtime-3.3_2.12-1.6.1.jar,S3_PATH_TO_JARS/spark-xml_2.12-0.18.0.jar')

spark = SparkSession.builder.config(conf=conf).getOrCreate()

sc = SparkContext.getOrCreate(conf=conf)  # 测试过.getOrCreate()，无效
glueContext = GlueContext(sc)
job = Job(glueContext)

问题现象 #

• 除spark.sql.extensions外，其他Spark配置均正常生效
• 执行验证代码后，spark.sql.extensions返回None
• 执行Iceberg的CALL存储过程时触发语法错误，常规SQL（SELECT/ALTER TABLE等）可正常运行

验证代码 #

print(spark.conf.get('spark.sql.catalog.glue_catalog'))
print(spark.conf.get('spark.sql.extensions'))

验证输出 #

org.apache.iceberg.spark.SparkCatalog
None

错误示例（执行Iceberg存储过程） #

sql = "call glue_catalog.system.rollback_to_snapshot('db.table_name', 123456)"
df = spark.sql(sql)

报错信息 #

ParseException: 
Syntax error at or near 'call'(line 1, pos 0)

== SQL ==
call glue_catalog.system.rollback_to_snapshot('db.table_name', 123456)
^^^

已知：db.table_name表存在，123456为有效快照ID。

解决思路 #

1. 改用Glue Notebook魔法命令配置核心参数 #

AWS Glue Notebook中，部分Spark核心配置需通过%conf魔法命令提前设置，而非代码中用SparkConf配置。在笔记本最顶部添加所有配置：

%idle_timeout 2880
%glue_version 4.0
%worker_type G.1X
%number_of_workers 2
%conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
%conf spark.sql.defaultCatalog=glue_catalog
%conf spark.sql.catalog.glue_catalog=org.apache.iceberg.spark.SparkCatalog
%conf spark.sql.catalog.glue_catalog.catalog-impl=org.apache.iceberg.aws.glue.GlueCatalog
%conf spark.sql.catalog.glue_catalog.io-impl=org.apache.iceberg.aws.s3.S3FileIO
%conf spark.sql.iceberg.handle-timestamp-without-timezone=true
%conf spark.sql.catalog.glue_catalog.warehouse=S3_PATH_TO_WAREHOUSE
%conf spark.jars=S3_PATH_TO_JARS/bundle-2.23.5.jar,S3_PATH_TO_JARS/iceberg-spark-runtime-3.3_2.12-1.6.1.jar,S3_PATH_TO_JARS/spark-xml_2.12-0.18.0.jar

from pyspark import SparkContext, SparkConf
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()
sc = SparkContext.getOrCreate()
glueContext = GlueContext(sc)
job = Job(glueContext)

注意：魔法命令必须放在所有代码（包括导入语句）之前。

2. 检查Iceberg版本兼容性 #

Glue 4.0基于Spark 3.3，当前使用的Iceberg 1.6.1需确保与Spark版本完全匹配（iceberg-spark-runtime-3.3_2.12）。若问题仍存在，可尝试降级到Iceberg 1.5.0或升级到1.7.0。

3. 优化SparkContext初始化顺序 #

代码中先创建SparkSession再获取SparkContext可能导致配置覆盖，改为通过GlueContext获取SparkSession：

glueContext = GlueContext(SparkContext.getOrCreate())
spark = glueContext.spark_session

4. 排查JAR包冲突 #

确保spark.jars中的依赖无冲突，可尝试移除非必要的JAR（如bundle-2.23.5.jar），仅保留Iceberg运行时和spark-xml依赖，再测试配置是否生效。

内容的提问来源于stack exchange，提问作者Ahmad