从AWS Glue连接Snowflake失败，报scala类缺失错误

问题现象 #

使用AWS Glue连接Snowflake时无法读取表，抛出java.lang.NoClassDefFoundError: scala/$less$colon$less错误，完整错误栈如下：

23/02/14 01:32:55 INFO Utils: Successfully started service 'sparkDriver' on port 38325.
23/02/14 01:32:59 INFO GlueContext: GlueMetrics configured and enabled
23/02/14 01:33:01 ERROR ProcessLauncher: Error from Python:Traceback (most recent call last):
  File "/tmp/TestSFConn.py", line 111, in <module>
    .option("dbtable", snowflake_database+"."+snowflake_schema+"."+source_table_name).load()
  File "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/sql/readwriter.py", line 210, in load
    return self._df(self._jreader.load())
  File "/opt/amazon/spark/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py", line 1305, in __call__
    answer, self.gateway_client, self.target_id, self.name)
  File "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 111, in deco
    return f(*a, **kw)
  File "/opt/amazon/spark/python/lib/py4j-0.10.9-src.zip/py4j/protocol.py", line 328, in get_return_value
    format(target_id, ".", name), value)
py4j.protocol.Py4JJavaError: An error occurred while calling o104.load.
: java.lang.NoClassDefFoundError: scala/$less$colon$less
    at net.snowflake.spark.snowflake.DefaultSource.shortName(DefaultSource.scala:44)
    at org.apache.spark.sql.execution.datasources.DataSource$.$anonfun$lookupDataSource$2(DataSource.scala:659)
    at org.apache.spark.sql.execution.datasources.DataSource$.$anonfun$lookupDataSource$2$adapted(DataSource.scala:659)
    at scala.collection.TraversableLike.$anonfun$filterImpl$1(TraversableLike.scala:247)
    at scala.collection.Iterator.foreach(Iterator.scala:937)
    at scala.collection.Iterator.foreach$(Iterator.scala:937)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1425)
    at scala.collection.IterableLike.foreach(IterableLike.scala:70)
    at scala.collection.IterableLike.foreach$(IterableLike.scala:69)
    at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
    at scala.collection.TraversableLike.filterImpl(TraversableLike.scala:246)
    at scala.collection.TraversableLike.filterImpl$(TraversableLike.scala:244)
    at scala.collection.AbstractTraversable.filterImpl(Traversable.scala:104)
    at scala.collection.TraversableLike.filter(TraversableLike.scala:258)

已配置内容 #

已在Glue作业详情的「依赖JAR路径」中添加以下JAR文件：

s3://aws-glue-poc/snowflake_files/spark-snowflake_2.13-2.11.1-spark_3.3.jar,
s3://aws-glue-poc/snowflake_files/snowflake-jdbc-3.13.27.jar

使用的代码 #

args = getResolvedOptions(sys.argv, ['JOB_NAME'])
sc = SparkContext()
sc.setLogLevel("ALL")
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'], args)

print("Spark session created")

try:
    
    SNOWFLAKE_SOURCE_NAME = "net.snowflake.spark.snowflake"
    snowflake_database="DEV_123"
    snowflake_schema="schema123"
    source_table_name="TABLE1"
    
    snowflake_options = {
        "sfURL": "XXXXXXXXXXXXXXXXXXXX.snowflakecomputing.com",
        "sfUser": "USER1",
        "sfPassword": "1234567",
        "sfDatabase": snowflake_database,
        "sfSchema": snowflake_schema,
        "sfWarehouse": "WAREHOUSE_1234",
        "tracing" : "ALL"
    }
    
    print("12345 - Before Read")
    df = spark.read\
        .format(SNOWFLAKE_SOURCE_NAME)\
        .options(**snowflake_options)\
        .option("dbtable", snowflake_database+"."+snowflake_schema+"."+source_table_name).load()
    
    df.show()
    print("12345 - After Read")
    
    df1 = df.select(df["*"])
    
    df1.write.format("snowflake") \
        .options(**snowflake_options) \
        .option("dbtable", "TABLE_23").mode("overwrite") \
        .save()


except Exception as glue_exception_error:
    print("##################### -- Error: "  + str(glue_exception_error) + " -- ##########################")
    raise    

问题原因及解决方法 #

核心原因 #

scala/$less$colon$less类找不到，是因为Scala版本不兼容：
AWS Glue 4.0（对应Spark 3.3）使用的是Scala 2.12版本，但你添加的spark-snowflake_2.13-2.11.1-spark_3.3.jar是基于Scala 2.13编译的，两者不匹配导致类加载失败。

解决步骤 #

1. 替换Spark-Snowflake连接器JAR：下载对应Scala 2.12版本的连接器，比如spark-snowflake_2.12-2.11.1-spark_3.3.jar，上传到S3的对应路径，替换原有的2.13版本JAR。
2. 验证版本匹配：确保连接器的Spark版本和Glue使用的Spark版本一致（Glue 4.0用Spark 3.3，所以选带spark_3.3的连接器版本）。
3. 检查JDBC兼容性：snowflake-jdbc-3.13.27版本与连接器2.11.1兼容，无需替换，但如果后续仍有问题，可以尝试匹配连接器推荐的JDBC版本。
4. 代码小优化：写入时的format("snowflake")建议改为和读取一致的format(SNOWFLAKE_SOURCE_NAME)（即"net.snowflake.spark.snowflake"），避免潜在的类路径解析问题。

内容的提问来源于stack exchange，提问作者AJR