通过私有链接连接AWS Glue与Snowflake时遇PySpark异常

已配置VPC端点，不使用PySpark时可正常通过私有链接连接Snowflake，但运行指定PySpark代码时触发异常：pyspark.sql.utils.IllegalArgumentException: Bad level "<BUILT-IN FUNCTION ALL>"。已尝试更换查询语句、切换不同版本JAR包（snowflake-jdbc-3.13.22.jar、spark-snowflake_2.12-2.11.0-spark_3.3.jar），问题仍未解决。

代码示例 #

from cryptography.hazmat.backends import default_backend
from cryptography.hazmat.primitives import serialization
from awsglue.context import GlueContext
from pyspark.context import SparkContext
from pyspark.sql import SQLContext
from awsglue.job import Job
import re

glueContext = GlueContext(SparkContext.getOrCreate())
spark = glueContext.spark_session
job = Job(glueContext)
key_file = """-----BEGIN ENCRYPTED PRIVATE KEY-----
-----END ENCRYPTED PRIVATE KEY-----
"""
p_key = serialization.load_pem_private_key(
str.encode(key_file),
password="<pass>".encode(),
backend=default_backend()
)

pkb = p_key.private_bytes(
    encoding=serialization.Encoding.PEM,
    format=serialization.PrivateFormat.PKCS8,
    encryption_algorithm=serialization.NoEncryption()
)

pkb = pkb.decode("UTF-8")
pkb = re.sub("-*(BEGIN|END) PRIVATE KEY-*\n","",pkb).replace("\n","")

sfOptions = {
"sfURL" : "",
"sfUser" : "",
"pem_private_key": pkb,
"sfDatabase" : "",
"sfSchema" : "",
"sfWarehouse" : "",
"continue_on_error": "",
"sfAccount" : "",
"tracing": all,
}

query = "SELECT COUNT(*) FROM my_table"
SNOWFLAKE_SOURCE_NAME = "net.snowflake.spark.snowflake"
df = spark.read.format(SNOWFLAKE_SOURCE_NAME).options(**sfOptions).option("query",  query).load()
df.show()

job.commit()

报错信息 #

Traceback (most recent call last):
  File "/tmp/test snowflake.py", line 79, in <module>
    df = spark.read.format(SNOWFLAKE_SOURCE_NAME).options(**sfOptions).option("query",  query).load()
  File "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/sql/readwriter.py", line 210, in load
    return self._df(self._jreader.load())
  File "/opt/amazon/spark/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py", line 1305, in __call__
    answer, self.gateway_client, self.target_id, self.name)
  File "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 117, in deco
    raise converted from None
pyspark.sql.utils.IllegalArgumentException: Bad level "<BUILT-IN FUNCTION ALL>"

解决方案 #

问题根源在于sfOptions字典中的tracing参数值错误：你直接使用了Python内置函数all，而Snowflake Spark Connector要求tracing的有效值是字符串类型的"ALL"、"DEBUG"、"INFO"等。

修改sfOptions中的tracing配置：

sfOptions = {
    # 其他配置项保持不变
    "tracing": "ALL",  # 把all改为字符串"ALL"
}

修改后重新运行代码即可解决该报错。

内容的提问来源于stack exchange，提问作者JorgeH