使用AWS Glue向Snowflake写入数据时遇<host>组不存在错误

问题背景 #

AWS Glue任务从RDS读取数据写入Snowflake，切换目标端后运行报错：

pyspark.sql.utils.IllegalArgumentException: No group with name <host>

使用可视化模式生成的脚本未做修改，相关日志与脚本如下：

驱动日志 #

23/03/29 09:45:32 ERROR GlueExceptionAnalysisListener: [Glue Exception Analysis] Last Executed Line number from script job-rds-to-snowflake-visual.py: 50
23/03/29 09:45:32 ERROR GlueExceptionAnalysisListener: [Glue Exception Analysis] {"Event":"GlueETLJobExceptionEvent","Timestamp":1680083132028,"Failure Reason":"Traceback (most recent call last):\n  File \"/tmp/job-rds-to-snowflake-visual.py\", line 50, in <module>\n    transformation_ctx=\"SnowflakeDataCatalog_node1680082896733\",\n  File \"/opt/amazon/lib/python3.6/site-packages/awsglue/dynamicframe.py\", line 819, in from_catalog\n    return self._glue_context.write_dynamic_frame_from_catalog(frame, db, table_name, redshift_tmp_dir, transformation_ctx, additional_options, catalog_id)\n  File \"/opt/amazon/lib/python3.6/site-packages/awsglue/context.py\", line 386, in write_dynamic_frame_from_catalog\n    makeOptions(self._sc, additional_options), catalog_id)\n  File \"/opt/amazon/spark/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py\", line 1305, in __call__\n    answer, self.gateway_client, self.target_id, self.name)\n  File \"/opt/amazon/spark/python/lib/pyspark.zip/pyspark/sql/utils.py\", line 117, in deco\n    raise converted from None\npyspark.sql.utils.IllegalArgumentException: No group with name <host>","Stack Trace":[{"Declaring Class":"deco","Method Name":"raise converted from None","File Name":"/opt/amazon/spark/python/lib/pyspark.zip/pyspark/sql/utils.py","Line Number":117},{"Declaring Class":"__call__","Method Name":"answer, self.gateway_client, self.target_id, self.name)","File Name":"/opt/amazon/spark/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py","Line Number":1305},{"Declaring Class":"write_dynamic_frame_from_catalog","Method Name":"makeOptions(self._sc, additional_options), catalog_id)","File Name":"/opt/amazon/lib/python3.6/site-packages/awsglue/context.py","Line Number":386},{"Declaring Class":"from_catalog","Method Name":"return self._glue_context.write_dynamic_frame_from_catalog(frame, db, table_name, redshift_tmp_dir, transformation_ctx, additional_options, catalog_id)","File Name":"/opt/amazon/lib/python3.6/site-packages/awsglue/dynamicframe.py","Line Number":819},{"Declaring Class":"<module>","Method Name":"transformation_ctx=\"SnowflakeDataCatalog_node1680082896733\",","File Name":"/tmp/job-rds-to-snowflake-visual.py","Line Number":50}],"Last Executed Line number":50,"script":"job-rds-to-snowflake-visual.py"}
23/03/29 09:45:32 ERROR ProcessLauncher: Error from Python:Traceback (most recent call last):
  File "/tmp/job-rds-to-snowflake-visual.py", line 50, in <module>
    transformation_ctx="SnowflakeDataCatalog_node1680082896733",
  File "/opt/amazon/lib/python3.6/site-packages/awsglue/dynamicframe.py", line 819, in from_catalog
    return self._glue_context.write_dynamic_frame_from_catalog(frame, db, table_name, redshift_tmp_dir, transformation_ctx, additional_options, catalog_id)
  File "/opt/amazon/lib/python3.6/site-packages/awsglue/context.py", line 386, in write_dynamic_frame_from_catalog
    makeOptions(self._sc, additional_options), catalog_id)
  File "/opt/amazon/spark/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py", line 1305, in __call__
    answer, self.gateway_client, self.target_id, self.name)
  File "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 117, in deco
    raise converted from None
pyspark.sql.utils.IllegalArgumentException: No group with name <host>
23/03/29 09:45:31 INFO GlueContext: getCatalogSink: catalogId: null, nameSpace: sf_audit_db, tableName: auditlog_dev_public_rds_auditlog, isRegisteredWithLF: false
23/03/29 09:45:26 WARN SharedState: URL.setURLStreamHandlerFactory failed to set FsUrlStreamHandlerFactory
23/03/29 09:45:24 INFO GlueContext: The DataSource in action : com.amazonaws.services.glue.JDBCDataSource
23/03/29 09:45:24 INFO GlueContext: Glue secret manager integration: secretId is not provided.
23/03/29 09:45:24 INFO GlueContext: nameSpace: pg_audit_db, tableName: supportdatabase_public_audit_log_condensed, connectionName conn-rds-pg-auditdb, vendor: postgresql
23/03/29 09:45:24 INFO GlueContext: getCatalogSource: transactionId: <not-specified> asOfTime: <not-specified> catalogPartitionIndexPredicate: <not-specified> 
23/03/29 09:45:24 INFO GlueContext: getCatalogSource: catalogId: null, nameSpace: pg_audit_db, tableName: supportdatabase_public_audit_log_condensed, isRegisteredWithLF: false, isGoverned: false, isRowFilterEnabled: false, useAdvancedFiltering: false, isTableFromSchemaRegistry: false
23/03/29 09:45:22 INFO GlueContext: GlueMetrics configured and enabled
23/03/29 09:45:19 INFO Utils: Successfully started service 'sparkDriver' on port 42465.

生成的脚本 #

import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
from awsglue import DynamicFrame


def sparkSqlQuery(glueContext, query, mapping, transformation_ctx) -> DynamicFrame:
    for alias, frame in mapping.items():
        frame.toDF().createOrReplaceTempView(alias)
    result = spark.sql(query)
    return DynamicFrame.fromDF(result, glueContext, transformation_ctx)


args = getResolvedOptions(sys.argv, ["JOB_NAME"])
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args["JOB_NAME"], args)

# Script generated for node RDS (Data Catalog)
RDSDataCatalog_node1 = glueContext.create_dynamic_frame.from_catalog(
    database="pg_audit_db",
    table_name="supportdatabase_public_audit_log_condensed",
    transformation_ctx="RDSDataCatalog_node1",
)

# Script generated for node SQL Query
SqlQuery0 = """
SELECT 
    *
FROM
      webapirequestlog
"""
SQLQuery_node1679649943271 = sparkSqlQuery(
    glueContext,
    query=SqlQuery0,
    mapping={"webapirequestlog": RDSDataCatalog_node1},
    transformation_ctx="SQLQuery_node1679649943271",
)

# Script generated for node Snowflake (Data Catalog)
SnowflakeDataCatalog_node1680082896733 = glueContext.write_dynamic_frame.from_catalog(
    frame=SQLQuery_node1679649943271,
    database="sf_audit_db",
    table_name="auditlog_dev_public_rds_auditlog",
    transformation_ctx="SnowflakeDataCatalog_node1680082896733",
)

job.commit()

排查方向 #

• 检查Snowflake连接配置的完整性：进入Glue控制台的连接页面，找到关联sf_audit_db的Snowflake连接，确认主机地址（host）已替换为实际的Snowflake账户域名（如xxx.snowflakecomputing.com），未保留<host>占位符；同时验证账户名、仓库、数据库、Schema等参数是否正确填写。
• 验证连接的可用性与权限：在Glue控制台手动测试该Snowflake连接，确认能正常连通；检查Glue任务使用的IAM角色是否拥有访问该连接的权限，若使用Secrets Manager存储凭证，需确保角色有读取对应密钥的权限。
• 检查Catalog表的元数据关联：确认auditlog_dev_public_rds_auditlog表已正确关联上述Snowflake连接，表属性中未出现配置缺失或占位符；同时检查表结构与Snowflake端目标表的字段、类型是否匹配。
• 显式指定连接参数（临时验证方案）：修改脚本，在write_dynamic_frame.from_catalog中添加additional_options参数，显式传入Snowflake连接信息，绕过Catalog配置的解析问题，示例如下：

  SnowflakeDataCatalog_node1680082896733 = glueContext.write_dynamic_frame.from_catalog(
      frame=SQLQuery_node1679649943271,
      database="sf_audit_db",
      table_name="auditlog_dev_public_rds_auditlog",
      additional_options={
          "sfURL": "your-snowflake-host.snowflakecomputing.com",
          "sfAccount": "your-account-id",
          "sfWarehouse": "your-warehouse",
          "sfDatabase": "sf_audit_db",
          "sfSchema": "public",
          "sfRole": "your-role"
      },
      transformation_ctx="SnowflakeDataCatalog_node1680082896733",
  )
  

• 确认Glue版本与驱动兼容性：检查当前Glue作业使用的版本（如Glue 3.0/4.0）是否支持对应版本的Snowflake Spark驱动，若版本不匹配，尝试升级Glue版本或添加兼容的驱动依赖包。

内容的提问来源于stack exchange，提问作者pauloutboy