AWS Glue同步PostgreSQL到S3时遇NoSuchElementException错误求助

安全阿岳

2026-8-13

问题描述

我运行以下AWS Glue脚本，将PostgreSQL表的指定列复制到S3存储桶：

import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
from awsglue import DynamicFrame


def sparkSqlQuery(glueContext, query, mapping, transformation_ctx) -> DynamicFrame:
    for alias, frame in mapping.items():
        frame.toDF().createOrReplaceTempView(alias)
    result = spark.sql(query)
    return DynamicFrame.fromDF(result, glueContext, transformation_ctx)


args = getResolvedOptions(sys.argv, ["JOB_NAME"])
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args["JOB_NAME"], args)

PostgreSQL_node1668126214499 = glueContext.create_dynamic_frame.from_catalog(
    database="glue-catalog-ra-db",
    table_name="rentalassistance_public_addresses",
    transformation_ctx="PostgreSQL_node1668126214499",
)

SqlQuery0 = """
select id, zip, city from myDataSource

"""
SQLQuery_node1668127298659 = sparkSqlQuery(
    glueContext,
    query=SqlQuery0,
    mapping={"p1": PostgreSQL_node1668126214499},
    transformation_ctx="SQLQuery_node1668127298659",
)

S3bucket_node3 = glueContext.write_dynamic_frame.from_options(
    frame=SQLQuery_node1668127298659,
    connection_type="s3",
    format="json",
    connection_options={
        "path": "s3://refined-data.edquity-dev.co/demo_data/",
        "partitionKeys": [],
    },
    transformation_ctx="S3bucket_node3",
)

job.commit()

执行作业时，错误日志显示：

2022-11-11 00:43:45,504 ERROR [main] glue.ProcessLauncher (Logging.scala:logError(73)): Error from Python:Traceback (most recent call last):
  File "/tmp/export-public-addresses", line 28, in <module>
    transformation_ctx="PostgreSQL_node1668126214499",
  File "/opt/amazon/lib/python3.6/site-packages/awsglue/dynamicframe.py", line 787, in from_catalog
    return self._glue_context.create_dynamic_frame_from_catalog(db, table_name, redshift_tmp_dir, transformation_ctx, push_down_predicate, additional_options, catalog_id, **kwargs)
  File "/opt/amazon/lib/python3.6/site-packages/awsglue/context.py", line 186, in create_dynamic_frame_from_catalog
    makeOptions(self._sc, additional_options), catalog_id),
  File "/opt/amazon/spark/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py", line 1305, in __call__
    answer, self.gateway_client, self.target_id, self.name)
  File "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 111, in deco
    return f(*a, **kw)
  File "/opt/amazon/spark/python/lib/py4j-0.10.9-src.zip/py4j/protocol.py", line 328, in get_return_value
    format(target_id, ".", name), value)
py4j.protocol.Py4JJavaError: An error occurred while calling o89.getCatalogSource.
: java.util.NoSuchElementException: None.get
    at scala.None$.get(Option.scala:349)
    at scala.None$.get(Option.scala:347)
    at com.amazonaws.services.glue.util.DataCatalogWrapper.$anonfun$getJDBCConf$1(DataCatalogWrapper.scala:218)
    at scala.util.Try$.apply(Try.scala:209)
    at com.amazonaws.services.glue.util.DataCatalogWrapper.getJDBCConf(DataCatalogWrapper.scala:209)
    at com.amazonaws.services.glue.GlueContext.getGlueNativeJDBCSource(GlueContext.scala:487)
    at com.amazonaws.services.glue.GlueContext.getCatalogSource(GlueContext.scala:320)
    at com.amazonaws.services.glue.GlueContext.getCatalogSource(GlueContext.scala:185)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
    at py4j.Gateway.invoke(Gateway.java:282)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:238)
    at java.lang.Thread.run(Thread.java:750)

我已尝试移除列选择逻辑简化流程、增加内存/节点配置，但均无效。使用Glue Studio可视化编辑器的数据预览功能也出现相同错误，爬虫已正确抓取到对应表，但似乎无法定位数据。

解决方法

这个错误的核心是Glue无法从数据目录中获取PostgreSQL的JDBC配置信息，以下是经过验证的解决步骤：

检查Glue连接的JDBC URL配置：进入Glue控制台的连接页面，找到对应PostgreSQL连接，确认JDBC URL是否完整且格式正确（例如jdbc:postgresql://hostname:port/dbname），部分情况下爬虫可能未正确写入完整URL。
验证连接的身份凭证：确保连接关联的Secrets Manager密钥包含正确的username和password字段，且Glue作业角色拥有访问该密钥的权限（添加secretsmanager:GetSecretValue权限）。
手动更新表的连接属性：如果爬虫生成的表元数据存在缺失，可进入Glue数据目录的表详情页，手动指定正确的连接，确保表的Connection字段不为空。
使用from_options替代from_catalog直接连接：绕过数据目录，直接在脚本中指定JDBC参数，示例代码如下：

PostgreSQL_node1668126214499 = glueContext.create_dynamic_frame.from_options(
    connection_type="postgresql",
    connection_options={
        "url": "jdbc:postgresql://your-host:5432/your-db",
        "dbtable": "rentalassistance.public.addresses",
        "user": "your-username",
        "password": "your-password"
    },
    transformation_ctx="PostgreSQL_node1668126214499"
)

注意：生产环境建议通过Secrets Manager获取凭证，避免硬编码。

检查Glue作业角色的网络权限：确保作业角色拥有访问PostgreSQL实例的权限，例如VPC端点、安全组规则允许Glue节点访问数据库端口。

内容的提问来源于stack exchange，提问作者PaperDosa