运行AWS Glue Studio作业时遇Invalid S3资源错误求助

开发者阿泽

2026-8-15

问题

作为AWS Glue新手，我已将CSV文件上传至S3存储桶，通过爬虫生成了Data Catalog数据库表，且该表在AWS Athena中能正常访问。但启动Glue Studio作业进行键映射操作时，遇到无法解决的Invalid S3 resource错误。作业使用的IAM角色已拥有该S3存储桶的访问权限，我尝试了Lake Formation（LF）标签、S3位置注册等所有能想到的访问配置，仍无法定位问题。

错误信息

npy4j.protocol.Py4JJavaError: An error occurred while calling o93.getDynamicFrame.
: java.lang.RuntimeException: class com.amazonaws.services.gluejobexecutor.model.InvalidInputException:Invalid S3 resource (Service: AWSLakeFormation; Status Code: 400; Error Code: InvalidInputException; Request ID: 208fb8b8-62d9-4ebb-9107-bd4b3b6e8119; Proxy: null) (Service: AWSGlueJobExecutor; Status Code: 400; Error Code: InvalidInputException; Request ID: 0685c719-d128-4596-87db-def2d5837a14; Proxy: null)

错误日志

2022-10-24 15:20:58,739 ERROR [main] glueexceptionanalysis.GlueExceptionAnalysisListener (Logging.scala:logError(9)): [Glue Exception Analysis] {
"Event": "GlueETLJobExceptionEvent",
"Timestamp": 1666624858735,
"Failure Reason": "Traceback (most recent call last):
  File \"/tmp/shipping_remap_1_yxd.py\", line 19, in &lt;module&gt;
    transformation_ctx=\"AmazonS3_node1666619146221\",
  File \"/opt/amazon/lib/python3.6/site-packages/awsglue/dynamicframe.py\", line 629, in from_catalog
    return self._glue_context.create_dynamic_frame_from_catalog(db, table_name, redshift_tmp_dir, transformation_ctx, push_down_predicate, additional_options, catalog_id, **kwargs)
  File \"/opt/amazon/lib/python3.6/site-packages/awsglue/context.py\", line 188, in create_dynamic_frame_from_catalog
    return source.getFrame(**kwargs)
  File \"/opt/amazon/lib/python3.6/site-packages/awsglue/data_source.py\", line 36, in getFrame
    jframe = self._jsource.getDynamicFrame()
  File \"/opt/amazon/spark/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py\", line 1305, in __call__
    answer, self.gateway_client, self.target_id, self.name)
  File \"/opt/amazon/spark/python/lib/pyspark.zip/pyspark/sql/utils.py\", line 111, in deco
    return f(*a, **kw)
  File \"/opt/amazon/spark/python/lib/py4j-0.10.9-src.zip/py4j/protocol.py\", line 328, in get_return_value
    format(target_id, \"\.\", name), value)
py4j.protocol.Py4JJavaError: An error occurred while calling o93.getDynamicFrame.
: java.lang.RuntimeException: class com.amazonaws.services.gluejobexecutor.model.InvalidInputException:Invalid S3 resource (Service: AWSLakeFormation; Status Code: 400; Error Code: InvalidInputException; Request ID: 208fb8b8-62d9-4ebb-9107-bd4b3b6e8119; Proxy: null) (Service: AWSGlueJobExecutor; Status Code: 400; Error Code: InvalidInputException; Request ID: 0685c719-d128-4596-87db-def2d5837a14; Proxy: null)
	at com.amazonaws.services.glue.remote.LakeformationCredentialsProvider.refresh(LakeformationCredentialsProvider.scala:50)
	at com.amazonaws.services.glue.remote.LakeformationCredentialsProvider.&lt;init&gt;(LakeformationCredentialsProvider.scala:77)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at com.amazonaws.services.glue.remote.MichiganAWSCredentialProviderProxy$.get(MichiganAWSCredentialProviderProxy.scala:14)
	at com.amazonaws.services.glue.util.FileSchemeWrapper.setHadoopConfiguration(FileSchemeWrapper.scala:43)
	at com.amazonaws.services.glue.util.FileSchemeWrapper.executeWith(FileSchemeWrapper.scala:82)
	at com.amazonaws.services.glue.util.FileSchemeWrapper.executeWithQualifiedScheme(FileSchemeWrapper.scala:90)
	at com.amazonaws.services.glue.HadoopDataSource.getDynamicFrame(DataSource.scala:556)
	at com.amazonaws.services.glue.DataSource.getDynamicFrame(DataSource.scala:101)
	at com.amazonaws.services.glue.DataSource.getDynamicFrame$(DataSource.scala:101)
	at com.amazonaws.services.glue.HadoopDataSource.getDynamicFrame(DataSource.scala:246)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
	at py4j.Gateway.invoke(Gateway.java:282)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.GatewayConnection.run(GatewayConnection.java:238)
	at java.lang.Thread.run(Thread.java:750)
",
"Stack Trace": [
    {
        "Declaring Class": "get_return_value",
        "Method Name": "format(target_id, \"\.\", name), value)",
        "File Name": "/opt/amazon/spark/python/lib/py4j-0.10.9-src.zip/py4j/protocol.py",
        "Line Number": 328
    },
    {
        "Declaring Class": "deco",
        "Method Name": "return f(*a, **kw)",
        "File Name": "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/sql/utils.py",
        "Line Number": 111
    },
    {
        "Declaring Class": "__call__",
        "Method Name": "answer, self.gateway_client, self.target_id, self.name)",
        "File Name": "/opt/amazon/spark/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py",
        "Line Number": 1305
    },
    {
        "Declaring Class": "getFrame",
        "Method Name": "jframe = self._jsource.getDynamicFrame()",
        "File Name": "/opt/amazon/lib/python3.6/site-packages/awsglue/data_source.py",
        "Line Number": 36
    },
    {
        "Declaring Class": "create_dynamic_frame_from_catalog",
        "Method Name": "return source.getFrame(**kwargs)",
        "File Name": "/opt/amazon/lib/python3.6/site-packages/awsglue/context.py",
        "Line Number": 188
    },
    {
        "Declaring Class": "from_catalog",
        "Method Name": "return self._glue_context.create_dynamic_frame_from_catalog(db, table_name, redshift_tmp_dir, transformation_ctx, push_down_predicate, additional_options, catalog_id, **kwargs)",
        "File Name": "/opt/amazon/lib/python3.6/site-packages/awsglue/dynamicframe.py",
        "Line Number": 629
    },
    {
        "Declaring Class": "&lt;module&gt;",
        "Method Name": "transformation_ctx=\"AmazonS3_node1666619146221\",",
        "File Name": "/tmp/shipping_remap_1_yxd.py",
        "Line Number": 19
    }
],
"Last Executed Line number": 19,
"script": "shipping_remap_1_yxd.py"
}

排查与解决方案

确认Lake Formation注册的S3路径精确匹配
检查Lake Formation中注册的S3位置是否和Glue表指向的存储路径完全一致，包括子目录、末尾斜杠等细节。路径不匹配会直接触发Invalid S3 resource错误，需确保注册路径与表存储路径完全对齐。
补充Glue作业角色的Lake Formation权限
即使IAM角色有S3权限，仍需在Lake Formation控制台给该角色授予：
- 目标数据库和表的SELECT权限
- 对应S3存储位置的DATA_LOCATION_ACCESS权限
验证Glue表的存储路径格式
检查Data Catalog中表的存储路径是否为标准s3://bucket-name/path/to/data/格式，无拼写错误、多余空格或特殊字符。
临时禁用Lake Formation凭证测试
在Glue作业的附加参数中添加--enable-lake-formation-credentials false，强制作业直接使用IAM角色权限。若作业正常运行，说明问题出在LF权限配置，再针对性调整。
重新注册S3存储桶到Lake Formation
若S3桶是首次注册LF，可能存在配置未生效的情况。取消注册后重新添加该S3位置，再重新授予权限。
核对Glue Studio数据源配置
确认作业的数据源节点正确选择了目标Data Catalog表，未选错数据库、表名或手动修改路径导致不匹配。

内容的提问来源于stack exchange，提问作者Fai Beyond

运行AWS Glue Studio作业时遇Invalid S3资源错误求助

错误信息 #

错误日志 #

错误信息

错误日志