AWS Glue 2.x调用writeFrame报错：pyWriteDynamicFrame方法不存在

Let's break down what's causing this error and how to fix it quickly:

The Root Cause #

Looking at your error stack trace and code, you're passing a Spark DataFrame (datasource1) to sink.writeFrame(), but this method expects a Glue DynamicFrame instead. In your code, you correctly converted datasource1 to datasource2 (a DynamicFrame), but you accidentally used the wrong variable in the write call.

Corrected Code #

Here's the fixed version of your code snippet:

datasource0 = glueContext.create_dynamic_frame.from_catalog(database = "my_db", table_name = "table1", transformation_ctx = "datasource0")
datasource1 = datasource0.toDF().withColumn("date", current_date().cast("string"))
# This converts the Spark DataFrame back to a Glue DynamicFrame
datasource2 = DynamicFrame.fromDF(datasource1, glueContext, "datasource2")
sink = glueContext.getSink(connection_type="s3", path="s3://my_bucket/output", enableUpdateCatalog=True)
sink.setFormat("json")
sink.setCatalogInfo(catalogDatabase='my_db', catalogTableName='table2')
# Use datasource2 (DynamicFrame) instead of datasource1 (Spark DataFrame)
sink.writeFrame(datasource2)
job.commit()

Additional Notes #

• When working with AWS Glue sinks, always ensure you're passing a DynamicFrame to writeFrame(). Spark DataFrames don't have the underlying Java methods that Glue's sink expects, which is why you got the pyWriteDynamicFrame method not found error.
• Double-check variable names when switching between Spark DataFrames and Glue DynamicFrames—it's an easy typo to make!

内容的提问来源于stack exchange，提问作者Rax Wunter