从AWS Glue创建Redshift表时TempDir参数报错求助

问题根源 #

你遇到的错误有两个核心原因：

• 提前提交了Glue作业：代码在初始化Job后立刻执行job.commit()，直接终止了作业生命周期，后续Redshift写入操作无法获取到你配置的--TempDir参数上下文。
• 连接选项键名错误：Glue的Redshift连接要求临时目录的键名为全小写的tempdir，你代码里用的TempDir不会被识别。

修正后的完整代码 #

import sys
import os  # 原代码缺失该导入，会导致os.environ调用报错
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
from awsglue.dynamicframe import DynamicFrame

## @params: [JOB_NAME]
job_name = os.environ.get('AWS_GLUE_JOB_NAME', 'testing1')
args = getResolvedOptions(sys.argv, ['JOB_NAME','TempDir'])
temp_dir = args['TempDir']
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'], args)

# 移除提前执行的job.commit()，这是关键错误点
# job.commit()

redshift_url = "jdbc:redshift://flighttest.ckbw5q2qccz6.eu-north1.redshift.amazonaws.com:5439/dev"
redshift_user = "awsuser"
redshift_password = "Prateek1997"
redshift_table = "dev.flight.details"
create_table_sql = f"""
    CREATE TABLE IF NOT EXISTS {redshift_table}(
    Carrier varchar(100),
    OriginAirportID INT,
    DestAirportID   INT,
    DepDelay    INT,
    ArrDelay    INT
    )
    DISTSTYLE KEY
    DISTKEY(OriginAirportID);
 """

spark._jvm.java.lang.Class.forName("com.amazon.redshift.jdbc42.Driver")
try:
    glueContext.write_dynamic_frame.from_options(
        frame=DynamicFrame.fromDF(spark.sql("SELECT 1 AS dummy"), glueContext, "dummy"),
        connection_type="redshift",
        connection_options={
            "url": redshift_url,
            "user": redshift_user,
            "password": redshift_password,
            "preactions": create_table_sql,
            "dbtable": "dev.flight_details",
            "tempdir": temp_dir  # 修改为全小写的键名
        }
    )
    print("Table created successfully")
except Exception as e:
    print("Failed to create table:", str(e))
    
database_name = "flight"
table_name = "prateekproject1"
df = glueContext.create_dynamic_frame.from_catalog(
    database = database_name,
    table_name = table_name
)
df.printSchema()
dataframe = df.toDF()
print("Sample Data:")
dataframe.show(10)

# 所有业务操作完成后，再提交作业
job.commit()

额外验证项 #

• 确认S3临时目录s3://prateekproject1/temp_dir/已创建，且Glue作业的IAM角色拥有该目录的读写权限。
• 检查Redshift集群安全组，允许Glue作业所在VPC访问Redshift的5439端口。

内容的提问来源于stack exchange，提问作者Prateek Goel