使用dbt Redshift Unload宏导出数据至S3时执行失败求助
Redshift Unload宏在dbt执行失败但直接运行SQL正常的问题
我尝试使用Redshift Package Macro将数据卸载至S3,编译后的代码直接在Redshift中运行正常,但通过dbt执行或构建模型时持续失败。
dbt代码
{{ config( { "post-hook": [ "{{ redshift.unload_table('analytics_dev_josh', 'stg_payment_transaction', s3_path='s3://bucket/folder', iam_role='arn:aws:iam::xxxxxxxxx:role/xxxxxxxxxx', header=True, delimiter=',', overwrite=true) }}" ] } ) }}
编译后的SQL代码
-- compile UNLOAD statement UNLOAD ('SELECT * FROM "analytics_dev_josh"."stg_payment_transaction"') TO 's3://bucket/folder' IAM_ROLE 'arn:aws:iam::xxxxxxxxx:role/xxxxxxxxxx' HEADER DELIMITER AS ',' NULL AS '' MAXFILESIZE AS 6 GB ESCAPE ALLOWOVERWRITE PARALLEL OFF
报错信息
14:03:06 Began running node model.analytics.unload_test 14:03:06 1 of 1 START sql table model analytics_dev_josh.unload_test .................... [RUN] 14:03:06 Acquiring new redshift connection 'model.analytics.unload_test' 14:03:06 Began compiling node model.analytics.unload_test 14:03:06 Writing injected SQL for node "model.analytics.unload_test" 14:03:06 Timing info for model.analytics.unload_test (compile): 2023-02-27 14:03:06.181313 => 2023-02-27 14:03:06.184251 14:03:06 Began executing node model.analytics.unload_test 14:03:06 Writing runtime sql for node "model.analytics.unload_test" 14:03:06 Using redshift connection "model.analytics.unload_test" 14:03:06 On model.analytics.unload_test: BEGIN 14:03:06 Opening a new connection, currently in state closed 14:03:06 Redshift adapter: Connecting to Redshift using 'database' credentials 14:03:06 SQL status: BEGIN in 0 seconds 14:03:06 Using redshift connection "model.analytics.unload_test" 14:03:06 On model.analytics.unload_test: /* {"app": "dbt", "dbt_version": "1.4.3", "profile_name": "user", "target_name": "default", "node_id": "model.analytics.unload_test"} */ create table "domusodatawarehouse"."analytics_dev_josh"."unload_test__dbt_tmp" as ( ); 14:03:06 Postgres adapter: Postgres error: syntax error at or near ")" LINE 13: ); ^ 14:03:06 On model.analytics.unload_test: ROLLBACK 14:03:06 Timing info for model.analytics.unload_test (execute): 2023-02-27 14:03:06.184611 => 2023-02-27 14:03:06.337104 14:03:06 On model.analytics.unload_test: Close 14:03:06 Database Error in model unload_test (models/stage/unload_test.sql) syntax error at or near ")" LINE 13: ); ^ compiled Code at target/run/analytics/models/stage/unload_test.sql 14:03:06 Sending event: {'category': 'dbt', 'action': 'run_model', 'label': '5db3e73a-88f6-4c8c-bd6f-570e08d52b43', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x7fce41a11130>]} 14:03:06 1 of 1 ERROR creating sql table model analytics_dev_josh.unload_test ........... [ERROR in 0.16s] 14:03:06 Finished running node model.analytics.unload_test
编译后的代码复制到Redshift中直接运行可正常工作,但通过dbt执行时会报错,看起来dbt尝试将Unload语句写入临时表时触发了语法错误。
内容的提问来源于stack exchange,提问作者Josh Dobson
相关产品推荐
相关产品推荐

