Astro SDK load_file写入BigQuery触发IndexError报错求助
Astro load_file加载CSV到BigQuery时触发索引越界错误
最近安装Astro CLI后,尝试通过astro sql的load_file工具将CSV文件加载至GCP BigQuery时触发报错,报错信息如下:
File "/usr/local/lib/python3.11/site-packages/astro/databases/base.py", line 788, in create_schema_if_applicable statement = self._create_schema_statement.format(schema) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ IndexError: Replacement index 1 out of range for positional args tuple
按照Astro官方文档要求传入指定参数后,仍触发该索引越界错误。以下是我的实现代码、DAG任务执行命令,寻求解决方案:
实现代码
from airflow.decorators import dag, task from datetime import datetime from airflow.operators.python_operator import PythonOperator from airflow.models.baseoperator import chain from airflow.providers.google.cloud.transfers.local_to_gcs import LocalFilesystemToGCSOperator from airflow.providers.google.cloud.operators.bigquery import \ BigQueryCheckOperator, BigQueryCreateEmptyDatasetOperator, BigQueryCreateEmptyTableOperator, \ BigQueryDeleteDatasetOperator from astro import sql as aql from astro.files import File from astro.sql.table import Table, Metadata from astro.constants import FileType @dag( start_date = datetime(2023,1,1), schedule = None, catchup=False, tags = ['retail'], ) def retail(): gcs_to_raw = aql.load_file( task_id='gcs_to_raw', input_file=File( 'gs://cloudbuild_3445455665/raw/online_Retail.csv', conn_id='gcp', filetype=FileType.CSV, ), output_table=Table( name='raw_invoices', conn_id='gcp', metadata=Metadata(schema="retail",)), use_native_support=False, )
执行命令
airflow tasks test retail gcs_to_raw 2023-01-01
内容的提问来源于stack exchange,提问作者user546298
相关产品推荐
相关产品推荐

