Airflow中GoogleCloudStorageToBigQueryOperator的schema_object解析报错求助

尝试通过Airflow DAG将GCS中的换行分隔JSON文件加载到BigQuery表，使用GoogleCloudStorageToBigQueryOperator并指定schema_object而非autodetect选项，但遇到Schema解析错误，相关信息如下：

Schema文件内容 #

[{"name": "id", "type": "INTEGER", "mode": "NULLABLE"},{"name": "description", "type": "INTEGER", "mode": "NULLABLE"}]

原始JSON数据文件 #

{"description":"HR Department","id":9}
{"description":"Restaurant Department","id":10}

Airflow算子代码 #

gcs_to_bq = GoogleCloudStorageToBigQueryOperator(
        task_id=table_name + "_gcs_to_bq",
        bucket=bucket_name,
        bigquery_conn_id="bigquery_default",
        google_cloud_storage_conn_id="google_cloud_storage_default",
        source_objects=[table_name + "/{{ ds_nodash }}/data_json/*.json"],
        schema_object=table_name+"/{{ ds_nodash }}/data_json/schema_file.json",
        allow_jagged_rows=True,
        ignore_unknown_values=True,
        source_format="NEWLINE_DELIMITED_JSON",
        destination_project_dataset_table=project_id
        + "."
        + data_set
        + "."
        + table_name,
        write_disposition="WRITE_TRUNCATE",
        create_disposition="CREATE_IF_NEEDED",
        dag=dag,
    )

报错信息 #

google.api_core.exceptions.BadRequest: 400 Error while reading data, error message: Failed to parse JSON: No object found when new array is started.; BeginArray returned false; Parser terminated before end of string File: schema_file.json

解决方案 #

问题根源

BigQuery要求schema_object对应的JSON Schema文件必须是顶层为对象的格式，而非直接的数组。正确的Schema结构需要包含fields字段，其值为描述列信息的数组。此外，原始数据中description字段是字符串类型，原Schema中错误定义为INTEGER，这会导致后续数据加载失败。

修正后的Schema文件

{
  "fields": [
    {"name": "id", "type": "INTEGER", "mode": "NULLABLE"},
    {"name": "description", "type": "STRING", "mode": "NULLABLE"}
  ]
}

额外检查项

• 确认GCS中schema_file.json的实际路径与schema_object参数配置完全匹配，包括{{ ds_nodash }}渲染后的日期路径是否正确
• 确保Schema文件无多余换行、语法错误，BigQuery对JSON格式的严谨性要求较高

内容的提问来源于stack exchange，提问作者MoRashad