在Dataiku中使用Python Recipe运行BigQuery SQL查询的认证问题求助
在Dataiku中使用Python Recipe运行BigQuery SQL查询的认证问题求助
我现在有个需求,要在Dataiku的Python Recipe里以指定格式运行BigQuery的SQL查询。在本地环境中,我可以通过设置服务账号凭证JSON文件的路径,用它来完成查询的认证,但到了Dataiku里,我就搞不清楚该怎么正确使用这个JSON文件了。
我已经尝试了两种方法,都遇到了错误,具体情况如下:
方法1:直接设置环境变量指定凭证文件
import dataiku from dataiku import pandasutils as pdu import pandas as pd import google.cloud.bigquery as bigquery from google.cloud import bigquery import os import pandas as pd cred_file = "adc.json" os.environ["GOOGLE_APPLICATION_CREDENTIALS"]=cred_file os.environ["GOOGLE_CLOUD_PROJECT"] = "PROJECT_ID" def query_bq(QUERY): client = bigquery.Client(project="PROJECT_ID") query_job = client.query(QUERY) # API request rows = query_job.result() # Waits for query to finish df = rows.to_dataframe() return df # Initialize BigQuery client client = bigquery.Client() # Step 1: Fetch SQL snippets from BigQuery fetch_sql_query = """ SELECT DISTINCT Col1, Col2, col_3, Col_4 FROM `BQ_Table_name` """ query_job = client.query(fetch_sql_query) snippets = query_job.result()
遇到的错误:
DefaultCredentialsError: File adc.json was not found.
方法2:使用Dataiku的托管文件夹存放凭证文件
我在Dataiku里创建了一个托管文件夹,把adc.json上传进去了,然后用下面的代码尝试加载凭证:
#created a managed folder in Dataiku and uploaded the adc.json file into it. folder_path = dataiku.Folder("folder_name").get_path() credentials = service_account.Credentials.from_service_account_file( "folder_path/adc.json" ) client = bigquery.Client(credentials=credentials, project='PROJECT_ID')
遇到的错误:
MalformedError: Service account info was not in the expected format, missing fields token_uri, client_email.
补充一下,方法1在我本地环境里运行完全没问题,只有在Dataiku的Python Notebook里执行才会出这些错误。
备注:内容来源于stack exchange,提问作者Sreeram Godavarthy
相关产品推荐
相关产品推荐

