使用BigQuery Storage API Java SDK写入数据遇权限错误，是否需新增角色？

你遇到的PERMISSION_DENIED: Request had insufficient authentication scopes错误，不是服务账号的IAM角色不够——你当前配置的BigQuery Admin、BigQuery Data Owner、BigQuery Data Editor已经拥有足够的写入权限，问题出在认证范围（Authentication Scopes）配置缺失。

解决步骤： #

• 必须确保客户端请求包含BigQuery相关的认证范围，需要添加的范围为：
  • https://www.googleapis.com/auth/bigquery（通用BigQuery权限范围，覆盖Storage API）
  • 或者更精准的https://www.googleapis.com/auth/bigquery.storage（仅针对BigQuery Storage API）

代码修改示例： #

在初始化BigQueryWriteClient时显式指定所需的认证范围：

import com.google.cloud.bigquery.storage.v1.BigQueryWriteClient;
import com.google.cloud.bigquery.storage.v1.BigQueryWriteSettings;
import com.google.auth.oauth2.GoogleCredentials;
import com.google.common.collect.ImmutableList;
import java.io.FileInputStream;
import java.util.HashMap;
import java.util.Map;
import com.google.cloud.bigquery.storage.v1.TableName;
import com.google.cloud.bigquery.storage.v1.WriteStream;
import com.google.cloud.bigquery.storage.v1.CreateWriteStreamRequest;
import com.google.cloud.bigquery.storage.v1.JsonStreamWriter;
import com.google.api.core.ApiFuture;
import com.google.api.core.ApiFutures;
import com.google.common.util.concurrent.MoreExecutors;
import org.json.JSONObject;
import org.json.JSONArray;
import java.time.Instant;

// 加载服务账号凭据
GoogleCredentials credentials = GoogleCredentials.fromStream(new FileInputStream("path/to/service-account-key.json"));

// 配置包含所需范围的客户端设置
BigQueryWriteSettings settings = BigQueryWriteSettings.newBuilder()
    .setCredentialsProvider(() -> credentials)
    .setScopes(ImmutableList.of("https://www.googleapis.com/auth/bigquery"))
    .build();

// 使用配置好的设置创建客户端
try (BigQueryWriteClient bigQueryWriteClient = BigQueryWriteClient.create(settings)) {
    // 你的写入逻辑
    Map<String, Object> metadata = new HashMap<>();
    metadata.put("table_name", "MetaData");
    metadata.put("timestamp", Instant.now().toString());
    metadata.put("is_processed", false);
    JSONObject jsonObject = new JSONObject(metadata);
    TableName parentTable = TableName.of("test", "test", "MetaData");

    WriteStream stream = WriteStream.newBuilder().setType(WriteStream.Type.COMMITTED).build();

    CreateWriteStreamRequest createWriteStreamRequest =
            CreateWriteStreamRequest.newBuilder()
                    .setParent(parentTable.toString())
                    .setWriteStream(stream)
                    .build();
    WriteStream writeStream = bigQueryWriteClient.createWriteStream(createWriteStreamRequest);

    JsonStreamWriter streamWriter = JsonStreamWriter.newBuilder(writeStream.getName(), writeStream.getTableSchema()).build();
    JSONArray arr = new JSONArray();
    arr.put(jsonObject);
    ApiFuture<AppendRowsResponse> future = streamWriter.append(arr, 1);
    ApiFutures.addCallback(
            future, new AppendCompleteCallback(), MoreExecutors.directExecutor());
}

额外说明： #

• IAM角色和认证范围是两个不同的概念：IAM角色决定你是否有权执行某个操作，而认证范围决定你的OAuth令牌能访问哪些API的权限范围，两者必须同时满足才能正常调用API。
• 如果你的代码运行在GCP托管环境（如GCE、GKE、Cloud Function），需要确保实例/资源的服务账号已配置包含BigQuery相关的访问范围，而不仅仅是IAM角色。

内容的提问来源于stack exchange，提问作者Ravi Teja