如何运行DLP扫描API并将结果保存至BigQuery？

前置依赖（Maven） #

<!-- Cloud Function 核心依赖 -->
<dependency>
  <groupId>com.google.cloud.functions</groupId>
  <artifactId>functions-framework-api</artifactId>
  <version>1.0.4</version>
  <scope>provided</scope>
</dependency>
<!-- Google Cloud DLP 客户端依赖 -->
<dependency>
  <groupId>com.google.cloud</groupId>
  <artifactId>google-cloud-dlp</artifactId>
  <version>3.40.0</version>
</dependency>

核心触发器代码 #

import com.google.cloud.functions.HttpFunction;
import com.google.cloud.functions.HttpRequest;
import com.google.cloud.functions.HttpResponse;
import com.google.cloud.dlp.v2.DlpServiceClient;
import com.google.privacy.dlp.v2.*;
import java.io.IOException;
import java.util.stream.Collectors;

public class DlpInspectToBqFunction implements HttpFunction {
  // 替换为实际GCP项目ID、预定义DLP模板ID
  private static final String GCP_PROJECT_ID = "your-project-id";
  private static final String DLP_TEMPLATE_NAME = String.format("projects/%s/inspectTemplates/your-template-id", GCP_PROJECT_ID);

  @Override
  public void service(HttpRequest request, HttpResponse response) throws IOException {
    // 从请求参数获取目标BigQuery的dataset和table
    String targetDataset = request.getFirstQueryParameter("dataset").orElseThrow(() -> new IllegalArgumentException("缺少dataset参数"));
    String targetTable = request.getFirstQueryParameter("table").orElseThrow(() -> new IllegalArgumentException("缺少table参数"));

    try (DlpServiceClient dlpServiceClient = DlpServiceClient.create()) {
      // 配置BigQuery输出规则
      BigQueryOutputConfig bqOutputConfig = BigQueryOutputConfig.newBuilder()
          .setTableId(TableId.newBuilder()
              .setProjectId(GCP_PROJECT_ID)
              .setDatasetId(targetDataset)
              .setTableId(targetTable)
              .build())
          .build();

      OutputStorageConfig outputStorageConfig = OutputStorageConfig.newBuilder()
          .setTable(bqOutputConfig)
          .build();

      // 配置DLP检测作业，示例为检测请求传入的文本内容，可按需替换为其他检测源
      String requestContent = request.getReader().lines().collect(Collectors.joining());
      InspectJobConfig inspectJobConfig = InspectJobConfig.newBuilder()
          .setInspectTemplateName(DLP_TEMPLATE_NAME)
          .setStorageConfig(StorageConfig.newBuilder()
              .setByteItem(ByteContentItem.newBuilder()
                  .setData(com.google.protobuf.ByteString.copyFromUtf8(requestContent))
                  .build())
              .build())
          .setOutputConfig(outputStorageConfig)
          .build();

      // 提交异步DLP作业
      DlpJob dlpJob = dlpServiceClient.createDlpJob(
          ProjectName.of(GCP_PROJECT_ID),
          inspectJobConfig
      );

      response.getWriter().write("DLP作业提交成功，作业ID：" + dlpJob.getName());
      response.setStatusCode(200);
    } catch (Exception e) {
      response.getWriter().write("作业提交失败：" + e.getMessage());
      response.setStatusCode(500);
    }
  }
}

注意事项 #

• Cloud Function运行时使用的服务账号需要至少具备以下权限：roles/dlp.jobsEditor（DLP作业创建权限）、roles/bigquery.dataEditor（目标数据集写权限）、roles/bigquery.jobUser（BigQuery作业运行权限）
• 目标BigQuery表无需提前创建，DLP作业执行完成后会自动按照检测结果的标准Schema创建表结构
• 示例中默认使用异步作业提交，小数据量场景可调整为同步检测，大数据量检测建议使用异步作业模式
• 如果检测数据源是已有的BigQuery表，只需要修改StorageConfig配置，指定源表的项目、数据集、表名即可，无需修改输出配置逻辑

内容的提问来源于stack exchange，提问作者RaptorX