新手求助：通过Terraform将Stackdriver日志导出至BigQuery无数据

Hey there! Let's walk through the likely issues here since your BigQuery dataset is staying empty even after setting up the logging sink and IAM permissions. I'll break down the problems and fixes step by step:

1. You're Applying IAM Permissions at the Wrong Level #

Your current code uses google_project_iam_member to grant the roles/bigquery.dataEditor role at the project level. While this might work in some cases, it's not the correct (or most secure) approach. The Logging Sink's writer identity only needs permissions directly on the target BigQuery dataset, not the entire project.

Fix: Switch to Dataset-Level IAM #

Replace your project-level IAM resource with a dataset-specific one:

resource "google_bigquery_dataset_iam_member" "bq_log_writer" {
  dataset_id = google_bigquery_dataset.stackdriver_logging.dataset_id
  project    = google_project.project.project_id
  role       = "roles/bigquery.dataEditor"
  member     = google_logging_project_sink.big_query.writer_identity
}

This ensures the sink's service account has exactly the access it needs to write logs to your dataset.

2. Check for Log Activity (or a Missing Filter) #

Your sink doesn't specify a filter parameter, which means it should export all project logs by default. But if your project hasn't generated any new logs since setting up the sink, the dataset will stay empty.

Quick Test: Generate Sample Logs #

Try triggering some log activity manually—spin up a GCE instance, make an API call, or use the GCP Logs Explorer to write a test log. Then wait a bit for the export to kick in.

If you want to narrow down the logs for testing, add a filter to your sink (example for GCE activity logs):

resource "google_logging_project_sink" "big_query" {
  name                  = "${google_project.project.project_id}-big_query-sink"
  project               = google_project.project.project_id
  destination           = "bigquery.googleapis.com/projects/${google_project.project.project_id}/datasets/${google_bigquery_dataset.stackdriver_logging.dataset_id}"
  unique_writer_identity = true
  filter                = "logName:\"projects/${google_project.project.project_id}/logs/compute.googleapis.com%2Factivity_log\""
}

3. Don't Forget About Export Delay #

Stackdriver logs don't show up in BigQuery instantly. There's usually a 5-15 minute delay (sometimes longer depending on log volume) between when a log is generated and when it lands in your dataset. If you just deployed your Terraform code, give it some time before checking again.

4. Verify Configuration in the GCP Console #

Double-check your setup in the GCP UI to rule out any hidden issues:

• Go to Logging > Logs Router and confirm your sink's destination points to the correct dataset, and there are no error messages listed.
• In BigQuery, navigate to your dataset's Permissions tab and make sure the sink's service account (looks like serviceAccount:xxx@logging.googleapis.com) has the BigQuery Data Editor role.

Full Corrected Terraform Code #

Here's the complete, fixed version of your code:

resource "google_bigquery_dataset" "stackdriver_logging" {
  dataset_id    = "stackdriver_logs"
  friendly_name = "stackdriver_logs"
  location      = "US"
  project       = google_project.project.project_id
}

resource "google_logging_project_sink" "big_query" {
  name                  = "${google_project.project.project_id}-big_query-sink"
  project               = google_project.project.project_id
  destination           = "bigquery.googleapis.com/projects/${google_project.project.project_id}/datasets/${google_bigquery_dataset.stackdriver_logging.dataset_id}"
  unique_writer_identity = true
  # Optional filter for testing—uncomment if needed
  # filter                = "logName:\"projects/${google_project.project.project_id}/logs/compute.googleapis.com%2Factivity_log\""
}

resource "google_bigquery_dataset_iam_member" "bq_log_writer" {
  dataset_id = google_bigquery_dataset.stackdriver_logging.dataset_id
  project    = google_project.project.project_id
  role       = "roles/bigquery.dataEditor"
  member     = google_logging_project_sink.big_query.writer_identity
}

内容的提问来源于stack exchange，提问作者qL9r0b51wvxnMsU