Terraform创建Pub Sub Lite主题配置峰值吞吐量的方案咨询

问题描述 #

我们计划通过Terraform部署与Reservation配合使用的Pub/Sub Lite实例。在控制台UI创建Pub/Sub Lite时，可以选择配置峰值发布吞吐量（MiB/s）和峰值订阅吞吐量（MiB/s），但当前官方Terraform文档说明google_pubsub_lite_topic资源不支持这两个参数的配置。

当前我们使用如下Terraform脚本创建Pub/Sub Lite实例：

resource "google_pubsub_lite_reservation" "pubsub_lite_reservation" {
  name = var.lite_reservation_name
  project = var.project
  region  = var.region
  throughput_capacity = var.throughput_capacity
}

resource "google_pubsub_lite_topic" "pubsub_lite_topic" {
  name    = var.topic_name
  project = var.project
  region  = var.region
  zone    = var.zone
  partition_config {
    count = var.partitions_count
    capacity {
      publish_mib_per_sec   = var.publish_mib_per_sec
      subscribe_mib_per_sec = var.subscribe_mib_per_sec
    }
  }

  retention_config {
    per_partition_bytes = var.per_partition_bytes
    period              = var.period
  }

  reservation_config {
    throughput_reservation = google_pubsub_lite_reservation.pubsub_lite_reservation.name
  }
}

当前配置中只能指定固定吞吐量容量，且capacity块为必填字段，无法设置峰值吞吐量上限。我们希望主题可动态调整吞吐量，同时基于为Lite Reservation设置的固定值限制吞吐量峰值，请问是否有可行的解决方案？ #

解决方案 #

有两种可落地的实现方案，均能满足你的需求：

方案1：使用Beta版Terraform Provider（原生支持） #

谷歌Beta版的Google Cloud Terraform Provider已经新增了Pub/Sub Lite弹性吞吐量的配置支持，你只需要切换为beta版的资源，新增elastic_capacity块配置峰值吞吐量即可，capacity块的配置为基础预留吞吐量，elastic_capacity配置的数值即为峰值上限：

# 先声明beta版provider
provider "google-beta" {
  project = var.project
  region  = var.region
}

resource "google_pubsub_lite_reservation" "pubsub_lite_reservation" {
  provider = google-beta
  name = var.lite_reservation_name
  project = var.project
  region  = var.region
  throughput_capacity = var.throughput_capacity
}

resource "google_pubsub_lite_topic" "pubsub_lite_topic" {
  provider = google-beta
  name    = var.topic_name
  project = var.project
  region  = var.region
  zone    = var.zone
  partition_config {
    count = var.partitions_count
    # 基础预留吞吐量，必填字段
    capacity {
      publish_mib_per_sec   = var.publish_mib_per_sec
      subscribe_mib_per_sec = var.subscribe_mib_per_sec
    }
    # 新增：峰值吞吐量配置
    elastic_capacity {
      publish_mib_per_sec   = var.peak_publish_mib_per_sec
      subscribe_mib_per_sec = var.peak_subscribe_mib_per_sec
    }
  }

  retention_config {
    per_partition_bytes = var.per_partition_bytes
    period              = var.period
  }

  reservation_config {
    throughput_reservation = google_pubsub_lite_reservation.pubsub_lite_reservation.name
  }
}

注：Reservation的总吞吐量容量会作为全局硬上限，所有绑定到该Reservation的主题实际总吞吐量不会超过你设置的throughput_capacity值，符合你对峰值限制的要求。

方案2：用local-exec后置配置（兼容稳定版Provider） #

如果你不想使用Beta版Provider，可以通过null_resource调用gcloud命令在主题创建完成后更新峰值参数：

# 原有资源保持不变
resource "google_pubsub_lite_reservation" "pubsub_lite_reservation" {
  name = var.lite_reservation_name
  project = var.project
  region  = var.region
  throughput_capacity = var.throughput_capacity
}

resource "google_pubsub_lite_topic" "pubsub_lite_topic" {
  name    = var.topic_name
  project = var.project
  region  = var.region
  zone    = var.zone
  partition_config {
    count = var.partitions_count
    capacity {
      publish_mib_per_sec   = var.publish_mib_per_sec
      subscribe_mib_per_sec = var.subscribe_mib_per_sec
    }
  }

  retention_config {
    per_partition_bytes = var.per_partition_bytes
    period              = var.period
  }

  reservation_config {
    throughput_reservation = google_pubsub_lite_reservation.pubsub_lite_reservation.name
  }
}

# 新增：后置更新峰值吞吐量
resource "null_resource" "update_lite_topic_peak_throughput" {
  triggers = {
    topic_id = google_pubsub_lite_topic.pubsub_lite_topic.id
  }
  provisioner "local-exec" {
    command = <<EOT
gcloud pubsub lite topics update ${google_pubsub_lite_topic.pubsub_lite_topic.name} \
  --zone=${var.zone} \
  --peak-publish-throughput=${var.peak_publish_mib_per_sec} \
  --peak-subscribe-throughput=${var.peak_subscribe_mib_per_sec}
EOT
  }
}

注：使用该方案需要确保运行Terraform的环境已安装gcloud并完成对应项目的权限认证。 #

内容的提问来源于stack exchange，提问作者Sanjay M. P.