如何通过Terraform将多文件上传至GCP存储桶？

本地文件夹结构 #

metrics/
     a_metric/
         dags/
             # 包含若干文件/子文件夹
     b_metric/
         dags/
             # 包含若干文件/子文件夹
     c_metric/
         dags/
             # 包含若干文件/子文件夹

需求 #

需要生成一个远程路径为键、本地路径为值的映射，示例结构如下：

a_metric
   local_file_path : remote_file_path

最终目标是将每个metrics/<metric_name>/dags/下的所有文件及子文件夹上传到GCS Bucket的dags/<metric_name>/路径下。

现有Terraform代码 #

locals {
  # Base variables 
  base_vars = yamldecode(file("base/vars.yaml"))

  
  #OK
  # Metrics subfolder names
  metrics = distinct([
    for dir_path in fileset("metrics/", "*/*/") :
    basename(dirname(dir_path))
  ])

  # Metrics configuration 
  metrics_config = flatten([
    for subfolder in local.metrics :
    [
      for file in fileset("metrics/${subfolder}", "vars.yaml") :
      merge(yamldecode(file("metrics/${subfolder}/${file}")), { subfolder = subfolder })
    ]
  ])


}

resource "google_storage_bucket_object" "default" {
  for_each = { for metric in local.metrics_config: metric.subfolder => metric }
  #for_each     = fileset("/metrics/${each.value.subfolder}/dags/", "*.py")

  name   = "dags/${each.value.subfolder}/"
  source = "metrics/${each.value.subfolder}/dags/"
  bucket = "bucket_test_dags_terraform_001" 
}

当前Terraform Plan输出 #

Terraform will perform the following actions:

  # google_storage_bucket_object.default["a_metric"] will be created
  + resource "google_storage_bucket_object" "default" {
      + bucket         = "bucket_test_dags_terraform_001"
      + content_type   = (known after apply)
      + crc32c         = (known after apply)
      + detect_md5hash = "different hash"
      + id             = (known after apply)
      + kms_key_name   = (known after apply)
      + md5hash        = (known after apply)
      + media_link     = (known after apply)
      + name           = "dags/a_metric/"
      + output_name    = (known after apply)
      + self_link      = (known after apply)
      + source         = "metrics/a_metric/dags/"
      + storage_class  = (known after apply)
    }

  # google_storage_bucket_object.default["b_metric"] will be created
  + resource "google_storage_bucket_object" "default" {
      + bucket         = "bucket_test_dags_terraform_001"
      + content_type   = (known after apply)
      + crc32c         = (known after apply)
      + detect_md5hash = "different hash"
      + id             = (known after apply)
      + kms_key_name   = (known after apply)
      + md5hash        = (known after apply)
      + media_link     = (known after apply)
      + name           = "dags/b_metric/"
      + output_name    = (known after apply)
      + self_link      = (known after apply)
      + source         = "metrics/b_metric/dags/"
      + storage_class  = (known after apply)
    }

  # google_storage_bucket_object.default["c_metric"] will be created
  + resource "google_storage_bucket_object" "default" {
      + bucket         = "bucket_test_dags_terraform_001"
      + content_type   = (known after apply)
      + crc32c         = (known after apply)
      + detect_md5hash = "different hash"
      + id             = (known after apply)
      + kms_key_name   = (known after apply)
      + md5hash        = (known after apply)
      + media_link     = (known after apply)
      + name           = "dags/c_metric/"
      + output_name    = (known after apply)
      + self_link      = (known after apply)
      + source         = "metrics/c_metric/dags/"
      + storage_class  = (known after apply)
    }

Plan: 3 to add, 0 to change, 0 to destroy.

问题点 #

当前代码仅会在GCS中创建对应目录的空对象，不会上传dags目录内的实际文件及子文件夹内容。

解决方案 #

要实现批量上传每个dags目录下的所有文件（包括子目录嵌套的文件），需要递归遍历所有文件并生成对应的本地-远程路径映射，为每个文件单独创建存储对象资源。修改后的代码如下：

locals {
  # 获取所有metrics子目录名称
  metrics = distinct([
    for dir_path in fileset("metrics/", "*/*/") :
    basename(dirname(dir_path))
  ])

  # 递归遍历所有dags目录下的文件，生成本地-远程路径映射
  dag_files = flatten([
    for metric in local.metrics : [
      for file_path in fileset("metrics/${metric}/dags/", "**") : {
        local_path  = "metrics/${metric}/dags/${file_path}"
        remote_path = "dags/${metric}/${file_path}"
      }
    ]
  ])

  # 转换为for_each可用的映射，用远程路径作为唯一键
  dag_files_map = {
    for file in local.dag_files : file.remote_path => file
  }
}

resource "google_storage_bucket_object" "dag_files" {
  for_each = local.dag_files_map

  name   = each.value.remote_path
  source = each.value.local_path
  bucket = "bucket_test_dags_terraform_001"
}

代码说明 #

• fileset("metrics/${metric}/dags/", "**")：使用**通配符递归遍历目标目录下的所有文件和子目录
• dag_files：生成包含每个文件本地路径、远程路径的结构化列表
• dag_files_map：将列表转换为映射结构，确保每个存储对象资源的唯一性
• 每个文件对应一个google_storage_bucket_object实例，实现逐个上传

内容的提问来源于stack exchange，提问作者lubi_nascimento