Terraform创建Cloud Composer报broken pipe错误如何处理

问题现象 #

部署过程中抛出如下错误：

错误：等待创建Environment失败：等待创建Environment失败：等待创建Environment失败：检索操作时出错：Get "https://composer.googleapis.com/v1beta1/projects/aayush-terraform/locations/us-central1/operations/ee459492-abb0-4646-893e-09d112219d79?alt=json&prettyPrint=false": write tcp 10.227.112.165:63811->142.251.12.95:443: write: broken pipe。初始环境已创建完成或仍处于创建流程中，资源清理操作失败，错误详情：等待环境创建完成时获取创建操作状态失败，但环境当前似乎仍处于'CREATING'状态。请等待操作执行完成后，手动删除对应环境，或将"projects/aayush-terraform/locations/us-central1/environments/example-composer-env"导入到你的Terraform状态文件中。

复现使用的Terraform配置如下：

terraform {
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "~>3.0"
    }
  }
}

variable "gcp_region" {
  type        = string
  description = "Region to use for GCP provider"
  default     = "us-central1"
}

variable "gcp_project" {
  type        = string
  description = "Project to use for this config"
  default     = "aayush-terraform"
}

provider "google" {
  region  = var.gcp_region
  project = var.gcp_project
}

resource "google_service_account" "test" {
  account_id   = "composer-env-account"
  display_name = "Test Service Account for Composer Environment"
}

resource "google_project_iam_member" "composer-worker" {
  role   = "roles/composer.worker"
  member = "serviceAccount:${google_service_account.test.email}"
}

resource "google_compute_network" "test" {
  name                    = "composer-test-network"
  auto_create_subnetworks = false
}

resource "google_compute_subnetwork" "test" {
  name          = "composer-test-subnetwork"
  ip_cidr_range = "10.2.0.0/16"
  region        = "us-central1"
  network       = google_compute_network.test.id
}

resource "google_composer_environment" "test" {
  name   = "example-composer-env"
  region = "us-central1"
  config {
    node_count = 3

    node_config {
      zone         = "us-central1-a"
      machine_type = "n1-standard-1"

      network    = google_compute_network.test.id
      subnetwork = google_compute_subnetwork.test.id

      service_account = google_service_account.test.name
    }
  }
}

已知现象补充：

• 报错触发后，Cloud Composer环境实际仍可正常创建完成
• 部署使用的服务账号已被授予Owner权限，排除权限缺失问题

根因说明 #

该问题和权限无关，是hashicorp/google 3.x版本的已知缺陷：
Cloud Composer环境创建周期通常为20~40分钟，属于长耗时GCP操作。3.x版本provider对长时操作的轮询逻辑未做连接保活和断线重试，轮询过程中TCP连接被中间链路或GCP网关主动断开后，会直接抛出broken pipe错误终止部署流程，不会自动续连查询操作最终状态，导致Terraform侧判定部署失败，但云侧实际创建流程仍在正常执行。

处理方案 #

已触发报错的场景处理 #

• 前往GCP Composer控制台查看对应环境状态，等待状态变为RUNNING（创建完成）或确认创建失败后再做后续操作，禁止报错后立刻重复执行terraform apply，避免资源重复创建或状态冲突。
• 如果需要保留已创建的环境，执行如下命令将云侧已存在的资源导入Terraform状态：

terraform import google_composer_environment.test projects/aayush-terraform/locations/us-central1/environments/example-composer-env

导入完成后执行terraform plan，确认无配置漂移即可正常开展后续运维操作。

• 如果不需要保留该环境，等环境状态稳定后直接在控制台删除，再重新执行部署即可。

永久规避方案 #

• 优先升级Google provider版本到4.x及以上稳定版本，新版本已经优化长时操作的轮询逻辑，增加了断线自动重试机制，可避免同类问题出现。
• 若因兼容问题暂无法升级3.x版本，需要调整provider配置增加重试、延长超时时间，同时给Composer资源显式设置匹配创建周期的超时阈值，配置示例如下：

provider "google" {
  region          = var.gcp_region
  project         = var.gcp_project
  request_timeout = "60m"
  retry {
    attempts = 5
    backoff  = "10s"
  }
}

resource "google_composer_environment" "test" {
  name   = "example-composer-env"
  region = "us-central1"
  # 显式设置资源操作超时
  timeouts {
    create = "60m"
    update = "60m"
    delete = "30m"
  }
  config {
    node_count = 3

    node_config {
      zone         = "us-central1-a"
      machine_type = "n1-standard-1"

      network    = google_compute_network.test.id
      subnetwork = google_compute_subnetwork.test.id

      service_account = google_service_account.test.name
    }
  }
}

内容的提问来源于stack exchange，提问作者Drive Upload