通过Terraform创建AWS MSK Kafka主题时遇连接拒绝问题求助

尝试使用Terraform的Mongey/kafka provider在AWS MSK Kafka集群中创建主题时，始终遇到连接拒绝错误。

集群配置代码：

resource "aws_msk_cluster" "this" {
  cluster_name           = local.cluster_name
  kafka_version          = var.kafka_version # 2.8.2.tiered
  number_of_broker_nodes = var.number_of_broker_nodes
  enhanced_monitoring    = var.enhanced_monitoring
  storage_mode           = var.cluster_storage_mode
  tags                   = var.tags

  broker_node_group_info {
    client_subnets  = var.broker_node_client_subnets
    instance_type   = var.broker_node_instance_type
    security_groups = var.security_groups_id
  }
}

通过以下命令获取bootstrap brokers：

aws kafka get-bootstrap-brokers --cluster-arn <arn>

返回结果：

{
    "BootstrapBrokerStringTls": "....kafka.eu-central-1.amazonaws.com:9094,....kafka.eu-central-1.amazonaws.com:9094,....kafka.eu-central-1.amazonaws.com:9094"
}

Mongey/kafka provider配置代码：

terraform {
  required_providers {
    kafka = {
      source = "Mongey/kafka"
    }
  }
}

provider "kafka" {
  bootstrap_servers = [
    "....kafka.eu-central-1.amazonaws.com:9094",
    "....kafka.eu-central-1.amazonaws.com:9094",
    "....kafka.eu-central-1.amazonaws.com:9094"
  ]
  tls_enabled = true
}

resource "kafka_topic" "topics" {
  for_each = var.kafka_topics

  name               = var.kafka_topics[each.key]
  replication_factor = 2
  partitions         = 100

  config = {
    "segment.ms"     = "20000"
    "cleanup.policy" = "compact"
  }
}

遇到的错误：

kafka: client has run out of available brokers to talk to: 3 errors occurred:
        connect: connection refused

希望完全通过Terraform完成操作，避免手动干预，询问是否缺少配置或需要更换provider。

• 安全组规则限制：AWS MSK集群的安全组必须允许Terraform运行环境（本地机器、CI/CD服务器等）的IP访问9094端口（TLS协议端口）。检查集群关联的安全组，添加入站规则，允许来源IP访问9094 TCP端口。

• 缺少IAM认证配置：AWS MSK默认启用IAM认证（若集群未修改认证配置），Mongey/kafka provider需配置IAM认证才能连接集群。更新provider配置：

  provider "kafka" {
    bootstrap_servers = [
      "....kafka.eu-central-1.amazonaws.com:9094",
      "....kafka.eu-central-1.amazonaws.com:9094",
      "....kafka.eu-central-1.amazonaws.com:9094"
    ]
    tls_enabled = true
    sasl_enabled = true
    sasl_mechanism = "AWS_MSK_IAM"
    aws_region = "eu-central-1"
    # 若运行环境无默认AWS凭证，需手动指定
    # aws_access_key_id = var.aws_access_key_id
    # aws_secret_access_key = var.aws_secret_access_key
  }
  

  同时确保执行Terraform的IAM实体（用户/角色）拥有kafka:CreateTopic等相关权限。

• 集群未完全就绪：MSK集群创建后需要时间完成初始化，先通过aws kafka describe-cluster --cluster-arn <arn>确认集群状态为ACTIVE，再执行Terraform应用。

• 网络可达性问题：Terraform运行环境必须能访问MSK集群所在的VPC网络。本地运行时可通过VPN、VPC对等连接或堡垒机打通网络；CI/CD环境需部署在能访问MSK集群VPC的子网内。

• provider版本兼容性：尝试升级Mongey/kafka provider到最新稳定版本，旧版本可能对AWS MSK支持存在缺陷。修改版本配置：

  terraform {
    required_providers {
      kafka = {
        source  = "Mongey/kafka"
        version = "~> 0.20.0" # 替换为最新稳定版本
      }
    }
  }
  

内容的提问来源于stack exchange，提问作者Homer