GCP实例间网络异常：无法初始化CockroachDB集群

问题 #

使用Terraform在GCP Compute Engine部署CockroachDB，参考官方《Deploy CockroachDB on Google Cloud Platform GCE (Insecure)》教程。目前遇到网络连接问题：本地与VM实例、VM实例间均无法正常建立连接，无法通过本地启动CockroachDB内置SQL客户端（对应教程第7步），怀疑和GCE TCP Proxy Load Balancing部署有关。

已执行操作 #

1. 启动集群：
   • instance1：cockroach start --insecure --advertise-addr=<internal IP instance1> --join=<internal IP instance1>,<internal IP instance2> --cache=.25 --max-sql-memory=.25 --background
   • instance2：cockroach start --insecure --advertise-addr=<internal IP instance2> --join=<internal IP instance1>,<internal IP instance2> --cache=.25 --max-sql-memory=.25 --background
2. 集群初始化：
   • 本地机器：cockroach init --insecure --host=<external IP instance1>
3. 启动内置SQL客户端（本地机器）：
   执行命令cockroach sql --insecure --host=google_compute_global_address.cockroachdb_tcp_lb_static_ipv4.address后，出现错误：

# Welcome to the CockroachDB SQL shell.
# All statements must be terminated by a semicolon.
# To exit, type: \q.
#
ERROR: server closed the connection.
Is this a CockroachDB node?
unexpected EOF
Failed running "sql"

附Terraform代码 #

instance.tf

resource "google_compute_instance" "cockroachdb_instance" {
  count        = var.instance_count
  name         = "cockroachdb-instance${count.index}"
  machine_type = var.machine_type
  tags         = ["ssh-server", "cockroachdb", "cockroachdb-instances"]
  boot_disk {
    initialize_params {
      image = var.image
    }
  }
  network_interface {
    network = google_compute_network.cockroachdb_network.self_link
    access_config {
      nat_ip = google_compute_address.static[count.index].address
    }
  }
  metadata = {
    sshKeys        = "${var.ssh_user}:${file(var.ssh_pub_key_file)}"
    startup-script = <<-EOT
      #!/bin/bash
      curl -L https://binaries.cockroachdb.com/cockroach-v23.1.8.linux-amd64.tgz | tar -xz
      sudo cp -i cockroach-v23.1.8.linux-amd64/cockroach /usr/local/bin/
      sudo mkdir -p /usr/local/lib/cockroach
      sudo cp -i cockroach-v23.1.8.linux-amd64/lib/libgeos.so /usr/local/lib/cockroach/
      sudo cp -i cockroach-v23.1.8.linux-amd64/lib/libgeos_c.so /usr/local/lib/cockroach/
      EOT
  }
}

resource "google_compute_address" "static" {
  count = var.instance_count
  name  = "ipv4-address-${count.index}"
}

load-balancing.tf

# Global static IPv4 addresses
resource "google_compute_global_address" "cockroachdb_tcp_lb_static_ipv4" {
  name       = "cockroachdb-tcp-lb-static-ipv4"
  ip_version = "IPV4"
}

# Global forwarding rule
resource "google_compute_global_forwarding_rule" "cockroachdb_ipv4_forwarding_rule" {
  name       = "cockroachdb-ipv4-forwarding-rule"
  ip_address = google_compute_global_address.cockroachdb_tcp_lb_static_ipv4.address
  target     = google_compute_target_tcp_proxy.cockroachdb_target_tcp_proxy.self_link
  port_range = "26257"
}

# TCP proxy
resource "google_compute_target_tcp_proxy" "cockroachdb_target_tcp_proxy" {
  name            = "cockroachdb-target-tcp-proxy"
  backend_service = google_compute_backend_service.cockroachdb_backend_service.self_link
}

# Backend service
resource "google_compute_backend_service" "cockroachdb_backend_service" {
  name          = "cockroachdb-backend-service"
  timeout_sec   = 300
  protocol      = "TCP"
  health_checks = [
    google_compute_health_check.cockroachdb_health_check.self_link,
  ]
  backend {
    group           = google_compute_instance_group.cockroachdb_instance_group.self_link
    max_utilization = 0.8
  }
}

# Health check
resource "google_compute_health_check" "cockroachdb_health_check" {
  name                = "cockroachdb-health-check"
  check_interval_sec  = 10
  timeout_sec         = 5
  unhealthy_threshold = 3
  healthy_threshold   = 2
  http_health_check {
    port         = 8080
    request_path = "/health?ready=1"
  }
}

# Instance group
resource "google_compute_instance_group" "cockroachdb_instance_group" {
  name        = "cockroachdb-instance-group"
  instances   = google_compute_instance.cockroachdb_instance[*].self_link
  named_port {
    name = "tcp26257"
    port = 26257
  }
}

network.tf

# Network resource
resource "google_compute_network" "cockroachdb_network" {
  name                    = "cockroachdb-network"
  auto_create_subnetworks = "true"
}

# Firewall rules
resource "google_compute_firewall" "ssh-server" {
  name          = "default-allow-ssh-terraform"
  network       = google_compute_network.cockroachdb_network.name
  source_ranges = ["0.0.0.0/0"]
  target_tags   = ["ssh-server"]
  allow {
    protocol = "tcp"
    ports    = ["22"]
  }
}

resource "google_compute_firewall" "cockroachdb" {
  name          = "default-allow-cockroachdb"
  network       = google_compute_network.cockroachdb_network.name
  source_ranges = ["0.0.0.0/0"]
  target_tags   = ["cockroachdb"]
  allow {
    protocol = "tcp"
    ports    = ["26257", "8080"]
  }
}

排查与修复方案 #

1. 检查负载均衡后端健康状态 #

登录GCP控制台，依次查看：

• Compute Engine > 实例组：确认cockroachdb-instance-group内的实例状态正常
• 负载均衡：查看cockroachdb-backend-service的健康检查结果，若实例未通过健康检查，负载均衡不会转发流量，需优先解决健康问题

2. 修正CockroachDB启动参数 #

当前启动命令未显式设置--listen-addr，默认可能仅监听localhost，导致无法接收外部请求。修改启动命令：

# instance1 启动命令
cockroach start --insecure --advertise-addr=<internal IP instance1> --listen-addr=0.0.0.0:26257 --join=<internal IP instance1>,<internal IP instance2> --cache=.25 --max-sql-memory=.25 --background

# instance2 启动命令
cockroach start --insecure --advertise-addr=<internal IP instance2> --listen-addr=0.0.0.0:26257 --join=<internal IP instance1>,<internal IP instance2> --cache=.25 --max-sql-memory=.25 --background

--listen-addr=0.0.0.0:26257允许实例监听所有网络接口的26257端口，确保内部和外部流量可到达。

3. 验证直接连接实例的可用性 #

跳过负载均衡，直接测试本地到单个实例的连接：

cockroach sql --insecure --host=<external IP instance1>

若仍失败：

• SSH登录实例，执行netstat -plnt | grep cockroach，确认输出包含:26257，验证端口是否监听
• 检查GCP防火墙规则default-allow-cockroachdb是否启用，且目标标签cockroachdb已绑定到实例

4. 修正负载均衡后端服务配置 #

在load-balancing.tf的google_compute_backend_service资源中，添加端口映射指定命名端口：

resource "google_compute_backend_service" "cockroachdb_backend_service" {
  name          = "cockroachdb-backend-service"
  timeout_sec   = 300
  protocol      = "TCP"
  health_checks = [
    google_compute_health_check.cockroachdb_health_check.self_link,
  ]
  backend {
    group           = google_compute_instance_group.cockroachdb_instance_group.self_link
    max_utilization = 0.8
    port_name       = "tcp26257" # 添加该行，绑定实例组的命名端口
  }
}

添加后，后端服务会正确将流量转发到实例的26257端口。

5. 使用实际LB静态IP执行SQL命令 #

需将命令中的google_compute_global_address.cockroachdb_tcp_lb_static_ipv4.address替换为实际IP。可通过Terraform输出获取：

# 在任意.tf文件中添加输出配置
output "cockroachdb_lb_ip" {
  value = google_compute_global_address.cockroachdb_tcp_lb_static_ipv4.address
}

执行terraform apply后，复制输出的IP执行：

cockroach sql --insecure --host=<LB静态IP>

6. 确保实例间内部通信正常 #

登录其中一个实例，尝试连接另一个实例的内部IP：

cockroach sql --insecure --host=<internal IP instance2>

若失败，添加内部子网防火墙规则（根据实际子网CIDR调整）：

resource "google_compute_firewall" "cockroachdb-internal" {
  name          = "allow-cockroachdb-internal"
  network       = google_compute_network.cockroachdb_network.name
  source_ranges = ["10.128.0.0/9"] # GCP默认子网CIDR范围
  target_tags   = ["cockroachdb"]
  allow {
    protocol = "tcp"
    ports    = ["26257", "8080"]
  }
}

内容的提问来源于stack exchange，提问作者Fala Baguete