GKE Autopilot环境下Rails应用负载均衡端点无法访问问题求助

• 负载均衡端点访问无响应，暂未查询到相关报错日志
• Chrome浏览器提示：

This site can’t be reached 35.213.138.112
took too long to respond.

• 该问题已出现6小时，未检索到Kubernetes和Ruby on Rails相关的对应参考文档，推测问题与Kubernetes配置有关。

Dockerfile配置 #

FROM ruby:2.6.8
RUN apt-get update -qq && apt-get install -y nodejs npm postgresql-client
RUN npm install --global yarn
WORKDIR /src/app
COPY Gemfile ./
ADD . /src/app
Copy vendor/gems/* /src/app/vender/gems/
ENV RAILS_ENV production
RUN bundle install --deployment --without development test

# ENV RAILS_ENV production
RUN bundle exec rake assets:precompile

# Start the main process.
CMD ["bundle", "exec", "rails", "server", "-b", "0.0.0.0"]

Deployment配置 #

apiVersion: apps/v1
kind: Deployment
metadata:
  labels: 
    name: "test-2"
  name: "test-2"
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: "test-2"
  template:
    metadata:
      labels:
        app: "test-2"
    spec:
      containers:
      - name: "test1-sha256-1"
        image: "gcr.io/path-to-image"
        ports:
        - containerPort: 3000
          protocol: TCP
        env:
          - name: PORT
            value: "3000"
          - name: RAILS_SERVE_STATIC_FILES
            value: "true"
      serviceAccountName: test-ksa
---
apiVersion: "autoscaling/v2beta1"
kind: "HorizontalPodAutoscaler"
metadata:
  name: "test-2-hsdfsdf"
  namespace: "default"
  labels:
    app: "test-2"
spec:
  scaleTargetRef:
    kind: "Deployment"
    name: "test-2"
    apiVersion: "apps/v1"
  minReplicas: 1
  maxReplicas: 5
  metrics:
  - type: "Resource"
    resource:
      name: "cpu"
      targetAverageUtilization: 80

Pod运行日志（运行状态正常） #

Puma starting in single mode...
* Puma version: 5.5.0 (ruby 2.6.8-p205) ("Zawgyi")
*  Min threads: 5
*  Max threads: 5
*  Environment: production
*          PID: 1
* Listening on http://0.0.0.0:3000

LoadBalancer服务配置 #

apiVersion: v1
kind: Service
metadata:
  annotations:
    cloud.google.com/neg: '{"ingress":true}'
  creationTimestamp: "2021-10-02T02:39:22Z"
  finalizers:
  - service.kubernetes.io/load-balancer-cleanup
  labels:
    name: test-no-nginx
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:labels:
          .: {}
          f:name: {}
      f:spec:
        f:externalTrafficPolicy: {}
        f:ports:
          .: {}
          k:{"port":80,"protocol":"TCP"}:
            .: {}
            f:port: {}
            f:protocol: {}
            f:targetPort: {}
        f:selector:
          .: {}
          f:app: {}
        f:sessionAffinity: {}
        f:type: {}
    manager: GoogleCloudConsole
    operation: Update
    time: "2021-10-02T02:39:22Z"
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:finalizers:
          .: {}
          v:"service.kubernetes.io/load-balancer-cleanup": {}
      f:status:
        f:loadBalancer:
          f:ingress: {}
    manager: kube-controller-manager
    operation: Update
    time: "2021-10-02T02:40:05Z"
  name: test-no-nginx-service
  namespace: default
  resourceVersion: "83916"
  uid: 9b560670-1f4d-41f7-8301-fd6b5c0e18d2
spec:
  clusterIP: 10.95.123.231
  clusterIPs:
  - 10.95.123.231
  externalTrafficPolicy: Cluster
  ports:
  - nodePort: 32663
    port: 80
    protocol: TCP
    targetPort: 3000
  selector:
    app: test-2
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer:
    ingress:
    - ip: 35.233.233.11

补充验证信息 #

在Deployment配置中添加nginx容器，且不为负载均衡设置targetPort时，可以正常访问到"Welcome to nginx!"页面，对应Deployment中nginx容器配置如下：

containers:
 - name: "nginx-1"
   image: "nginx:latest"

• 首先核对访问IP：Chrome报错里的IP是35.213.138.112，但是LoadBalancer实际分配的公网IP是35.233.233.11，先确认访问的是服务分配的正确公网IP。
• 验证Rails服务本地可用性：进入运行中的Pod，执行curl 127.0.0.1:3000，如果无法正常返回响应，说明Rails服务本身启动异常。大概率是生产环境初始化卡住，常见原因包括：
  • 数据库连接失败，未配置正确的数据库访问地址、账号密码
  • 依赖加载异常：Dockerfile里存在拼写错误，Copy vendor/gems/* /src/app/vender/gems/ 目标路径拼写为vender，正确应为vendor，会导致本地依赖无法正常加载
  • 未配置SECRET_KEY_BASE环境变量，Rails生产环境启动强制要求该参数
    建议添加环境变量RAILS_LOG_TO_STDOUT=true到Deployment配置中，开启Rails日志输出到标准输出，方便排查启动报错。
• 验证集群内部链路连通性：在集群内启动临时容器，执行curl 10.95.123.231:80，如果无法正常返回，说明Service到Pod的链路存在配置问题，可先检查Service的selector是否与Pod标签完全匹配，当前配置是匹配的，可进一步检查kube-proxy运行状态。
• 验证公网访问规则：如果集群内部访问正常，公网访问超时，检查GCP VPC防火墙规则，确认已放行公网到LoadBalancer 80端口的入站流量。

内容的提问来源于stack exchange，提问作者user