无法连接nginx部署与LoadBalancer后的redis-sentinel服务

最终目标 #

将Nginx部署在Redis-Sentinel Pod前端，使用企业SSL证书加密访问。Redis本身运行正常且测试通过，问题集中在Nginx部署及关联Service配置，后续尝试改用Ingress方案仍未解决连接问题。

最新更新（Ingress尝试） #

使用MicroK8s且已启用Ingress组件，操作步骤如下：

1. 获取Ingress命名空间下的ConfigMap：

   kubectl get configmap --namespace ingress
   

2. 编辑nginx-load-balancer-microk8s-conf配置：

   kubectl edit configmap nginx-load-balancer-microk8s-conf --namespace ingress
   

3. 添加TCP服务转发规则：

   data:
     tcp-services: |-
       6379: "default/redis-service:6379:redis-ssl"
       26379: "default/redis-sentinel-service:26379:redis-ssl"
   

4. 滚动更新Ingress DaemonSet，并将证书Secret复制到Ingress命名空间
5. 创建LoadBalancer类型的Service：

   apiVersion: v1
   kind: Service
   metadata:
     name: redis-service-lb
     namespace: redis
   spec:
     type: LoadBalancer
     loadBalancerIP: 10.250.0.44
     ports:
     - port: 6379
       targetPort: 6379
       protocol: TCP
       name: tcp-redis
     - port: 26379
       targetPort: 26379
       protocol: TCP
       name: tcp-redis-sentinel
     selector:
       app: redis
   

当前状态 #

• 集群事件显示IP分配正常：44s Normal IPAllocated service/redis-sentinel-service-lb Assigned IP ["10.250.0.41"]
• Service运行状态正常：

  redis-service-lb          LoadBalancer   10.152.183.56    10.250.0.44   6379:31301/TCP,26379:32502/TCP   5m49s
  

• 外部仍无法通过nc或redis-cli建立连接

历史操作记录（初始Nginx代理方案） #

Nginx配置（通过ConfigMap挂载） #

server {
    listen 6379 ssl;
    server_name ki44.MyDomain.com;

    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;

    sendfile        on;
    keepalive_timeout  65;

    ssl_protocols                               TLSv1.2 TLSv1.3;
    ssl_prefer_server_ciphers                   on;
    ssl_certificate                             /etc/nginx/certs/tls.crt;
    ssl_certificate_key                         /etc/nginx/certs/tls.key;

    location / {
        proxy_pass http://redis-sentinel.redis.svc.cluster.local:6379;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

server {
    listen 26379 ssl;
    server_name ki44.MyDomain.com;

    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;

    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    sendfile        on;
    keepalive_timeout  65;

    ssl_protocols                               TLSv1.2 TLSv1.3;
    ssl_prefer_server_ciphers                   on;
    ssl_certificate                             /etc/nginx/certs/tls.crt;
    ssl_certificate_key                         /etc/nginx/certs/tls.key;

    location / {
        proxy_pass http://redis-sentinel.redis.svc.cluster.local:23679;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

关联Deployment与Service配置 #

apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis-nginx
  namespace: redis
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx-proxy
  template:
    metadata:
      labels:
        app: nginx-proxy
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 6379
          name: redis
          hostPort: 6379
        - containerPort: 26379
          name: sentinel
          hostPort: 26379
        volumeMounts:
        - name: config-volume
          mountPath: /etc/nginx/conf.d
        - name: cert-volume
          mountPath: /etc/nginx/certs
      volumes:
      - name: config-volume
        configMap:
          name: nginx-config
      - name: cert-volume
        secret:
          secretName: redis-ssl
      nodeSelector:
        location: internal
        type: worker
---
apiVersion: v1
kind: Service
metadata:
  name: redis
  namespace: redis
spec:
  type: LoadBalancer
  loadBalancerIP: 10.250.0.44
  ports:
    - port: 6379
      name: redis
      targetPort: 6379
      protocol: TCP
    - port: 26379
      name: sentinel
      targetPort: 26379
      protocol: TCP
  selector:
    app: redis-nginx

当前状态 #

• Nginx Pod启动正常，进入Pod后可通过nc访问Redis-Sentinel端口
• 外部执行nc -zv ki44.MyDomain.com 6379和nc -zv ki44.MyDomain.com 26379均无法连接
• Service状态显示IP已分配：

  NAME                      TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                          AGE
  redis-sentinel-headless   ClusterIP      None             <none>        6379/TCP,26379/TCP               3h30m
  redis-sentinel            ClusterIP      10.152.183.130   <none>        6379/TCP,26379/TCP               3h30m
  redis                     LoadBalancer   10.152.183.218   10.250.0.44   6379:30420/TCP,26379:30154/TCP   55m
  

• ki44.MyDomain.com已正确解析到10.250.0.44，Pod日志、事件均无异常

已尝试的解决方法 #

• 将proxy_pass目标从redis-sentinel.redis.svc.cluster.local改为redis-sentinel-headless.redis.svc.cluster.local
• 添加/移除Deployment中的hostPort配置
• 移除SSL配置，测试明文连接
• 单独测试6379端口
• 在同命名空间创建测试Pod，通过内部IP访问Nginx代理成功，Nginx日志有请求记录

已知信息 #

• 排除防火墙和网络连通性问题
• ki44.MyDomain.com对应IP无其他服务监听6379/26379端口
• Nginx Pod内6379/26379端口处于监听状态，且能内部连通Redis-Sentinel
• 集群内其他LoadBalancer服务运行正常，此为首次配置前端Nginx

内容的提问来源于stack exchange，提问作者Rachel Ambler