Minikube中Nginx连接间歇性异常及配置问题咨询

问题现象与环境信息 #

Nginx错误日志 #

2024/03/21 07:07:05 [error] 22#22: *9 connect() failed (111: Connection refused) while connecting to upstream, client: 10.244.0.1, server: 127.0.0.1, request: "GET / HTTP/1.1", upstream: "http://10.101.14.185:80/", host: "192.168.49.2:30590"
10.244.0.1 - - [21/Mar/2024:07:07:05 +0000] "GET / HTTP/1.1" 502 497 "-" "curl/8.3.0" "-"
10.244.0.1 - - [21/Mar/2024:07:07:06 +0000] "GET / HTTP/1.1" 307 0 "-" "curl/8.3.0" "-"
10.244.0.1 - - [21/Mar/2024:07:07:07 +0000] "GET / HTTP/1.1" 307 0 "-" "curl/8.3.0" "-"
2024/03/21 07:07:08 [error] 22#22: *15 connect() failed (111: Connection refused) while connecting to upstream, client: 10.244.0.1, server: 127.0.0.1, request: "GET / HTTP/1.1", upstream: "http://10.101.14.185:80/", host: "192.168.49.2:30590"
10.244.0.1 - - [21/Mar/2024:07:07:08 +0000] "GET / HTTP/1.1" 502 497 "-" "curl/8.3.0" "-"

集群资源状态 #

NAME                                    READY   STATUS    RESTARTS   AGE
pod/abc-deployment-7bf886b77d-fqnn4     1/1     Running   0          15m
pod/abc-deployment-7bf886b77d-xg7n4     1/1     Running   0          15m
pod/frontend-9b5d57d49-2gmlc            1/1     Running   0          6m33s
pod/mongodb-deployment-0                1/1     Running   0          15m

NAME                       TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
service/abc-service        ClusterIP      10.101.14.185   <none>        80/TCP         15m
service/frontend           LoadBalancer   10.99.135.28    <pending>     80:30590/TCP   15m
service/kubernetes         ClusterIP      10.96.0.1       <none>        443/TCP        18m
service/mongodb-service2   ClusterIP      10.96.89.248    <none>        27018/TCP      15m

NAME                               READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/abc-deployment     2/2     2            2           15m
deployment.apps/frontend           1/1     1            1           6m33s

NAME                                          DESIRED   CURRENT   READY   AGE
replicaset.apps/abc-deployment-7bf886b77d     2         2         2       15m
replicaset.apps/frontend-9b5d57d49            1         1         1       6m33s

NAME                                  READY   AGE
statefulset.apps/mongodb-deployment   1/1     15m

Nginx配置（default.conf） #

server {
    listen       80;
    listen  [::]:80;
    server_name  127.0.0.1;

    #access_log  /var/log/nginx/host.access.log  main;

    location / {
      proxy_pass http://abc-service;
      proxy_set_header Connection keep-alive;
            #        root   /usr/share/nginx/html;
            #        index  index.html index.htm;
    }

abc-service详情 #

Name:              abc-service
Namespace:         default
Labels:            <none>
Annotations:       <none>
Selector:          app=abc
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.101.14.185
IPs:               10.101.14.185
Port:              <unset>  80/TCP
TargetPort:        1740/TCP
Endpoints:         10.244.0.4:1740,10.244.0.5:1740,10.244.0.9:1740
Session Affinity:  None
Events:            <none>

核心问题 #

1. Nginx与abc-service的连接出现间歇性中断；
2. 尽管使用了LoadBalancer类型的frontend Service，但请求未转发至多个abc Pod，仅指向单个Pod；
3. 仅配置了2个abc-deployment副本，但abc-service存在3个Endpoint，最后一个Endpoint的作用是什么？

问题解答 #

1. 间歇性连接中断的原因 #

第三个Endpoint 10.244.0.9:1740 对应frontend Pod的IP（和hosts文件中的IP一致）。由于abc-service的Selector是app=abc，而frontend Pod的标签也包含app=abc，导致它被错误纳入服务后端列表。但frontend Pod运行的是Nginx，并不监听1740端口，当kube-proxy轮询到这个无效Endpoint时，就会出现连接拒绝错误，这就是间歇性中断的根源。

2. 请求未转发到多个abc Pod的问题 #

首先明确：frontend Service是用来暴露Nginx Pod的，请求先到Nginx再转发到abc-service。abc-service默认是轮询策略，但有两个干扰因素：

• 无效Endpoint干扰：部分请求会被转发到无效的frontend Pod，只有成功的请求才会落到正常abc Pod上，容易造成"只转发到单个Pod"的错觉；
• Nginx长连接复用：配置中的proxy_set_header Connection keep-alive会让Nginx和abc-service保持长连接，kube-proxy对长连接会固定路由到同一个Pod，短时间内请求都会集中到一个Pod。

要让Nginx正确轮询所有abc Pod，建议修改Nginx配置，开启域名解析和失败重试：

resolver kube-dns.kube-system.svc.cluster.local valid=30s;

server {
    listen       80;
    listen  [::]:80;
    server_name  127.0.0.1;

    location / {
      proxy_pass http://abc-service;
      proxy_set_header Connection keep-alive;
      # 失败时自动切换到下一个后端
      proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;
    }
}

另外，Minikube的LoadBalancer需要执行minikube tunnel才能获取外部IP，你当前用的是NodePort（30590）访问，这部分不影响转发逻辑。

3. 第三个Endpoint的来源 #

第三个Endpoint就是frontend Pod的IP，因为它的标签匹配了abc-service的Selectorapp=abc，属于误选。这个Endpoint完全无效，需要修改frontend Deployment的标签（去掉app=abc），或者修改abc-service的Selector（比如改为app=abc,component=backend），确保只选中abc-deployment的Pod。

内容的提问来源于stack exchange，提问作者newbietostack