Kubernetes中Pod可Ping通Redis服务但无法建立连接求助

首次在Kubernetes部署Pod时遇到连通性问题：SpringBoot应用Pod可通过Redis服务名redis-task-queue-service ping通Redis Pod，但无法建立Redis连接。以下是相关信息，请求排查原因：

可Ping通验证 #

/app # ping redis-task-queue-service
PING redis-task-queue-service (10.244.7.237): 56 data bytes
64 bytes from 10.244.7.237: seq=0 ttl=64 time=0.172 ms
64 bytes from 10.244.7.237: seq=1 ttl=64 time=0.108 ms

连接失败日志 #

2023-06-27T01:23:37.643Z  INFO 1 --- [           main] .c.s.SignalCoverageControllerApplication : Starting SignalCoverageControllerApplication v0.1.0-dev.0.uncommitted+DUST.6448.k8s.e3852e9 using Java 17-ea with PID 1 (/app/app.jar started by root in /app)
2023-06-27T01:23:37.648Z  INFO 1 --- [           main] .c.s.SignalCoverageControllerApplication : No active profile set, falling back to 1 default profile: "default"
2023-06-27T01:23:38.547Z  INFO 1 --- [           main] .s.d.r.c.RepositoryConfigurationDelegate : Multiple Spring Data modules found, entering strict repository configuration mode
2023-06-27T01:23:38.551Z  INFO 1 --- [           main] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data Redis repositories in DEFAULT mode.
2023-06-27T01:23:38.599Z  INFO 1 --- [           main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 26 ms. Found 0 Redis repository interfaces.
2023-06-27T01:23:39.363Z  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port(s): 8081 (http)
2023-06-27T01:23:39.378Z  INFO 1 --- [           main] o.apache.catalina.core.StandardService   : Starting service [Tomcat]
2023-06-27T01:23:39.378Z  INFO 1 --- [           main] o.apache.catalina.core.StandardEngine    : Starting Servlet engine: [Apache Tomcat/10.1.8]
2023-06-27T01:23:39.566Z  INFO 1 --- [           main] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
2023-06-27T01:23:39.568Z  INFO 1 --- [           main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 1832 ms
2023-06-27T01:23:40.872Z ERROR 1 --- [           main] a.c.c.s.config.RedisConfig               : Error trying to connect to Redis at redis-task-queue-service:6380 - Unable to connect to Redis
2023-06-27T01:23:41.450Z  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 8081 (http) with context path ''
2023-06-27T01:23:41.464Z  INFO 1 --- [           main] .c.s.SignalCoverageControllerApplication : Started SignalCoverageControllerApplication in 4.514 seconds (process running for 5.336)
2023-06-27T01:23:41.467Z  INFO 1 --- [           main] c.c.socketio.SocketIOServer              : Session store / pubsub factory used: MemoryStoreFactory (local session store only)
2023-06-27T01:23:41.495Z  INFO 1 --- [           main] a.c.c.s.r.connection.RedisConnection     : Attempting connection to Redis @ redis-task-queue-service:6380
2023-06-27T01:23:41.495Z  INFO 1 --- [ntLoopGroup-7-1] c.c.socketio.SocketIOServer              : SocketIO server started at port: 3001
2023-06-27T01:23:41.501Z ERROR 1 --- [           main] .c.s.SignalCoverageControllerApplication : No Redis instance as per configuration available to connect to. Exiting.
2023-06-27T01:23:41.582Z  WARN 1 --- [ionShutdownHook] .s.c.a.CommonAnnotationBeanPostProcessor : Destroy method on bean with name 'redisConnection' threw an exception: java.lang.NullPointerException: Cannot invoke "io.lettuce.core.api.StatefulRedisConnection.close()" because "this.connection" is null

Redis Service信息 #

redis-task-queue-service   ClusterIP   None             <none>        6380/TCP            15m

Redis Pod信息 #

redis-task-queue-0                           1/1     Running   0              15m     10.244.7.237   minikube   <none>           <none>

Redis Service配置文件（redis-task-queue-service.yaml） #

apiVersion: v1
kind: Service
metadata:
  name: {{ .Values.taskQueueName }}-service
  labels:
    app: {{ .Values.taskQueueName }}
spec:
  ports:
  - port: {{ .Values.taskQueuePort }}
    name: {{ .Values.taskQueueName }}
  clusterIP: None
  selector:
    app: {{ .Values.taskQueueName }}

参数说明：taskQueuePort为6380，taskQueueName为"redis-task-queue"

SpringBoot应用部署配置文件（controller-deployment.yaml） #

apiVersion: apps/v1
kind: Deployment
metadata:
  name: signal-coverage-controller
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      app: signal-coverage-controller
  template:
    metadata:
      labels:
        app: signal-coverage-controller
    spec:
      containers:
        - name: signal-coverage-controller
          image: {{ .Values.controllerImage }}
          imagePullPolicy: {{ .Values.imagePullPolicy }}
          env:
            - name: REDIS_HOST
              value: "{{ .Values.redisHost }}"
            - name: REDIS_PORT
              value: "{{ .Values.redisPort }}"
            - name: CONTROLLER_TIMEOUT_MS
              value: "{{ .Values.controllerTimeoutMs }}"
            - name: SERVER_PORT
              value: "{{ .Values.serverPort }}"
            - name: WEBSOCKET_HOST
              value: "{{ .Values.websocketHost }}"
            - name: WEBSOCKET_PORT
              value: "{{ .Values.websocketPort }}"
          resources:
            requests:
              #cpu: {{ .Values.resources.requests.cpu }}
              memory: {{ .Values.resources.requests.memory }}
            limits:
              #cpu: {{ .Values.resources.limits.cpu }}
              memory: {{ .Values.resources.limits.memory }}
      imagePullSecrets:
        - name: {{ .Values.secretName }}

环境变量说明：REDIS_HOST为"redis-task-queue-service"，REDIS_PORT为6380

排查方向 #

• 端口映射不匹配：当前Redis Service未指定targetPort，默认会用port（6380）作为Pod的目标端口，但如果Redis容器实际监听的是默认6379端口，就会导致连接失败。可以进入Redis Pod执行ss -tulpn查看实际监听端口，或者检查Redis启动命令确认配置的端口。
• Redis绑定限制：如果Redis配置了bind 127.0.0.1，会只允许本地连接，拒绝外部Pod的访问。可以查看Redis配置文件或执行redis-cli config get bind确认运行时配置。
• 网络策略拦截：检查当前命名空间是否存在NetworkPolicy，是否阻止了SpringBoot Pod访问Redis Pod的6380端口。执行kubectl get networkpolicy查看策略规则。
• 应用连接参数错误：比如Redis开启了密码认证，但应用未配置密码；或者Redis启用了SSL，但应用未开启对应配置。检查应用的Redis连接参数是否与Redis实际配置一致。

内容的提问来源于stack exchange，提问作者Ben Whitely