AWS EKS中MongoDB的getaddrinfo EAI_AGAIN DNS问题求助

问题场景与现象 #

在AWS EKS集群中部署MongoDB实例后，从与EKS节点组同VPC的堡垒机访问该Pod时，执行mongosh命令出现以下错误：

mongosh --host 10.40.236.36 --port 30000 --authenticationDatabase admin --username xxx --password
Enter password: *****************Current sessionID:  62f6f1c02c2833b3fbe209a2
Connecting to:    mongodb://10.40.236.36:30000
getaddrinfo EAI_AGAIN myapp-migration-test-db-0.myapp-migration-test-db-svc.mongodb.svc.cluster.local

网络连通性验证 #

堡垒机到MongoDB节点的网络连通性正常：

ubuntu@ip-10-40-1-54:~$ telnet 10.40.236.36 30000
Trying 10.40.236.36...
Connected to 10.40.236.36

命令执行差异 #

mongorestore/mongodump命令可以正常执行：

ubuntu@ip-10-40-1-54:~$ mongorestore -u app --host 10.40.236.36 --port 30000 ~/data
Enter password:

2022-08-13T01:36:21.136+0000    preparing collections to restore from
2022-08-13T01:36:21.137+0000    0 document(s) restored successfully. 0 document(s) failed to restore.

但mongosh/mongomirror会触发getaddrinfo EAI_AGAIN错误。

相关Kubernetes资源描述 #

NodePort Service（migration-nodeport-svc） #

kubectl describe svc migration-nodeport-svc
Name:                     migration-nodeport-svc
Namespace:                mongodb
Labels:                   app=myapp-migration-test-db-svc
                          controller=mongodb-enterprise-operator
                          controller-revision-hash=myapp-migration-test-db-8bc7c6fd5
                          pod-anti-affinity=myapp-migration-test-db
                          statefulset.kubernetes.io/pod-name=myapp-migration-test-db-0
Selector:                 app=myapp-migration-test-db-svc,controller-revision-hash=myapp-migration-test-db-8bc7c6fd5,controller=mongodb-enterprise-operator,pod-anti-affinity=myapp-migration-test-db,statefulset.kubernetes.io/pod-name=myapp-migration-test-db-0
Type:                     NodePort
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       172.20.127.110
IPs:                      172.20.127.110
Port:                     <unset>  27017/TCP
TargetPort:               27017/TCP
NodePort:                 <unset>  30000/TCP
Endpoints:                10.40.84.120:27017
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

ClusterIP Service（myapp-migration-test-db-svc） #

(base) ➜  ~ kubectl describe svc myapp-migration-test-db-svc
Name:              myapp-migration-test-db-svc
Namespace:         mongodb
Labels:            app=myapp-migration-test-db-svc
                   controller=mongodb-enterprise-operator
Annotations:       <none>
Selector:          app=myapp-migration-test-db-svc,controller=mongodb-enterprise-operator
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                None
IPs:               None
Port:              mongodb  27017/TCP
TargetPort:        27017/TCP
Endpoints:         10.40.185.120:27017,10.40.222.131:27017,10.40.90.18:27017
Session Affinity:  None
Events:            <none>

MongoDB Pod（myapp-migration-test-db-0） #

kubectl describe pod myapp-migration-test-db-0
Name:                 myapp-migration-test-db-0
Namespace:            mongodb
Node:                 ip-10-40-236-36.ec2.internal/10.40.236.36
Labels:               app=myapp-migration-test-db-svc
                      controller=mongodb-enterprise-operator
                      controller-revision-hash=myapp-migration-test-db-8bc7c6fd5
                      pod-anti-affinity=myapp-migration-test-db
                      statefulset.kubernetes.io/pod-name=myapp-migration-test-db-0
Status:               Running
IP:                   10.40.84.120
IPs:
  IP:           10.40.84.120
Controlled By:  StatefulSet/myapp-migration-test-db
Init Containers:
  mongodb-enterprise-init-database:
    Image:          quay.io/mongodb/mongodb-enterprise-init-database:1.0.11
    Image ID:       docker-pullable://quay.io/mongodb/mongodb-enterprise-init-database@sha256:xxx
    Host Port:      <none>
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Fri, 12 Aug 2022 10:40:43 -0400
      Finished:     Fri, 12 Aug 2022 10:40:43 -0400
    Ready:          True
    Restart Count:  0
    Environment:
      AWS_DEFAULT_REGION:           us-east-1
      AWS_REGION:                   us-east-1
      AWS_ROLE_ARN:                 arn:aws:iam::xxxx:role/yyy
      AWS_WEB_IDENTITY_TOKEN_FILE:  /var/run/secrets/eks.amazonaws.com/serviceaccount/token
    Mounts:
      /opt/scripts from database-scripts (rw)
      /var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-hj5rn (ro)
Containers:
  mongodb-enterprise-database:
    Image:         quay.io/mongodb/mongodb-enterprise-database:2.0.2
    Image ID:      docker-pullable://quay.io/mongodb/mongodb-enterprise-database@xxx
    Port:          27017/TCP
    Host Port:     0/TCP
    Command:
      /opt/scripts/agent-launcher.sh
    State:          Running
      Started:      Fri, 12 Aug 2022 10:40:47 -0400
    Ready:          True
    Restart Count:  0
    Liveness:       exec [/opt/scripts/probe.sh] delay=60s timeout=30s period=30s #success=1 #failure=6
    Readiness:      exec [/opt/scripts/readinessprobe] delay=5s timeout=1s period=5s #success=1 #failure=4
    Environment:
      AGENT_FLAGS:                         -logFile,/var/log/mongodb-mms-automation/automation-agent.log,
      BASE_URL:                            https://mongodb.myapp.tools
      GROUP_ID:                            xxxxx
      LOG_LEVEL:                           WARN
      SSL_REQUIRE_VALID_MMS_CERTIFICATES:  true
      USER_LOGIN:                          nexuobkj
      AWS_DEFAULT_REGION:                  us-east-1
      AWS_REGION:                          us-east-1
      AWS_ROLE_ARN:                        arn:aws:iam::xxxx:role/yyy
      AWS_WEB_IDENTITY_TOKEN_FILE:         /var/run/secrets/eks.amazonaws.com/serviceaccount/token
    Mounts:
      /data from data (rw,path="data")
      /journal from data (rw,path="journal")
      /mongodb-automation/agent-api-key from agent-api-key (rw)
      /mongodb-automation/tls from secret-certs (rw)
      /mongodb-automation/tls/ca from secret-ca (ro)
      /opt/scripts from database-scripts (ro)
      /var/log/mongodb-mms-automation from data (rw,path="logs")
      /var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-hj5rn (ro)

附加信息 #

曾尝试部署LoadBalancer Service使用端口27017，但仍遇到相同问题。

解决思路 #

1. 强制直接连接，绕过集群节点发现 #

错误中的myapp-migration-test-db-0.myapp-migration-test-db-svc.mongodb.svc.cluster.local是Kubernetes内部DNS名称，堡垒机不在集群内无法解析。mongosh/mongomirror默认会尝试获取MongoDB集群的节点列表并连接，而mongorestore/mongodump仅做单节点连接，不会触发节点名称解析。

执行mongosh时添加--directConnection=true参数，强制直接连接到指定IP和端口：

mongosh --host 10.40.236.36 --port 30000 --authenticationDatabase admin --username xxx --password --directConnection=true

2. 检查MongoDB绑定地址与集群配置 #

• 进入MongoDB Pod，查看mongod.conf中的net.bindIp配置，或执行db.adminCommand('getCmdLineOpts')查看启动参数，确认是否绑定了集群内部IP。
• 如果是副本集部署，MongoDB会返回集群节点的内部DNS名称给客户端。若需要外部访问，可调整副本集配置，将节点地址替换为堡垒机能访问的IP或可解析域名；但更简便的方式是使用directConnection直接连接单节点。

3. 验证Service转发逻辑 #

当前NodePort Service的ExternalTrafficPolicy为Cluster，流量会转发到集群内任意节点。虽然telnet连通性正常，但如果是多节点副本集，需确保每个节点都有对应的外部访问入口；若仅需临时访问，使用directConnection即可绕过该问题。

内容的提问来源于stack exchange，提问作者titanium1905