Kubernetes集群部署AWX遇数据库迁移失败问题求助

问题现象 #

在Kubernetes集群部署AWX后，awx-task和awx-web Pod持续等待数据库迁移完成，无法正常启动。Pod日志显示重复尝试等待迁移：

$ kubectl logs pod/awx-task-ff5874db-8qqfq
[wait-for-migrations] Waiting for database migrations...
[wait-for-migrations] Attempt 1 of 30
[wait-for-migrations] Waiting 0.5 seconds before next attempt
[wait-for-migrations] Attempt 2 of 30
[wait-for-migrations] Waiting 1 seconds before next attempt
[wait-for-migrations] Attempt 3 of 30
[wait-for-migrations] Waiting 2 seconds before next attempt
[wait-for-migrations] Attempt 4 of 30
[wait-for-migrations] Waiting 4 seconds before next attempt
[wait-for-migrations] Attempt 5 of 30
[wait-for-migrations] Waiting 8 seconds before next attempt
[wait-for-migrations] Attempt 6 of 30
[wait-for-migrations] Waiting 16 seconds before next attempt
[wait-for-migrations] Attempt 7 of 30
[wait-for-migrations] Waiting 30 seconds before next attempt
[wait-for-migrations] Attempt 8 of 30
[wait-for-migrations] Waiting 30 seconds before next attempt
[wait-for-migrations] Attempt 9 of 30
[wait-for-migrations] Waiting 30 seconds before next attempt
[wait-for-migrations] Attempt 10 of 30
[wait-for-migrations] Waiting 30 seconds before next attempt
[wait-for-migrations] Attempt 11 of 30
[wait-for-migrations] Waiting 30 seconds before next attempt
[wait-for-migrations] Attempt 12 of 30
[wait-for-migrations] Waiting 30 seconds before next attempt
[wait-for-migrations] Attempt 13 of 30
[wait-for-migrations] Waiting 30 seconds before next attempt
[wait-for-migrations] Attempt 14 of 30
[wait-for-migrations] Waiting 30 seconds before next attempt
[wait-for-migrations] Attempt 15 of 30
[wait-for-migrations] Waiting 30 seconds before next attempt

手动进入容器执行迁移命令awx-manage migrate，抛出数据库连接错误：

bash-5.1$ awx-manage migrate
Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection
    self.connect()
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 270, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/postgresql/base.py", line 275, in get_new_connection
    connection = self.Database.connect(**conn_params)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/psycopg/connection.py", line 728, in connect
    raise ex.with_traceback(None)
psycopg.OperationalError: connection is bad: Name or service not known

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/bin/awx-manage", line 8, in <module>
    sys.exit(manage())
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/__init__.py", line 184, in manage
    if (connection.pg_version // 10000) < 12:
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/connection.py", line 15, in __getattr__
    return getattr(self._connections[self._alias], item)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/functional.py", line 57, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/postgresql/base.py", line 436, in pg_version
    with self.temporary_connection():
  File "/usr/lib64/python3.9/contextlib.py", line 119, in __enter__
    return next(self.gen)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 705, in temporary_connection
    with self.cursor() as cursor:
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 330, in cursor
    return self._cursor()
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 306, in _cursor
    self.ensure_connection()
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection
    self.connect()
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection
    self.connect()
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 270, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/postgresql/base.py", line 275, in get_new_connection
    connection = self.Database.connect(**conn_params)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/psycopg/connection.py", line 728, in connect
    raise ex.with_traceback(None)
django.db.utils.OperationalError: connection is bad: Name or service not known

问题分析 #

错误信息connection is bad: Name or service not known明确指向AWX容器无法解析数据库服务的域名，导致无法和PostgreSQL建立连接。核心原因可能是：

• 数据库服务的DNS解析失败
• 数据库配置中的服务名称拼写错误
• 数据库Pod或Service未正常运行

解决步骤 #

1. 检查数据库服务状态

   • 确认PostgreSQL Pod是否正常运行：

     kubectl get pods -n <AWX命名空间> | grep postgres
     

   • 确认数据库Service是否存在且状态正常：

     kubectl get svc -n <AWX命名空间> | grep postgres
     

2. 验证DNS解析能力

   • 进入awx-task或awx-web容器，测试数据库服务的DNS解析：

     kubectl exec -it <AWX-Pod名称> -n <AWX命名空间> -- nslookup <PostgreSQL服务名称>
     

   • 如果解析失败，检查集群DNS组件（如CoreDNS）是否正常运行，或确认服务名称拼写是否正确。

3. 核对AWX数据库配置

   • 查看AWX使用的ConfigMap或Secret中的数据库连接参数，确认DATABASE_HOST的值与PostgreSQL Service的名称一致：

     kubectl get configmap <AWX-ConfigMap名称> -n <AWX命名空间> -o yaml
     kubectl get secret <AWX-Secret名称> -n <AWX命名空间> -o yaml | grep -A5 DATABASE_HOST
     

   • 若使用外部数据库，确认DATABASE_HOST的IP或域名正确，且网络连通性正常。

4. 重启相关Pod

   • 修正配置后，删除awx-task和awx-web Pod，让Deployment重新创建实例：

     kubectl delete pod <awx-task-Pod名称> <awx-web-Pod名称> -n <AWX命名空间>
     

内容的提问来源于stack exchange，提问作者rabin sedhain