You need to enable JavaScript to run this app.
优惠活动
大模型
产品
解决方案
定价
更多

Azure App Service部署Airflow运行大数据DAG出现PostgreSQL域名解析失败怎么解决

avatar
API 探路者
2026-9-28
问题背景

我查阅了Stack Overflow上的同类问题，大多针对Docker环境，对我的场景没有参考价值。我的业务场景如下：
我们将Airflow的Docker镜像部署在Azure App Service上，对接托管的Azure Database for PostgreSQL server（版本11），环境相关版本信息如下：

Python = 3.8  
Apache Airflow = 2.1.4  
SQL Alchemy = 1.3.24  
Executor = Local

该环境大部分时间运行正常，但在运行处理数GB级大量数据的DAG时，会突然出现心跳异常问题。我已尝试在Airflow Config中通过sql_alchemy_connect_args参数配置Keep Alives，也调高了web_server_master_timeout和web_server_worker_timeout的参数值，均未解决问题。
报错信息如下：

{base_job.py:222} ERROR - LocalTaskJob heartbeat got an exception
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2336, in _wrap_pool_connect
        return fn()
      File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 364, in connect
        return _ConnectionFairy._checkout(self)
      File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 778, in _checkout
        fairy = _ConnectionRecord.checkout(pool)
      File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 495, in checkout
        rec = pool._do_get()
      File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/impl.py", line 241, in _do_get
        return self._create_connection()
      File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 309, in _create_connection
        return _ConnectionRecord(self)
      File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 440, in __init__
        self.__connect(first_connect_check=True)
      File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 661, in __connect
        pool.logger.debug("Error on connect(): %s", e)
      File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
        compat.raise_(
      File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
        raise exception
      File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 656, in __connect
        connection = pool._invoke_creator(self)
      File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
        return dialect.connect(*cargs, **cparams)
      File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 508, in connect
        return self.dbapi.connect(*cargs, **cparams)
      File "/usr/local/lib/python3.8/site-packages/psycopg2/__init__.py", line 122, in connect
        conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
    psycopg2.OperationalError: could not translate host name "<address>" to address: Temporary failure in name resolution
    
    
    The above exception was the direct cause of the following exception:
    
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/site-packages/airflow/jobs/base_job.py", line 194, in heartbeat
        session.merge(self)
      File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 2166, in merge
        return self._merge(
      File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 2244, in _merge
        merged = self.query(mapper.class_).get(key[1])
      File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 1018, in get
        return self._get_impl(ident, loading.load_on_pk_identity)
      File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 1135, in _get_impl
        return db_load_fn(self, primary_key_identity)
      File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/loading.py", line 286, in load_on_pk_identity
        return q.one()
      File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3490, in one
        ret = self.one_or_none()
      File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3459, in one_or_none
        ret = list(self)
      File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3535, in __iter__
        return self._execute_and_instances(context)
      File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3556, in _execute_and_instances
        conn = self._get_bind_args(
      File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3571, in _get_bind_args
        return fn(
      File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3550, in _connection_from_session
        conn = self.session.connection(**kw)
      File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 1142, in connection
        return self._connection_for_bind(
      File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 1150, in _connection_for_bind
        return self.transaction._connection_for_bind(
      File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 433, in _connection_for_bind
        conn = bind._contextual_connect()
      File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2302, in _contextual_connect
        self._wrap_pool_connect(self.pool.connect, None),
      File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2339, in _wrap_pool_connect
        Connection._handle_dbapi_exception_noconnection(
      File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1583, in _handle_dbapi_exception_noconnection
        util.raise_(
      File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
        raise exception
      File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2336, in _wrap_pool_connect
        return fn()
      File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 364, in connect
        return _ConnectionFairy._checkout(self)
      File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 778, in _checkout
        fairy = _ConnectionRecord.checkout(pool)
      File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 495, in checkout
        rec = pool._do_get()
      File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/impl.py", line 241, in _do_get
        return self._create_connection()
      File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 309, in _create_connection
        return _ConnectionRecord(self)
      File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 440, in __init__
        self.__connect(first_connect_check=True)
      File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 661, in __connect
        pool.logger.debug("Error on connect(): %s", e)
      File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
        compat.raise_(
      File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
        raise exception
      File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 656, in __connect
        connection = pool._invoke_creator(self)
      File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
        return dialect.connect(*cargs, **cparams)
      File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 508, in connect
        return self.dbapi.connect(*cargs, **cparams)
      File "/usr/local/lib/python3.8/site-packages/psycopg2/__init__.py", line 122, in connect
        conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
    sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) could not translate host name "<address>" to address: Temporary failure in name resolution
根因分析

报错核心是PostgreSQL主机名临时解析失败，结合场景触发条件为大流量数GB数据处理时出现，根因可归为以下几类：

  • Azure App Service的DNS缓存耗尽：处理大数据时，Airflow worker和PostgreSQL的交互频率骤升，App Service默认的DNS缓存TTL较短，高并发请求下超出缓存容量会触发频繁递归查询，Azure的递归DNS服务有QPS限制，超限后会丢弃请求导致解析失败
  • 网络带宽打满：大数据处理时出站带宽被数据传输占满，DNS查询报文被丢包无法得到响应
  • SQLAlchemy连接池配置不合理：1.3.24版本的SQLAlchemy默认未开启连接池回收，长连接断开后重连时刚好遇到DNS解析异常，此前配置的keepalive仅针对已建立的TCP连接，对新建立连接的DNS解析阶段无效
  • 你之前调整的web_server_master_timeout、web_server_worker_timeout属于Web UI的超时配置，和任务心跳的数据库连接逻辑无关，因此无法解决该问题
排查&解决思路
  • 验证DNS解析稳定性：在DAG中增加前置调试步骤，运行大数据处理任务前持续nslookup PostgreSQL域名5分钟，记录解析成功率和耗时，确认高负载下是否存在解析成功率下降的情况
  • 优化App Service DNS配置：在App Service的应用配置中添加WEBSITE_DNS_SERVER变量，设置为168.63.129.16（Azure内部递归DNS地址），同时添加WEBSITE_ENABLE_DNS_CACHE=true开启内置DNS缓存，延长缓存TTL
  • 调整SQLAlchemy连接池参数：修改Airflow的sql_alchemy_connect_args配置，增加以下参数：
    {
  "pool_recycle": 300,
  "pool_pre_ping": True,
  "pool_size": 10,
  "max_overflow": 20
}
    其中pool_pre_ping会在连接取出前验证可用性，pool_recycle强制5分钟回收一次连接，避免长时间闲置连接断开后重连失败
  • 配置静态解析跳过DNS查询：如果上述方案都无效，可以直接在App Service的启动脚本中把PostgreSQL的域名和对应IP写入/etc/hosts文件，完全规避DNS解析环节
  • 排查带宽占用：在Azure监控面板查看App Service的出站带宽指标，确认大任务运行时是否达到实例带宽上限，如果超限升级App Service的实例规格

内容的提问来源于stack exchange，提问作者Vinay Kulkarni

相关产品推荐
方舟 Agent Plan

超全模态模型 × Harness 升级，最新支持 Deepseek-V4.1-Flash、GLM-5.3 系列、Doubao-Seedream-5.0-pro、Kimi-K3 (部分)， 限时 9.9 元起

产品介绍相关文档
相关产品推荐
方舟 Agent Plan

超全模态模型 × Harness 升级，最新支持 Deepseek-V4.1-Flash、GLM-5.3 系列、Doubao-Seedream-5.0-pro、Kimi-K3 (部分)， 限时 9.9 元起

产品介绍相关文档
火山引擎 最新活动
全面开放
Doubao-Seedance-2.5全新模型升级

模型再升级,30秒超长叙事, 模态参考扩容

HOT
方舟Codingplan

模型自由，工具不限，最新支持 Deepseek-V4 系列、GLM-5.3 系列

HOT
方舟Agent plan

超全模态模型 × Harness 升级，最新支持 Deepseek-V4.1-Flash、GLM-5.3 系列

HOT
新用户特惠专场

大模型19元起，Al应用9.9元畅享，新人首购爆款尽享优惠

最近更新时间：2026.09.28 21:06:01