You need to enable JavaScript to run this app. Docker Airflow安装Snowflake Provider报Snowflake模块导入错误
优惠活动
大模型
产品
解决方案
定价
更多

Docker Airflow安装Snowflake Provider报Snowflake模块导入错误

Airflow 部署 Snowflake Provider 导入报错排查

问题现象

  • 已在Docker部署的Airflow环境中完成apache-airflow-providers-snowflake包安装
  • 运行Snowflake关联数据处理任务时触发报错:No module named Snowflake
  • 报错堆栈指向Snowflake Hook的导入逻辑,经初步定位,问题为模块自引用导入冲突:存放在可导入路径下的Snowflake Hook源码文件被命名为snowflake.py,文件内部存在导入官方snowflake-connector-python模块的逻辑,触发了Python导入机制的路径优先级冲突。

验证用测试代码

以下为用于复现问题的测试DAG代码:

import logging

import airflow
import datetime
from airflow import DAG
#from snowflake.connector import *
#import snowflake.connector
from airflow.operators.python import PythonOperator
from airflow.providers.snowflake.hooks.snowflake import SnowflakeHook


logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

args = {"owner": "Airflow", "start_date": datetime.datetime.now()}

dag = DAG(
    "snowflake_poc", default_args=args, schedule_interval=None
)

#create_insert_query = [
#    """INSERT INTO STRUCTURED.error_details_log (SELECT PARSE_JSON('{"row_num": "13456"}'),current_date());""",
#]


def row_count(**context):
    dwh_hook = SnowflakeHook(snowflake_conn_id="snowflake_conn")
    result = dwh_hook.get_first("select count(*) from STRUCTURED.LKP_GENDER")
    print(result)
    logging.info("Number of rows in `STRUCTURED.LKP_GENDER`  - %s", result[0])


with dag:
    get_count = PythonOperator(task_id="get_count", python_callable=row_count)
get_count

环境依赖清单

以下为当前Airflow运行环境下执行pip freeze获取的全量已安装依赖版本:

$ pip freeze
adal==1.2.6
alembic==1.5.5
amqp==2.6.1
apache-airflow @ file:///opt/airflow
apache-airflow-providers-snowflake==1.1.0
apispec==3.3.2
appdirs==1.4.4
argcomplete==1.12.2
asn1crypto==1.4.0
attrs==20.3.0
azure-batch==10.0.0
azure-common==1.1.26
azure-core==1.11.0
azure-cosmos==3.2.0
azure-datalake-store==0.0.51
azure-identity==1.5.0
azure-keyvault==4.1.0
azure-keyvault-certificates==4.2.1
azure-keyvault-keys==4.3.1
azure-keyvault-secrets==4.2.0
azure-kusto-data==0.0.45
azure-mgmt-containerinstance==1.5.0
azure-mgmt-core==1.2.2
azure-mgmt-datalake-nspkg==3.0.1
azure-mgmt-datalake-store==0.5.0
azure-mgmt-nspkg==3.0.2
azure-mgmt-resource==15.0.0
azure-nspkg==3.0.2
azure-storage-blob==12.7.1
azure-storage-common==2.1.0
azure-storage-file==2.1.0
Babel==2.9.0
bcrypt==3.2.0
billiard==3.6.3.0
boto3==1.15.18
botocore==1.18.18
cached-property==1.5.2
cachetools==4.2.1
cattrs==1.2.0
celery==4.4.7
certifi==2020.12.5
cffi==1.14.5
chardet==3.0.4
click==7.1.2
clickclick==20.10.2
cloudpickle==1.4.1
colorama==0.4.4
colorlog==4.7.2
commonmark==0.9.1
connexion==2.7.0
croniter==0.3.37
cryptography==3.4.6
dask==2021.2.0
defusedxml==0.6.0
dill==0.3.3
distlib==0.3.1
distributed==2.19.0
dnspython==1.16.0
docker==3.7.3
docker-pycreds==0.4.0
docutils==0.16
elasticsearch==7.5.1
elasticsearch-dbapi==0.1.0
elasticsearch-dsl==7.3.0
email-validator==1.1.2
eventlet==0.30.1
filelock==3.0.12
Flask==1.1.2
Flask-AppBuilder==3.1.1
Flask-Babel==1.0.0
Flask-Caching==1.9.0
Flask-JWT-Extended==3.25.1
Flask-Login==0.4.1
Flask-OpenID==1.2.5
Flask-SQLAlchemy==2.4.4
Flask-WTF==0.14.3
flower==0.9.7
gevent==21.1.2
google-ads==7.0.0
google-api-core==1.26.0
google-api-python-client==1.12.8
google-auth==1.27.0
google-auth-httplib2==0.0.4
google-auth-oauthlib==0.4.2
google-cloud-automl==2.1.0
google-cloud-bigquery==2.9.0
google-cloud-bigquery-datatransfer==3.0.0
google-cloud-bigquery-storage==2.3.0
google-cloud-bigtable==1.7.0
google-cloud-container==1.0.1
google-cloud-core==1.6.0
google-cloud-datacatalog==3.0.0
google-cloud-dataproc==2.2.0
google-cloud-dlp==1.0.0
google-cloud-kms==2.2.0
google-cloud-language==1.3.0
google-cloud-logging==2.2.0
google-cloud-memcache==0.3.0
google-cloud-monitoring==2.0.1
google-cloud-os-login==2.1.0
google-cloud-pubsub==2.3.0
google-cloud-redis==2.0.0
google-cloud-secret-manager==1.0.0
google-cloud-spanner==1.19.1
google-cloud-speech==1.3.2
google-cloud-storage==1.36.0
google-cloud-tasks==2.1.0
google-cloud-texttospeech==1.0.1
google-cloud-translate==1.7.0
google-cloud-videointelligence==1.16.1
google-cloud-vision==1.0.0
google-cloud-workflows==0.2.0
google-crc32c==1.1.2
google-resumable-media==1.2.0
googleapis-common-protos==1.52.0
graphviz==0.16
greenlet==1.0.0
grpc-google-iam-v1==0.12.3
grpcio==1.35.0
grpcio-gcp==0.2.2
gunicorn==19.10.0
HeapDict==1.0.1
httplib2==0.19.0
humanize==3.2.0
hvac==0.10.8
idna==2.10
importlib-metadata==1.7.0
importlib-resources==1.5.0
inflection==0.5.1
iso8601==0.1.14
isodate==0.6.0
itsdangerous==1.1.0
Jinja2==2.11.3
jmespath==0.10.0
json-merge-patch==0.2
jsonschema==3.2.0
kombu==4.6.11
kubernetes==11.0.0
lazy-object-proxy==1.4.3
ldap3==2.9
libcst==0.3.17
lockfile==0.12.2
Mako==1.1.4
Markdown==3.3.3
MarkupSafe==1.1.1
marshmallow==3.10.0
marshmallow-enum==1.5.1
marshmallow-oneofschema==2.1.0
marshmallow-sqlalchemy==0.23.1
msal==1.9.0
msal-extensions==0.3.0
msgpack==1.0.2
msrest==0.6.21
msrestazure==0.6.4
mypy-extensions==0.4.3
mysql-connector-python==8.0.22
mysqlclient==1.3.14
natsort==7.1.1
numpy==1.20.1
oauthlib==2.1.0
openapi-spec-validator==0.2.9
oscrypto==1.2.1
packaging==20.9
pandas==1.2.2
pandas-gbq==0.14.1
paramiko==2.7.2
pendulum==2.1.2
portalocker==1.7.1
prison==0.1.3
prometheus-client==0.8.0
proto-plus==1.13.0
protobuf==3.15.1
psutil==5.8.0
psycopg2-binary==2.8.6
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycparser==2.20
pycryptodomex==3.10.1
pydata-google-auth==1.1.0
Pygments==2.8.0
PyJWT==1.7.1
PyNaCl==1.4.0
pyOpenSSL==19.1.0
pyparsing==2.4.7
pyrsistent==0.17.3
pysftp==0.2.9
python-daemon==2.3.0
python-dateutil==2.8.1
python-editor==1.0.4
python-http-client==3.3.2
python-ldap==3.3.1
python-nvd3==0.15.0
python-slugify==4.0.1
python3-openid==3.2.0
pytz==2020.5
pytzdata==2020.1
PyYAML==5.4.1
redis==3.5.3
requests==2.25.1
requests-oauthlib==1.1.0
rich==9.2.0
rsa==4.7.1
s3transfer==0.3.4
sendgrid==6.6.0
setproctitle==1.2.2
six==1.15.0
slack-sdk==3.4.0
snowflake-connector-python==2.3.10
snowflake-sqlalchemy==1.2.4
sortedcontainers==2.3.0
SQLAlchemy==1.3.23
SQLAlchemy-JSONField==1.0.0
SQLAlchemy-Utils==0.36.8
sshtunnel==0.1.5
starkbank-ecdsa==1.1.0
statsd==3.3.0
swagger-ui-bundle==0.0.8
tabulate==0.8.8
tblib==1.7.0
tenacity==6.2.0
termcolor==1.1.0
text-unidecode==1.3
toolz==0.11.1
tornado==6.1
typing-extensions==3.7.4.3
typing-inspect==0.6.0
unicodecsv==0.14.1
uritemplate==3.0.1
urllib3==1.25.11
vine==1.3.0
virtualenv==20.4.2
watchtower==0.7.3
websocket-client==0.57.0
Werkzeug==1.0.1
WTForms==2.3.3
zict==2.0.0
zipp==3.4.0
zope.event==4.5.0
zope.interface==5.2.0

根因说明

Python的模块导入机制会按照sys.path的顺序检索可加载的模块文件,路径优先级为:当前脚本所在工作目录 > 项目可导入目录 > site-packages全局安装目录。
当在Airflow的DAG目录、插件目录或运行工作目录下存在名为snowflake.py的手动拷贝Hook文件时,代码执行from snowflake import connector这类导入逻辑时,解释器会优先加载这个本地的snowflake.py,而非全局安装的snowflake-connector-python官方包。而这个本地的snowflake.py文件本身又需要导入官方Snowflake SDK模块,就形成了自引用的循环导入,最终抛出No module named Snowflake的错误。

解决步骤

  1. 排查Airflow运行时的所有可导入路径
    • 检查dags/、plugins/、容器运行工作目录下的所有文件,找到所有命名为snowflake.py的冲突文件
    • 这类手动放置的源码文件完全多余:已经通过pip安装了apache-airflow-providers-snowflake,对应Hook的源码已经随包部署到全局site-packages路径下,无需单独拷贝
    • 将冲突文件从可导入路径中移除或重命名,保证文件名不与第三方包同名
  2. 导入有效性验证
    • 进入Airflow运行容器,执行以下命令验证模块是否可正常加载:
      python -c "from airflow.providers.snowflake.hooks.snowflake import SnowflakeHook; print('导入校验通过')"
    • 无报错输出则说明冲突已解除
  3. 版本兼容性确认
    从现有依赖清单来看,Snowflake相关组件的版本组合完全兼容:
    • Provider版本:apache-airflow-providers-snowflake==1.1.0
    • 连接器版本:snowflake-connector-python==2.3.10
    • ORM适配版本:snowflake-sqlalchemy==1.2.4
      无需额外升级或调整依赖版本,解决同名文件冲突后即可正常运行任务。

内容的提问来源于stack exchange,提问作者Vidya821

相关产品推荐
方舟 Agent Plan

超全模态模型 × Harness 升级,最新支持 Deepseek-V4.1-Flash、GLM-5.3 系列、Doubao-Seedream-5.0-pro、Kimi-K3 (部分), 限时 9.9 元起

最近更新时间:2026.11.01 18:15:10