Docker Airflow安装Snowflake Provider报Snowflake模块导入错误
Airflow 部署 Snowflake Provider 导入报错排查
问题现象
- 已在Docker部署的Airflow环境中完成
apache-airflow-providers-snowflake包安装 - 运行Snowflake关联数据处理任务时触发报错:
No module named Snowflake - 报错堆栈指向Snowflake Hook的导入逻辑,经初步定位,问题为模块自引用导入冲突:存放在可导入路径下的Snowflake Hook源码文件被命名为
snowflake.py,文件内部存在导入官方snowflake-connector-python模块的逻辑,触发了Python导入机制的路径优先级冲突。
验证用测试代码
以下为用于复现问题的测试DAG代码:
import logging import airflow import datetime from airflow import DAG #from snowflake.connector import * #import snowflake.connector from airflow.operators.python import PythonOperator from airflow.providers.snowflake.hooks.snowflake import SnowflakeHook logging.basicConfig(level=logging.INFO) logger = logging.getLogger(__name__) args = {"owner": "Airflow", "start_date": datetime.datetime.now()} dag = DAG( "snowflake_poc", default_args=args, schedule_interval=None ) #create_insert_query = [ # """INSERT INTO STRUCTURED.error_details_log (SELECT PARSE_JSON('{"row_num": "13456"}'),current_date());""", #] def row_count(**context): dwh_hook = SnowflakeHook(snowflake_conn_id="snowflake_conn") result = dwh_hook.get_first("select count(*) from STRUCTURED.LKP_GENDER") print(result) logging.info("Number of rows in `STRUCTURED.LKP_GENDER` - %s", result[0]) with dag: get_count = PythonOperator(task_id="get_count", python_callable=row_count) get_count
环境依赖清单
以下为当前Airflow运行环境下执行pip freeze获取的全量已安装依赖版本:
$ pip freeze adal==1.2.6 alembic==1.5.5 amqp==2.6.1 apache-airflow @ file:///opt/airflow apache-airflow-providers-snowflake==1.1.0 apispec==3.3.2 appdirs==1.4.4 argcomplete==1.12.2 asn1crypto==1.4.0 attrs==20.3.0 azure-batch==10.0.0 azure-common==1.1.26 azure-core==1.11.0 azure-cosmos==3.2.0 azure-datalake-store==0.0.51 azure-identity==1.5.0 azure-keyvault==4.1.0 azure-keyvault-certificates==4.2.1 azure-keyvault-keys==4.3.1 azure-keyvault-secrets==4.2.0 azure-kusto-data==0.0.45 azure-mgmt-containerinstance==1.5.0 azure-mgmt-core==1.2.2 azure-mgmt-datalake-nspkg==3.0.1 azure-mgmt-datalake-store==0.5.0 azure-mgmt-nspkg==3.0.2 azure-mgmt-resource==15.0.0 azure-nspkg==3.0.2 azure-storage-blob==12.7.1 azure-storage-common==2.1.0 azure-storage-file==2.1.0 Babel==2.9.0 bcrypt==3.2.0 billiard==3.6.3.0 boto3==1.15.18 botocore==1.18.18 cached-property==1.5.2 cachetools==4.2.1 cattrs==1.2.0 celery==4.4.7 certifi==2020.12.5 cffi==1.14.5 chardet==3.0.4 click==7.1.2 clickclick==20.10.2 cloudpickle==1.4.1 colorama==0.4.4 colorlog==4.7.2 commonmark==0.9.1 connexion==2.7.0 croniter==0.3.37 cryptography==3.4.6 dask==2021.2.0 defusedxml==0.6.0 dill==0.3.3 distlib==0.3.1 distributed==2.19.0 dnspython==1.16.0 docker==3.7.3 docker-pycreds==0.4.0 docutils==0.16 elasticsearch==7.5.1 elasticsearch-dbapi==0.1.0 elasticsearch-dsl==7.3.0 email-validator==1.1.2 eventlet==0.30.1 filelock==3.0.12 Flask==1.1.2 Flask-AppBuilder==3.1.1 Flask-Babel==1.0.0 Flask-Caching==1.9.0 Flask-JWT-Extended==3.25.1 Flask-Login==0.4.1 Flask-OpenID==1.2.5 Flask-SQLAlchemy==2.4.4 Flask-WTF==0.14.3 flower==0.9.7 gevent==21.1.2 google-ads==7.0.0 google-api-core==1.26.0 google-api-python-client==1.12.8 google-auth==1.27.0 google-auth-httplib2==0.0.4 google-auth-oauthlib==0.4.2 google-cloud-automl==2.1.0 google-cloud-bigquery==2.9.0 google-cloud-bigquery-datatransfer==3.0.0 google-cloud-bigquery-storage==2.3.0 google-cloud-bigtable==1.7.0 google-cloud-container==1.0.1 google-cloud-core==1.6.0 google-cloud-datacatalog==3.0.0 google-cloud-dataproc==2.2.0 google-cloud-dlp==1.0.0 google-cloud-kms==2.2.0 google-cloud-language==1.3.0 google-cloud-logging==2.2.0 google-cloud-memcache==0.3.0 google-cloud-monitoring==2.0.1 google-cloud-os-login==2.1.0 google-cloud-pubsub==2.3.0 google-cloud-redis==2.0.0 google-cloud-secret-manager==1.0.0 google-cloud-spanner==1.19.1 google-cloud-speech==1.3.2 google-cloud-storage==1.36.0 google-cloud-tasks==2.1.0 google-cloud-texttospeech==1.0.1 google-cloud-translate==1.7.0 google-cloud-videointelligence==1.16.1 google-cloud-vision==1.0.0 google-cloud-workflows==0.2.0 google-crc32c==1.1.2 google-resumable-media==1.2.0 googleapis-common-protos==1.52.0 graphviz==0.16 greenlet==1.0.0 grpc-google-iam-v1==0.12.3 grpcio==1.35.0 grpcio-gcp==0.2.2 gunicorn==19.10.0 HeapDict==1.0.1 httplib2==0.19.0 humanize==3.2.0 hvac==0.10.8 idna==2.10 importlib-metadata==1.7.0 importlib-resources==1.5.0 inflection==0.5.1 iso8601==0.1.14 isodate==0.6.0 itsdangerous==1.1.0 Jinja2==2.11.3 jmespath==0.10.0 json-merge-patch==0.2 jsonschema==3.2.0 kombu==4.6.11 kubernetes==11.0.0 lazy-object-proxy==1.4.3 ldap3==2.9 libcst==0.3.17 lockfile==0.12.2 Mako==1.1.4 Markdown==3.3.3 MarkupSafe==1.1.1 marshmallow==3.10.0 marshmallow-enum==1.5.1 marshmallow-oneofschema==2.1.0 marshmallow-sqlalchemy==0.23.1 msal==1.9.0 msal-extensions==0.3.0 msgpack==1.0.2 msrest==0.6.21 msrestazure==0.6.4 mypy-extensions==0.4.3 mysql-connector-python==8.0.22 mysqlclient==1.3.14 natsort==7.1.1 numpy==1.20.1 oauthlib==2.1.0 openapi-spec-validator==0.2.9 oscrypto==1.2.1 packaging==20.9 pandas==1.2.2 pandas-gbq==0.14.1 paramiko==2.7.2 pendulum==2.1.2 portalocker==1.7.1 prison==0.1.3 prometheus-client==0.8.0 proto-plus==1.13.0 protobuf==3.15.1 psutil==5.8.0 psycopg2-binary==2.8.6 pyasn1==0.4.8 pyasn1-modules==0.2.8 pycparser==2.20 pycryptodomex==3.10.1 pydata-google-auth==1.1.0 Pygments==2.8.0 PyJWT==1.7.1 PyNaCl==1.4.0 pyOpenSSL==19.1.0 pyparsing==2.4.7 pyrsistent==0.17.3 pysftp==0.2.9 python-daemon==2.3.0 python-dateutil==2.8.1 python-editor==1.0.4 python-http-client==3.3.2 python-ldap==3.3.1 python-nvd3==0.15.0 python-slugify==4.0.1 python3-openid==3.2.0 pytz==2020.5 pytzdata==2020.1 PyYAML==5.4.1 redis==3.5.3 requests==2.25.1 requests-oauthlib==1.1.0 rich==9.2.0 rsa==4.7.1 s3transfer==0.3.4 sendgrid==6.6.0 setproctitle==1.2.2 six==1.15.0 slack-sdk==3.4.0 snowflake-connector-python==2.3.10 snowflake-sqlalchemy==1.2.4 sortedcontainers==2.3.0 SQLAlchemy==1.3.23 SQLAlchemy-JSONField==1.0.0 SQLAlchemy-Utils==0.36.8 sshtunnel==0.1.5 starkbank-ecdsa==1.1.0 statsd==3.3.0 swagger-ui-bundle==0.0.8 tabulate==0.8.8 tblib==1.7.0 tenacity==6.2.0 termcolor==1.1.0 text-unidecode==1.3 toolz==0.11.1 tornado==6.1 typing-extensions==3.7.4.3 typing-inspect==0.6.0 unicodecsv==0.14.1 uritemplate==3.0.1 urllib3==1.25.11 vine==1.3.0 virtualenv==20.4.2 watchtower==0.7.3 websocket-client==0.57.0 Werkzeug==1.0.1 WTForms==2.3.3 zict==2.0.0 zipp==3.4.0 zope.event==4.5.0 zope.interface==5.2.0
根因说明
Python的模块导入机制会按照sys.path的顺序检索可加载的模块文件,路径优先级为:当前脚本所在工作目录 > 项目可导入目录 > site-packages全局安装目录。
当在Airflow的DAG目录、插件目录或运行工作目录下存在名为snowflake.py的手动拷贝Hook文件时,代码执行from snowflake import connector这类导入逻辑时,解释器会优先加载这个本地的snowflake.py,而非全局安装的snowflake-connector-python官方包。而这个本地的snowflake.py文件本身又需要导入官方Snowflake SDK模块,就形成了自引用的循环导入,最终抛出No module named Snowflake的错误。
解决步骤
- 排查Airflow运行时的所有可导入路径
- 检查
dags/、plugins/、容器运行工作目录下的所有文件,找到所有命名为snowflake.py的冲突文件 - 这类手动放置的源码文件完全多余:已经通过pip安装了
apache-airflow-providers-snowflake,对应Hook的源码已经随包部署到全局site-packages路径下,无需单独拷贝 - 将冲突文件从可导入路径中移除或重命名,保证文件名不与第三方包同名
- 检查
- 导入有效性验证
- 进入Airflow运行容器,执行以下命令验证模块是否可正常加载:
python -c "from airflow.providers.snowflake.hooks.snowflake import SnowflakeHook; print('导入校验通过')" - 无报错输出则说明冲突已解除
- 进入Airflow运行容器,执行以下命令验证模块是否可正常加载:
- 版本兼容性确认
从现有依赖清单来看,Snowflake相关组件的版本组合完全兼容:- Provider版本:
apache-airflow-providers-snowflake==1.1.0 - 连接器版本:
snowflake-connector-python==2.3.10 - ORM适配版本:
snowflake-sqlalchemy==1.2.4
无需额外升级或调整依赖版本,解决同名文件冲突后即可正常运行任务。
- Provider版本:
内容的提问来源于stack exchange,提问作者Vidya821
相关产品推荐
相关产品推荐

