已安装SharePlum但Airflow DAG仍报ModuleNotFoundError，求排查方向

错误信息 #

Broken DAG: [/opt/airflow/dags/sharepoint_test.py] Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/opt/airflow/dags/sharepoint_test.py", line 8, in <module>
    from shareplum import Site, Office365
ModuleNotFoundError: No module named 'shareplum'

已验证情况 #

执行pip list | grep Sha确认SharePlum已安装：

airflow@airflowhost:/opt/airflow$ pip list | grep Sha
SharePlum                                0.5.1

报错代码片段 #

import csv
from airflow.models import DAG
from airflow.operators.python import PythonOperator
from airflow.utils.dates import days_ago
from shareplum import Site, Office365  # 此处导入报错
from shareplum.site import Version
import json, os

USERNAME = ''
PASSWORD = ''
SHAREPOINT_URL = ''
SHAREPOINT_SITE = ''
SHAREPOINT_DOC = ''

class SharePoint:
    def auth(self):
        self.authcookie = Office365(SHAREPOINT_URL, username=USERNAME, password=PASSWORD).GetCookies()
        self.site = Site(SHAREPOINT_SITE, version=Version.v365, authcookie=self.authcookie)
        return self.site

    def connect_folder(self, folder_name):
        self.auth_site = self.auth()
        self.sharepoint_dir = '\\'.join([SHAREPOINT_DOC, folder_name])
        self.folder = self.auth_site.Folder(self.sharepoint_dir)
        return self.folder

    def download_file(self, file_name, folder_name):
        self._folder = self.connect_folder(folder_name)
        return self._folder.get_file(file_name)

def process_file():
    file_name = 'sample.pdf'
    folder_name = 'Reference'
    file  = SharePoint().download_file(file_name, folder_name)

with DAG("sharepoint_test",
         schedule_interval=None,
         start_date=days_ago(2)) as dag:

    process_file = PythonOperator(task_id="process-file",
                                  python_callable=process_file)

    process_file

• Python环境不匹配：你用pip安装SharePlum的Python环境，和Airflow运行DAG时使用的环境不是同一个。比如Airflow用pip3对应Python3，而你把包装到了Python2；或者Airflow使用虚拟环境，包却装在了全局环境。

  • 解决：执行airflow info查看Airflow使用的Python路径，用对应路径的pip重新安装，比如/path/to/airflow/python -m pip install shareplum。

• Airflow组件环境不一致：分布式部署时，Scheduler机器装了SharePlum，但Worker机器没装；或者Worker使用独立虚拟环境，未安装该包。

  • 解决：在所有运行Airflow Scheduler和Worker的机器上，确保对应Python环境安装SharePlum。

• PythonOperator自定义环境冲突：如果PythonOperator指定了python_path参数，或者配置了PYTHONPATH环境变量，导致运行时无法识别全局安装的包。

  • 解决：检查PythonOperator参数，确保指定的环境包含SharePlum；调整PYTHONPATH加入包所在目录。

• 包安装权限问题：SharePlum被安装到需要管理员权限的目录，Airflow运行用户（如airflow）无访问权限，导致无法导入。

  • 解决：用pip install --user shareplum将包安装到Airflow用户的个人目录，或者给Airflow用户开放包目录的读取权限。

• Airflow缓存未更新：DAG解析缓存未刷新，导致无法识别新安装的包。

  • 解决：重启Airflow Scheduler和Worker服务，或手动清理Airflow缓存目录。

内容的提问来源于stack exchange，提问作者Henry