Azure Pipelines已安装pandas仍遇ModuleNotFoundError问题求助

问题根源 #

每个Pipeline作业都运行在独立的虚拟机实例上，哪怕指定了同一个vmImage，作业之间也不会共享环境、已安装的包或文件。第一个作业InstallPackages中安装的pandas仅存在于它运行的虚拟机中，第二个作业RunScript启动的是全新虚拟机，自然找不到该依赖。

解决方案 #

方案1：合并作业（最简单直接）

将依赖安装和脚本运行放到同一个作业里，共享同一个运行环境：

stages:
- stage: scrape_text
  jobs:
  - job: RunScraper
    displayName: 'Install dependencies and run scraper'
    pool:
      vmImage: 'ubuntu-latest'
    steps:
      - task: UsePythonVersion@0
        inputs:
          versionSpec: '3.9'
          addToPath: true
      - script: |
          python -m pip install --upgrade pip
          python -m pip install pandas
        displayName: 'Install dependencies'
      - task: PythonScript@0
        displayName: 'Run scraper script'
        inputs:
          scriptSource: 'filePath'
          scriptPath: 'path/to/text_scrapper.py'

方案2：使用Pipeline缓存（适合多作业/多次运行场景）

通过Azure DevOps的缓存任务，将pip安装的包缓存起来，第二个作业直接读取缓存：

stages:
- stage: scrape_text
  jobs:
  - job: InstallPackages
    displayName: 'Install packages and cache'
    pool:
      vmImage: 'ubuntu-latest'
    steps:
      - task: UsePythonVersion@0
        inputs:
          versionSpec: '3.9'
          addToPath: true
      - task: Cache@2
        inputs:
          key: 'pip | "$(Agent.OS)" | requirements.txt' # 若有requirements.txt，用它做key更精准
          restoreKeys: |
            pip | "$(Agent.OS)"
          path: $(Agent.ToolsDirectory)/python/3.9.*/x64/lib/python3.9/site-packages
        displayName: 'Cache pip packages'
      - script: |
          python -m pip install --upgrade pip
          python -m pip install pandas
        displayName: 'Install dependencies'

  - job: RunScript
    displayName: 'Scrape text'
    pool:
      vmImage: 'ubuntu-latest'
    dependsOn: InstallPackages
    steps:
    - task: UsePythonVersion@0
      inputs:
        versionSpec: '3.9'
        addToPath: true
    - task: Cache@2
      inputs:
        key: 'pip | "$(Agent.OS)" | requirements.txt'
        restoreKeys: |
          pip | "$(Agent.OS)"
        path: $(Agent.ToolsDirectory)/python/3.9.*/x64/lib/python3.9/site-packages
      - task: PythonScript@0
        displayName: 'Run scraper script'
        inputs:
          scriptSource: 'filePath'
          scriptPath: 'path/to/text_scrapper.py'

方案3：在第二个作业中重复安装依赖

如果必须拆分作业，可在第二个作业中再次执行依赖安装步骤：

stages:
- stage: scrape_text
  jobs:
  - job: InstallPackages
    displayName: 'Install packages'
    pool:
      vmImage: 'ubuntu-latest'
    steps:
      - task: UsePythonVersion@0
        inputs:
          versionSpec: '3.9'
          addToPath: true
      - script: |
          python -m pip install --upgrade pip
          python -m pip install pandas
        displayName: 'Install dependencies'

  - job: RunScript
    displayName: 'Scrape text'
    pool:
      vmImage: 'ubuntu-latest'
    dependsOn: InstallPackages
    steps:
    - task: UsePythonVersion@0
      inputs:
        versionSpec: '3.9'
        addToPath: true
    - script: |
        python -m pip install --upgrade pip
        python -m pip install pandas
      displayName: 'Install dependencies again'
    - task: PythonScript@0
      displayName: 'Run scraper script'
      inputs:
        scriptSource: 'filePath'
        scriptPath: 'path/to/text_scrapper.py'

内容的提问来源于stack exchange，提问作者Timothy Liew