Azure DevOps Pipeline中PySpark卸载卡住问题求助

Azure DevOps测试流水线中，执行pip uninstall pyspark时始终无法完成，耗时超4小时，重启任务后问题依旧。系统已识别到PySpark 3.3.1的安装，但卸载流程卡在文件列表展示阶段。

流水线配置 #

trigger: none

jobs:
- job: 'QA_Pipeline'
  timeoutInMinutes: 300
  pool: vmss-deep-dev-pool-002
  
  steps:
  - task: UsePythonVersion@0
    inputs:
      versionSpec: '3.8'

  - script:  |
       pip install --upgrade  --force-reinstall pip setuptools wheel
       pip uninstall pyarrow
       pip uninstall pyspark
       pip install -I azure-cli==2.18.0 
       pip install -I databricks-connect==9.1.30
    displayName: 'Install modules'
  ...
  ...

相关日志 #

Collecting pip
  Using cached pip-23.0-py3-none-any.whl (2.1 MB)
Collecting setuptools
  Using cached setuptools-67.0.0-py3-none-any.whl (1.1 MB)
Collecting wheel
  Using cached wheel-0.38.4-py3-none-any.whl (36 kB)
Installing collected packages: wheel, setuptools, pip
  Attempting uninstall: wheel
    Found existing installation: wheel 0.38.4
    Uninstalling wheel-0.38.4:
      Successfully uninstalled wheel-0.38.4
  Attempting uninstall: setuptools
    Found existing installation: setuptools 67.0.0
    Uninstalling setuptools-67.0.0:
      Successfully uninstalled setuptools-67.0.0
  Attempting uninstall: pip
    Found existing installation: pip 23.0
    Uninstalling pip-23.0:
      Successfully uninstalled pip-23.0
Successfully installed pip-23.0 setuptools-67.0.0 wheel-0.38.4
WARNING: Skipping pyarrow as it is not installed.
Found existing installation: pyspark 3.3.1
Uninstalling pyspark-3.3.1:
Would remove:
    /opt/hostedtoolcache/Python/3.8.16/x64/bin/beeline
    /opt/hostedtoolcache/Python/3.8.16/x64/bin/beeline.cmd
    /opt/hostedtoolcache/Python/3.8.16/x64/bin/docker-image-tool.sh
    /opt/hostedtoolcache/Python/3.8.16/x64/bin/find-spark-home
    /opt/hostedtoolcache/Python/3.8.16/x64/bin/find-spark-home.cmd
    /opt/hostedtoolcache/Python/3.8.16/x64/bin/find_spark_home.py
    /opt/hostedtoolcache/Python/3.8.16/x64/bin/load-spark-env.cmd
    /opt/hostedtoolcache/Python/3.8.16/x64/bin/load-spark-env.sh
    /opt/hostedtoolcache/Python/3.8.16/x64/bin/pyspark
    /opt/hostedtoolcache/Python/3.8.16/x64/bin/pyspark.cmd
    /opt/hostedtoolcache/Python/3.8.16/x64/bin/pyspark2.cmd
    /opt/hostedtoolcache/Python/3.8.16/x64/bin/run-example
    /opt/hostedtoolcache/Python/3.8.16/x64/bin/run-example.cmd
    /opt/hostedtoolcache/Python/3.8.16/x64/bin/spark-class
    /opt/hostedtoolcache/Python/3.8.16/x64/bin/spark-class.cmd
    /opt/hostedtoolcache/Python/3.8.16/x64/bin/spark-class2.cmd
    /opt/hostedtoolcache/Python/3.8.16/x64/bin/spark-shell
    /opt/hostedtoolcache/Python/3.8.16/x64/bin/spark-shell.cmd
    /opt/hostedtoolcache/Python/3.8.16/x64/bin/spark-shell2.cmd
    /opt/hostedtoolcache/Python/3.8.16/x64/bin/spark-sql
    /opt/hostedtoolcache/Python/3.8.16/x64/bin/spark-sql.cmd
    /opt/hostedtoolcache/Python/3.8.16/x64/bin/spark-sql2.cmd
    /opt/hostedtoolcache/Python/3.8.16/x64/bin/spark-submit
    /opt/hostedtoolcache/Python/3.8.16/x64/bin/spark-submit.cmd
    /opt/hostedtoolcache/Python/3.8.16/x64/bin/spark-submit2.cmd
    /opt/hostedtoolcache/Python/3.8.16/x64/bin/sparkR
    /opt/hostedtoolcache/Python/3.8.16/x64/bin/sparkR.cmd
    /opt/hostedtoolcache/Python/3.8.16/x64/bin/sparkR2.cmd
    /opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/pyspark-3.3.1.dist-info/*
    /opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/pyspark/*

解决建议 #

• 添加非交互式卸载参数：pip uninstall默认等待用户确认，流水线无交互输入会导致命令挂起。修改命令为pip uninstall -y pyspark，自动确认卸载操作。
• 手动删除PySpark文件：若添加-y后仍卡住，直接删除日志中列出的PySpark相关文件和目录，在脚本中加入：

  rm -rf /opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/pyspark
  rm -rf /opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/pyspark-3.3.1.dist-info
  rm -f /opt/hostedtoolcache/Python/3.8.16/x64/bin/beeline* /opt/hostedtoolcache/Python/3.8.16/x64/bin/docker-image-tool.sh /opt/hostedtoolcache/Python/3.8.16/x64/bin/find-spark-home* /opt/hostedtoolcache/Python/3.8.16/x64/bin/find_spark_home.py /opt/hostedtoolcache/Python/3.8.16/x64/bin/load-spark-env* /opt/hostedtoolcache/Python/3.8.16/x64/bin/pyspark* /opt/hostedtoolcache/Python/3.8.16/x64/bin/run-example* /opt/hostedtoolcache/Python/3.8.16/x64/bin/spark-class* /opt/hostedtoolcache/Python/3.8.16/x64/bin/spark-shell* /opt/hostedtoolcache/Python/3.8.16/x64/bin/spark-sql* /opt/hostedtoolcache/Python/3.8.16/x64/bin/spark-submit* /opt/hostedtoolcache/Python/3.8.16/x64/bin/sparkR*
  

• 降级pip版本：当前使用的pip 23.0可能存在兼容性问题，尝试降级到稳定版本后再卸载。修改pip升级命令为pip install --upgrade --force-reinstall pip==22.3.1 setuptools wheel。
• 使用虚拟环境隔离依赖：创建Python虚拟环境，在干净环境中安装所需依赖，无需卸载系统级PySpark。修改脚本步骤为：

  python -m venv .venv
  source .venv/bin/activate
  pip install --upgrade pip setuptools wheel
  pip install -I azure-cli==2.18.0 
  pip install -I databricks-connect==9.1.30
  

内容的提问来源于stack exchange，提问作者Manish