M1 Mac上使用pipenv安装Python依赖遭遇解析失败问题

问题背景 #

在M1芯片Mac上搭建基于Intel架构创建的Python项目，使用pyenv+pipenv管理依赖，执行pipenv install时出现依赖解析失败错误，错误信息如下：

ERROR:pip.subprocessor:[present-rich] pip subprocess to install build dependencies exited with 1
[ResolutionFailure]: File "/Users/xur19/.pyenv/versions/3.8.10/lib/python3.8/site-packages/pipenv/resolver.py", line 782, in _main
[ResolutionFailure]: resolve_packages(
[ResolutionFailure]: File "/Users/xur19/.pyenv/versions/3.8.10/lib/python3.8/site-packages/pipenv/resolver.py", line 730, in resolve_packages
[ResolutionFailure]: results, resolver = resolve(
[ResolutionFailure]: File "/Users/xur19/.pyenv/versions/3.8.10/lib/python3.8/site-packages/pipenv/resolver.py", line 709, in resolve
[ResolutionFailure]: return resolve_deps(
[ResolutionFailure]: File "/Users/xur19/.pyenv/versions/3.8.10/lib/python3.8/site-packages/pipenv/utils/resolver.py", line 1105, in resolve_deps
[ResolutionFailure]: results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
[ResolutionFailure]: File "/Users/xur19/.pyenv/versions/3.8.10/lib/python3.8/site-packages/pipenv/utils/resolver.py", line 894, in actually_resolve_deps
[ResolutionFailure]: resolver.resolve()
[ResolutionFailure]: File "/Users/xur19/.pyenv/versions/3.8.10/lib/python3.8/site-packages/pipenv/utils/resolver.py", line 689, in resolve
[ResolutionFailure]: raise ResolutionFailure(message=str(e))
[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
Hint: try $ pipenv lock --pre if it is a pre-release dependency.
ERROR: pip subprocess to install build dependencies exited with 1

项目已有现成的Pipfile和Pipfile.lock，搭建步骤如下：

brew install pyenv
pyenv install 3.8.7
pyenv global 3.8.7
echo -e 'if command -v pyenv 1>/dev/null 2>&1; then\n  eval "$(pyenv init)"\nfi' >> ~/.zshrc
brew install python
pip install pipenv
pipenv shell
pipenv install

已尝试的方案均失败：

• 适配ARM架构：改用Python 3.8.10，移除cx-oracle，执行pipenv lock --pre、pipenv install --skip-lock，删除Pipfile.lock仅用Pipfile
• 通过Rosetta 2运行：安装x86版本Homebrew，用arch -x86_64前缀执行操作，开启Rosetta终端修改路径

项目Pipfile内容：

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]
autopep8 = "*"
flake8 = "*"
black = "==20.8b1"

[packages]
celery = "==4.4.7"
channels = "==3.0.3"
channels-redis = "==3.2.0"
#cx_Oracle = "==8.2.1" # Django DATABASE
Django = "==2.2.24"
django-allauth = "==0.44.0"
django-celery-beat = "==2.2.0"
django-cors-headers = "==3.7.0"
django-enumfield = "==1.5"
django-environ = "==0.4.5"
django-extensions = "==3.1.2"
django-filter = "==2.1.0"
django-graphql-jwt = "==0.3.1"
django-graphql-persist = "==0.1.1"
django-jet = "==1.0.8"
django-redis = "==4.12.1"
django-reversion = "==3.0.9"
django-reversion-compare = "==0.14.0"
django-measurement = "==3.2.3"
django-jsoneditor = "==0.1.6"
django-storages = {extras = ["google"],version = "==1.11.1"}
flower = "==0.9.7"
future = "==0.18.2"
google-auth = "==1.29.0"
google-auth-oauthlib = "==0.4.4"
google-cloud-storage = "==1.37.1"
google-cloud-pubsub = "==2.4.0"
graphene = "==2.1.7"
graphql-core = "==2.2"
graphene-django = "==2.7"
graphene-file-upload = "==1.3.0"
graphql-relay = "==2"
graphql-server-core = "==1.1.1"
gunicorn = "==19.9.0"
ijson = "==3.1.4"
ldap3 = "==2.9"
matplotlib = "==3.4.1"
munch = "==2.5.0"
numpy = "==1.20.2"
openpyxl = "==3.0.7"
pandas = "==1.2.4"
pillow = "==8.2.0"
psycopg2-binary = "==2.8.6"
pyhumps = "==1.6.1"
pyjwt = "==1.7.1"
pymongo = "==3.11.3"
pyteomics = "==4.4.2"
python-pptx = "==0.6.18"
pytz = "==2018.9"
redis = "==3.2.0" # Doesn't get installed by django-redis
requests = "==2.21.0"
tqdm = "==4.60.0"
xlrd = "==2.0.1"
jinja2 = "==3.0.2"
premailer = "==3.10.0"
docxtpl = "==0.14.2"
inflection = "==0.4.0"
sympy = "==1.9"

[requires]
python_version = "3.7"

解决建议 #

1. 修正Python版本匹配问题 #

Pipfile中指定的python_version = "3.7"，但实际安装的是3.8.x版本，版本不匹配会导致依赖解析混乱：

• 用pyenv安装Python 3.7.16（3.7系列最后一个稳定版，适配M1）：

  pyenv install 3.7.16
  pyenv local 3.7.16 # 仅当前项目使用该版本
  

• 删除原有的Pipfile.lock，重新生成：

  rm Pipfile.lock
  pipenv lock
  

2. 替换不兼容的旧版本依赖 #

部分依赖版本过旧，没有适配ARM架构的预编译包，需要升级/降级到兼容版本：

• psycopg2-binary：2.8.6版本在M1上编译容易失败，升级到2.9.5：psycopg2-binary = "==2.9.5"
• numpy：1.20.2对M1支持有限，升级到1.21.6：numpy = "==1.21.6"
• matplotlib：3.4.1升级到3.5.3：matplotlib = "==3.5.3"
• jinja2：3.0.2和Django 2.2.x存在兼容性问题，降级到2.11.3：jinja2 = "==2.11.3"

3. 强制使用Rosetta 2运行完整环境 #

之前的Rosetta尝试可能没彻底覆盖所有环节，按以下步骤重新配置：

1. 打开终端时右键选择「使用Rosetta打开」
2. 卸载当前pyenv和pipenv：

   pip uninstall -y pipenv
   rm -rf ~/.pyenv
   

3. 安装x86版本的Homebrew（已安装则跳过）：

   arch -x86_64 /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
   echo 'alias brew86="arch -x86_64 /usr/local/bin/brew"' >> ~/.zshrc
   source ~/.zshrc
   

4. 用x86 Homebrew安装pyenv：

   brew86 install pyenv
   

5. 安装x86架构的Python 3.7.16：

   arch -x86_64 pyenv install 3.7.16
   pyenv local 3.7.16
   

6. 重新安装pipenv：

   pip install pipenv
   

7. 重新执行安装：

   pipenv install --skip-lock
   

4. 手动处理编译失败的依赖 #

如果仍有依赖编译失败，安装必要的编译工具：

brew install openssl readline sqlite3 xz zlib

对于特定依赖，比如ldap3，可以指定环境变量：

export CPPFLAGS="-I$(brew --prefix openssl)/include"
export LDFLAGS="-L$(brew --prefix openssl)/lib"
pipenv install

内容的提问来源于stack exchange，提问作者Robert X