pip安装含嵌套git+https私有仓库依赖失败的解决咨询

问题 #

当requirements.txt通过git+https引入私有仓库包，且该包的requirements.txt又通过git+https引入另一个私有仓库包时，第二个git+https依赖无法被识别。在Client项目执行pip3 install -r requirements.txt时，报错：Expected end or semicolon (after name and no valid version specifier)。

完整错误信息 #

Resolved https://x-token-auth:****@bitbucket.org/name/library1.git
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error
  
  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [66 lines of output]
      /private/var/folders/r3/g2zzxbp952qcpf7z4tlzrsvc0000gn/T/pip-build-env-yejy0smn/overlay/lib/python3.11/site-packages/setuptools/config/pyprojecttoml.py:66: _BetaConfiguration: Support for `[tool.setuptools]` in `pyproject.toml` is still *beta*.
        config = read_configuration(filepath, True, ignore_option_errors, dist)
      /private/var/folders/r3/g2zzxbp952qcpf7z4tlzrsvc0000gn/T/pip-build-env-yejy0smn/overlay/lib/python3.11/site-packages/setuptools/config/expand.py:132: SetuptoolsWarning: File '/private/var/folders/r3/g2zzxbp952qcpf7z4tlzrsvc0000gn/T/pip-req-build-2kfhkxk4/README.md' cannot be found
        return '\n'.join(
      Traceback (most recent call last):
        File "/private/var/folders/r3/g2zzxbp952qcpf7z4tlzrsvc0000gn/T/pip-build-env-yejy0smn/overlay/lib/python3.11/site-packages/setuptools/_vendor/packaging/requirements.py", line 35, in __init__
          parsed = parse_requirement(requirement_string)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/private/var/folders/r3/g2zzxbp952qcpf7z4tlzrsvc0000gn/T/pip-build-env-yejy0smn/overlay/lib/python3.11/site-packages/setuptools/_vendor/packaging/_parser.py", line 64, in parse_requirement
          return _parse_requirement(Tokenizer(source, rules=DEFAULT_RULES))
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/private/var/folders/r3/g2zzxbp952qcpf7z4tlzrsvc0000gn/T/pip-build-env-yejy0smn/overlay/lib/python3.11/site-packages/setuptools/_vendor/packaging/_parser.py", line 82, in _parse_requirement
          url, specifier, marker = _parse_requirement_details(tokenizer)
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/private/var/folders/r3/g2zzxbp952qcpf7z4tlzrsvc0000gn/T/pip-build-env-yejy0smn/overlay/lib/python3.11/site-packages/setuptools/_vendor/packaging/_parser.py", line 126, in _parse_requirement_details
          marker = _parse_requirement_marker(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/private/var/folders/r3/g2zzxbp952qcpf7z4tlzrsvc0000gn/T/pip-build-env-yejy0smn/overlay/lib/python3.11/site-packages/setuptools/_vendor/packaging/_parser.py", line 147, in _parse_requirement_marker
          tokenizer.raise_syntax_error(
        File "/private/var/folders/r3/g2zzxbp952qcpf7z4tlzrsvc0000gn/T/pip-build-env-yejy0smn/overlay/lib/python3.11/site-packages/setuptools/_vendor/packaging/_tokenizer.py", line 163, in raise_syntax_error
          raise ParserSyntaxError(
      setuptools.extern.packaging._tokenizer.ParserSyntaxError: Expected end or semicolon (after name and no valid version specifier)
          git+https://x-token-auth:****@bitbucket.org/name/library2.git
             ^
      
      The above exception was the direct cause of the following exception:
      
      Traceback (most recent call last):
        File "/Users/username/src/testnestedreq/client/venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/Users/username/src/testnestedreq/client/venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/Users/username/src/testnestedreq/client/venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
          return hook(config_settings)
                 ^^^^^^^^^^^^^^^^^^^^^
        File "/private/var/folders/r3/g2zzxbp952qcpf7z4tlzrsvc0000gn/T/pip-build-env-yejy0smn/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 341, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=['wheel'])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/private/var/folders/r3/g2zzxbp952qcpf7z4tlzrsvc0000gn/T/pip-build-env-yejy0smn/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 323, in _get_build_requires
          self.run_setup()
        File "/private/var/folders/r3/g2zzxbp952qcpf7z4tlzrsvc0000gn/T/pip-build-env-yejy0smn/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 338, in run_setup
          exec(code, locals())
        File "<string>", line 4, in <module>
        File "/private/var/folders/r3/g2zzxbp952qcpf7z4tlzrsvc0000gn/T/pip-build-env-yejy0smn/overlay/lib/python3.11/site-packages/setuptools/__init__.py", line 107, in setup
          return distutils.core.setup(**attrs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/private/var/folders/r3/g2zzxbp952qcpf7z4tlzrsvc0000gn/T/pip-build-env-yejy0smn/overlay/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 159, in setup
          dist.parse_config_files()
        File "/private/var/folders/r3/g2zzxbp952qcpf7z4tlzrsvc0000gn/T/pip-build-env-yejy0smn/overlay/lib/python3.11/site-packages/setuptools/dist.py", line 908, in parse_config_files
          pyprojecttoml.apply_configuration(self, filename, ignore_option_errors)
        File "/private/var/folders/r3/g2zzxbp952qcpf7z4tlzrsvc0000gn/T/pip-build-env-yejy0smn/overlay/lib/python3.11/site-packages/setuptools/config/pyprojecttoml.py", line 67, in apply_configuration
          return _apply(dist, config, filepath)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/private/var/folders/r3/g2zzxbp952qcpf7z4tlzrsvc0000gn/T/pip-build-env-yejy0smn/overlay/lib/python3.11/site-packages/setuptools/config/_apply_pyprojecttoml.py", line 49, in apply
          dist._finalize_requires()
        File "/private/var/folders/r3/g2zzxbp952qcpf7z4tlzrsvc0000gn/T/pip-build-env-yejy0smn/overlay/lib/python3.11/site-packages/setuptools/dist.py", line 607, in _finalize_requires
          self._move_install_requirements_markers()
        File "/private/var/folders/r3/g2zzxbp952qcpf7z4tlzrsvc0000gn/T/pip-build-env-yejy0smn/overlay/lib/python3.11/site-packages/setuptools/dist.py", line 647, in _move_install_requirements_markers
          inst_reqs = list(_reqs.parse(spec_inst_reqs))
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/private/var/folders/r3/g2zzxbp952qcpf7z4tlzrsvc0000gn/T/pip-build-env-yejy0smn/overlay/lib/python3.11/site-packages/setuptools/_vendor/packaging/requirements.py", line 37, in __init__
          raise InvalidRequirement(str(e)) from e
      setuptools.extern.packaging.requirements.InvalidRequirement: Expected end or semicolon (after name and no valid version specifier)
          git+https://x-token-auth:****@bitbucket.org/name/library2.git
             ^
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

项目结构 #

• Client：依赖Library1
• Library1：依赖Library2
• Library2：业务实现项目

各项目requirements.txt内容 #

• Client的requirements.txt：

python-dotenv==1.0.0
git+https://x-token-auth:****@bitbucket.org/name/library1.git@main

• Library1的requirements.txt：

python-dotenv==1.0.0
git+https://x-token-auth:****@bitbucket.org/name/library2.git@main

• Library2的requirements.txt：

python-dotenv==1.0.0

环境信息 #

• Mac Ventura 13.2，M1芯片
• Python 3.11.1
• pip 23.1.2
• setuptools 65.5.0
• 使用python3 -m venv venv创建并激活虚拟环境

已尝试方案 #

• 在Library1项目中执行pip3 install -r requirements.txt可正常完成
• 移除Library1中对Library2的依赖后，Client安装可成功，但无法使用Library2
• 将Library2直接添加到Client的requirements.txt中安装正常，但需Client知晓Library1的依赖，不够合理

解决方案 #

问题核心是：pip安装git+https格式的包时，会将其作为源码包处理，此时setuptools负责解析依赖，但setuptools的依赖解析规则与pip不一致，无法识别git+https格式的依赖声明。以下是三种可行解决方法：

方法1：给Library1添加标准包配置文件（推荐） #

在Library1根目录创建setup.py和pyproject.toml，用PEP 508标准格式声明依赖，让pip能正确识别嵌套的git私有仓库依赖：

setup.py示例：

from setuptools import setup, find_packages

setup(
    name="library1",
    version="0.1.0",
    packages=find_packages(),
    install_requires=[
        "python-dotenv==1.0.0",
        # PEP 508格式声明git私有仓库依赖
        "library2 @ git+https://x-token-auth:****@bitbucket.org/name/library2.git@main"
    ]
)

pyproject.toml示例：

[build-system]
requires = ["setuptools>=61.0"]
build-backend = "setuptools.build_meta"

配置完成后，Client执行pip3 install -r requirements.txt时，会自动通过Library1的setup.py解析并安装Library2。

方法2：使用pip的--process-dependency-links参数（兼容旧场景） #

在Client项目执行安装命令时添加该参数，强制pip处理源码包中的依赖链接：

pip3 install -r requirements.txt --process-dependency-links

注意：该参数在pip 19.0+中已标记为废弃，但部分场景仍可生效。

方法3：本地可编辑模式安装（临时调试用） #

先将Library1克隆到本地，以可编辑模式安装，让pip直接读取Library1的requirements.txt并正确解析：

# 克隆Library1到本地
git clone https://x-token-auth:****@bitbucket.org/name/library1.git
cd library1
# 可编辑模式安装Library1
pip install -e .
# 回到Client目录执行安装
cd ../client
pip install -r requirements.txt

内容的提问来源于stack exchange，提问作者warghs