含Git仓库依赖的Scrapy爬虫部署时Egg化失败求助

我有一个名为the_application的Scrapy项目，依赖Git仓库中的the_library库。执行scrapyd-deploy --include-dependencies命令部署时，出现如下报错：

'build/lib' does not exist -- can't clean it
'build/bdist.linux-x86_64' does not exist -- can't clean it
'build/scripts-3.10' does not exist -- can't clean it
  Running command git clone --filter=blob:none --quiet 'ssh://****@github.com/hrafnthor/my-library.git' /tmp/pip-install-no2qcx7z/my_library_19b012bf7ea747e9b2efaf39df47abac
  WARNING: Generating metadata for package my_library produced metadata for project name unknown. Fix your #egg=my_library fragments.
ERROR: Could not find a version that satisfies the requirement my_library (unavailable) (from versions: none)
ERROR: No matching distribution found for my_library (unavailable)
Traceback (most recent call last):
  File "/home/hrafn/Documents/dev/python/the_application/scraper/setup.py", line 5, in <module>
    setup(
  File "/home/hrafn/.local/lib/python3.10/site-packages/setuptools/__init__.py", line 108, in setup
    return distutils.core.setup(**attrs)
  File "/home/hrafn/.local/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 185, in setup
    return run_commands(dist)
  File "/home/hrafn/.local/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
    dist.run_commands()
  File "/home/hrafn/.local/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
    self.run_command(cmd)
  File "/home/hrafn/.local/lib/python3.10/site-packages/setuptools/dist.py", line 1221, in run_command
    super().run_command(command)
  File "/home/hrafn/.local/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
    cmd_obj.run()
  File "/home/hrafn/.local/lib/python3.10/site-packages/uberegg.py", line 35, in run
    self._install(self.requirements, self.bdist_dir)
  File "/home/hrafn/.local/lib/python3.10/site-packages/uberegg.py", line 45, in _install
    return subprocess.check_call(
  File "/usr/lib/python3.10/subprocess.py", line 369, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/usr/bin/python3', '-m', 'pip', 'install', '-U', '-t', 'build/bdist.linux-x86_64/egg', '-r', 'requirements.txt']' returned non-zero exit status 1.

从栈追踪信息判断，问题出在the_library的依赖定义上，但我不确定具体解决方法。

the_library的pyproject.toml内容： #

[project]
name = "the_library"
version = "0.0.1"
description = ""
readme = "README.md"
requires-python = ">=3.7"
dependencies = []

[tool.setuptools]
package-dir = {"" = "src"}

[tool.setuptools.packages.find]
where = ["src"]

[build-system]
requires = ["setuptools", "wheel"]
build-backend = "setuptools.build_meta"

the_application的依赖配置： #

项目是通过scrapy startproject the_application生成的标准Scrapy项目。

requirements.txt相关内容（由pip-compile requirements.in生成）：

# omitting other dependencies

the_library @ git+ssh://git@github.com/hrafnthor/the_library.git@main
    # via -r requirements.in

# omitting other dependencies

requirements.in内容：

git+ssh://git@github.com/hrafnthor/the_library.git@main
scrapy

我曾尝试在requirements.in的依赖定义后追加#egg=the_library，但未解决问题，报错始终指向Egg化时缺少版本信息。

更新1：
我研究了uberegg库的作用，它是scrapyd的依赖，看起来只是封装了setuptools并添加了文件遍历和日志功能。问题出在uberegg.py第45行调用subprocess.check_call()时抛出的错误。

奇怪的是，直接执行栈追踪中提到的命令python3 -m pip install -U -t build/bdist.linux-x86_64/egg -r requirements.txt时并未出现问题。所以问题可能与scrapyd仍在使用已废弃多年的egg格式有关？

解决方案 #

1. 修正Git依赖的格式
   确保requirements.in中的Git依赖格式完全正确，需将#egg=the_library直接追加到Git地址后：

   git+ssh://git@github.com/hrafnthor/the_library.git@main#egg=the_library
   scrapy
   

   执行pip-compile requirements.in重新生成requirements.txt后再尝试部署。

2. 排查环境差异
   直接执行pip命令正常但scrapyd调用失败，可能是环境变量或权限问题：

   • 确认scrapyd使用的Python环境与本地测试的版本一致
   • 检查scrapyd进程是否能访问Git仓库的SSH密钥（可能进程未加载用户密钥）

3. 绕过egg打包流程
   scrapyd的--include-dependencies依赖uberegg处理egg格式，而egg已被淘汰，可改用以下方式部署：

   • 本地将所有依赖安装到项目目录：pip install -r requirements.txt -t ./vendor
   • 修改Scrapy项目的settings.py，添加PYTHONPATH = ['./vendor']
   • 直接执行scrapyd-deploy部署（去掉--include-dependencies参数）

4. 验证the_library的打包正确性
   在the_library仓库根目录执行python -m build，检查是否能生成正常的wheel包，确认打包过程无隐藏错误。

内容的提问来源于stack exchange，提问作者Hrafn