将Scrapy爬虫部署到Zyte时input.csv缺失引发报错求助

本地运行Scrapy爬虫时可正常读取input.csv文件，但通过shub deploy部署到Zyte（原Scrapinghub）后，构建包未包含该文件，运行时抛出文件找不到的错误。

报错信息 #

Traceback (most recent call last):
  File "<frozen zipimport>", line 177, in get_data
KeyError: 'webscrap/resources/input.csv'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/scrapy/core/engine.py", line 127, in _next_request
    request = next(slot.start_requests)
  File "/app/__main__.egg/webscrap/spiders/website_scraper.py", line 13, in start_requests
    zipcodes_csv = pkgutil.get_data("webscrap", "resources/input.csv")
  File "/usr/local/lib/python3.8/pkgutil.py", line 637, in get_data
    return loader.get_data(resource_name)
  File "<frozen zipimport>", line 179, in get_data
OSError: [Errno 0] : 'webscrap/resources/input.csv'

读取文件的代码 #

zipcodes_csv = pkgutil.get_data("webscrap", "resources/input.csv")
with io.TextIOWrapper(io.BytesIO(zipcodes_csv), encoding='utf-8') as file:
    csvreader = csv.DictReader(file)

当前setup.py内容 #

setup(
    name         = 'project',
    version      = '1.0',
    packages     = find_packages(),
    entry_points = {'scrapy': ['settings = webscrap.settings']},
    package_data={
        'project': ['resources/*.csv']
    },
    include_package_data=True,
)

项目目录结构 #

• 根目录下含webscrap文件夹（Scrapy项目主目录）
• webscrap内有resources文件夹，input.csv存放在该文件夹中
• 根目录下包含setup.py等文件

问题出在setup.py的package_data配置错误，资源文件属于webscrap包而非project，需按以下步骤修正：

1. 修改setup.py中的package_data配置：

setup(
    name         = 'project',
    version      = '1.0',
    packages     = find_packages(),
    entry_points = {'scrapy': ['settings = webscrap.settings']},
    package_data={
        # 将包名从'project'改为实际的'webscrap'
        'webscrap': ['resources/*.csv']
    },
    include_package_data=True,
)

2. 在resources文件夹内添加空的__init__.py文件，让Python将其识别为webscrap包的子模块，确保资源文件能被正确打包。

3. 重新执行shub deploy部署，此时input.csv会被包含在构建包中，服务器运行即可正常读取文件。

内容的提问来源于stack exchange，提问作者Muhammad Ahmad