Scrapy项目打包为exe后运行报错，求原因及解决方法

问题 #

我开发的Scrapy项目在本地运行正常，但用PyInstaller打包成main.exe后，运行时控制台只显示Traceback开头信息就报错。以下是项目的main.py和Spider类代码，求错误原因及解决办法。

main.py代码 #

from rascraper.spiders.spiderone import PostsSpider
from scrapy.crawler import CrawlerProcess
from scrapy.utils.project import get_project_settings


def main():
    process = CrawlerProcess(get_project_settings())
    process.crawl(PostsSpider)
    process.start()


if __name__ == '__main__':
    main()

Spider类代码（rascraper/spiders/spiderone.py） #

import scrapy


class PostsSpider(scrapy.Spider):
    name = 'posts'

    # artist = input(f'Artist Name:')
    # filter = input(f'filter on Country? (y/n):')
    #
    # if filter == 'y':
    #     country = input(f'Country:')
    #     start_urls = [
    #         f'https://ra.co/dj/{artist}/past-events?country={country}'
    #     ]
    #
    # elif filter == 'n':
    #     start_urls = [
    #         f'https://ra.co/dj/{artist}/past-events'
    #     ]

    HEADERS = {
        'accept': '/*',
        'accept-encoding': 'gzip, deflate, br',
        'accept-language': 'nl-NL,nl;q=0.9,en-US;q=0.8,en;q=0.7,fr;q=0.6',
        'authorization': 'df67dacc9c704696b908a618dd4f59be',
        'cache-control': 'max-age=0',
        'content-type': 'application/json',
        'origin': 'https://ra.co',
        'referer': 'https://ra.co/',
        'sec-ch-ua': '"Not_A Brand";v="99", "Google Chrome";v="109", "Chromium";v="109"',
        'sec-ch-ua-mobile': '?0',
        'sec-ch-ua-platform': 'Windows',
        'sec-fetch-dest': 'empty',
        'sec-fetch-mode': 'cors',
        'sec-fetch-site': 'same-site',
    }


    def parse(self, response):
        for post in response.css('li.Column-sc-18hsrnn-0.inVJeD'):
            date = post.css('.Text-sc-1t0gn2o-0.jmZufm::text').get()
            event = post.css('.Text-sc-1t0gn2o-0.Link__StyledLink-k7o46r-0.dXQVFW::text').get()
            location = post.css('.Text-sc-1t0gn2o-0.Link__StyledLink-k7o46r-0.echVma::text').get()
            venue = post.css('.Text-sc-1t0gn2o-0.Link__StyledLink-k7o46r-0.dxNiKF::text').get()
            acts = post.css('.Text-sc-1t0gn2o-0.bYvpkM::text').get()

            item = {}
            item['Date'] = date
            item['Event'] = event
            item['Location'] = location
            item['Venue'] = venue
            item['Acts'] = acts

            yield item

错误原因分析 #

1. Scrapy配置文件缺失：PyInstaller打包时不会自动包含Scrapy项目的settings.py、items.py等核心配置文件，get_project_settings()无法读取到必要配置，导致CrawlerProcess初始化失败。
2. 动态依赖未被捕获：Scrapy内部存在大量动态导入的模块（如中间件、管道），PyInstaller的自动检测机制无法完全识别这些隐藏依赖。
3. 控制台输出截断：打包后的exe运行时，错误信息可能因输出缓冲被截断，仅显示Traceback开头，真实错误被隐藏。

解决步骤 #

步骤1：手动加载Scrapy配置 #

修改main.py，直接导入项目settings模块，避免依赖自动查找逻辑：

import sys
import io

# 强制刷新输出，避免错误信息截断
sys.stdout = io.TextIOWrapper(sys.stdout.buffer, encoding='utf-8', line_buffering=True)
sys.stderr = io.TextIOWrapper(sys.stderr.buffer, encoding='utf-8', line_buffering=True)

from rascraper.spiders.spiderone import PostsSpider
from scrapy.crawler import CrawlerProcess
from scrapy.settings import Settings
import rascraper.settings  # 导入项目settings


def main():
    # 手动初始化配置
    settings = Settings()
    settings.setmodule(rascraper.settings, priority='project')
    process = CrawlerProcess(settings)
    process.crawl(PostsSpider)
    process.start()


if __name__ == '__main__':
    main()

步骤2：用Spec文件配置打包参数 #

1. 生成初始spec文件：

pyi-makespec --name main main.py

2. 编辑main.spec，补充依赖和资源路径：

import os
from scrapy.utils.misc import walk_modules

# 获取Scrapy所有隐藏依赖
def get_scrapy_hidden_imports():
    imports = []
    for module in walk_modules('scrapy'):
        imports.append(module)
    return imports

# 项目核心文件路径
project_root = os.path.dirname(os.path.abspath('rascraper'))
datas = [
    (os.path.join(project_root, 'rascraper/settings.py'), 'rascraper'),
    (os.path.join(project_root, 'rascraper/items.py'), 'rascraper'),
    (os.path.join(project_root, 'rascraper/pipelines.py'), 'rascraper'),
    (os.path.join(project_root, 'rascraper/middlewares.py'), 'rascraper'),
]

a = Analysis(
    ['main.py'],
    pathex=[],
    binaries=[],
    datas=datas,
    hiddenimports=get_scrapy_hidden_imports() + ['rascraper.spiders.spiderone', 'rascraper.settings'],
    hookspath=[],
    hooksconfig={},
    runtime_hooks=[],
    excludes=[],
    win_no_prefer_redirects=False,
    win_private_assemblies=False,
    cipher=None,
    noarchive=False,
)
pyz = PYZ(a.pure, a.zipped_data, cipher=None)

exe = EXE(
    pyz,
    a.scripts,
    a.binaries,
    a.zipfiles,
    a.datas,
    [],
    name='main',
    debug=False,
    bootloader_ignore_signals=False,
    strip=False,
    upx=True,
    upx_exclude=[],
    runtime_tmpdir=None,
    console=True,  # 保持控制台窗口打开，查看完整错误
    disable_windowed_traceback=False,
    target_arch=None,
    codesign_identity=None,
    entitlements_file=None,
)

步骤3：使用Spec文件打包 #

pyinstaller main.spec

额外修复点 #

• 取消Spider中start_urls的注释，或在main()函数中动态传递起始URL（注意：不要在类定义阶段使用input()，会导致打包过程卡住，建议把输入逻辑移到main()中）。

内容的提问来源于stack exchange，提问作者AxelotlZ