Scrapy内置CrawlOnce中间件重启任务时触发struct.error: unpack requires a buffer of 4 bytes错误的求助

大家好，我在使用Scrapy内置的CrawlOnce中间件时碰到了一个棘手的问题：当我停止爬虫任务后再重启，偶尔会抛出如下错误：

Traceback (most recent call last):
  File "/app/tasks/crawl.py", line 250, in crawl
    crawler_proc.start()
  File "/usr/local/lib/python3.9/site-packages/scrapy/crawler.py", line 346, in start
    reactor.run(installSignalHandlers=False)  # blocking call
  File "/usr/local/lib/python3.9/site-packages/twisted/internet/base.py", line 1318, in run
    self.mainLoop()
  File "/usr/local/lib/python3.9/site-packages/twisted/internet/base.py", line 1328, in mainLoop
    reactorBaseSelf.runUntilCurrent()
--- <exception caught here> ---
  File "/usr/local/lib/python3.9/site-packages/twisted/internet/base.py", line 994, in runUntilCurrent
    call.func(*call.args, **call.kw)
  File "/usr/local/lib/python3.9/site-packages/scrapy/utils/reactor.py", line 51, in __call__
    return self._func(*self._a, **self._kw)
  File "/usr/local/lib/python3.9/site-packages/scrapy/core/engine.py", line 157, in _next_request
    self.crawl(request)
  File "/usr/local/lib/python3.9/site-packages/scrapy/core/engine.py", line 247, in crawl
    self._schedule_request(request, self.spider)
  File "/usr/local/lib/python3.9/site-packages/scrapy/core/engine.py", line 252, in _schedule_request
    if not self.slot.scheduler.enqueue_request(request):  # type: ignore[union-attr]
  File "/usr/local/lib/python3.9/site-packages/scrapy/core/scheduler.py", line 241, in enqueue_request
    dqok = self._dqpush(request)
  File "/usr/local/lib/python3.9/site-packages/scrapy/core/scheduler.py", line 280, in _dqpush
    self.dqs.push(request)
  File "/usr/local/lib/python3.9/site-packages/scrapy/pqueues.py", line 89, in push
    self.queues[priority] = self.qfactory(priority)
  File "/usr/local/lib/python3.9/site-packages/scrapy/pqueues.py", line 76, in qfactory
    return create_instance(
  File "/usr/local/lib/python3.9/site-packages/scrapy/utils/misc.py", line 166, in create_instance
    instance = objcls.from_crawler(crawler, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/scrapy/squeues.py", line 68, in from_crawler
    return cls(crawler, key)
  File "/usr/local/lib/python3.9/site-packages/scrapy/squeues.py", line 64, in __init__
    super().__init__(key)
  File "/usr/local/lib/python3.9/site-packages/scrapy/squeues.py", line 23, in __init__
    super().__init__(path, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/queuelib/queue.py", line 208, in __init__
    (self.size,) = struct.unpack(self.SIZE_FORMAT, qsize)
struct.error: unpack requires a buffer of 4 bytes

我试过删除请求队列来解决这个错误，但这又带来了新的问题：如果爬虫流程里存在分页逻辑，当我在某个中间分页URL处停止爬虫，删除队列后会丢失分页状态。因为我的爬虫只从输入文件里提供的基础URL开始启动，那些动态生成的后续分页请求就完全找不到了。

有没有朋友遇到过类似的问题，或者有什么办法能既解决这个struct.error的问题，又能保留住分页的状态呢？麻烦大家支支招！

备注：内容来源于stack exchange，提问作者ProblemSolver