使用scrapyd-deploy部署Scrapy时触发ValueError问题求助

Let's break down what's happening here and how to fix it quickly.

The Root Cause #

When you deploy a Scrapy project with scrapyd-deploy, Scrapyd packages your entire project into a .egg file (which is essentially a ZIP archive). Your logging configuration is trying to write debug.log inside this egg file using BASE_DIR—but egg archives are read-only, and you can't write files directly into them. That's exactly why you're getting the NotADirectoryError: the path /tmp/first_scrapy-1526919848-13w8cqlm.egg/debug.log points to a location inside the compressed egg, not a real directory on the filesystem.

Local runs work fine because BASE_DIR points to your actual project folder on your hard drive, which is a writable directory—no compression involved.

The Fix: Move the Log File to a Writable External Path #

You need to update your LOGGING configuration to point to a directory that's outside the egg archive and writable by the Scrapyd process. Here are two reliable options:

Option 1: Use the System Temporary Directory

This is great for quick debugging, since temporary directories are always writable (just note that logs might be cleaned up automatically by your OS):

First, add an import for tempfile at the top of your settings.py, then update the filename in your logging handler:

import os
import tempfile
import django

BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
django.setup()

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'handlers': {
        'file': {
            'level': 'WARNING',
            'class': 'logging.FileHandler',
            # Write logs to the system's temp directory instead of the egg
            'filename': os.path.join(tempfile.gettempdir(), 'first_scrapy_debug.log'),
        },
        'mail_admins': {
            'level': 'ERROR',
            'class': 'django.utils.log.AdminEmailHandler'
        },
    },
    'loggers': {
        'django': {
            'handlers': ['file'],
            'level': 'WARNING',
            'propagate': True,
        },
    },
}

Option 2: Use a Custom Persistent Log Directory

For production, you'll want logs to stick around. Create a dedicated directory for your Scrapy project logs (make sure the user running Scrapyd has write permissions to it), then update your config:

import os
import django

BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
django.setup()

# Define a persistent log path (create this directory first!)
LOG_DIR = '/var/log/scrapyd/first_scrapy'
# Ensure the directory exists (no error if it's already there)
os.makedirs(LOG_DIR, exist_ok=True)

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'handlers': {
        'file': {
            'level': 'WARNING',
            'class': 'logging.FileHandler',
            'filename': os.path.join(LOG_DIR, 'debug.log'),
        },
        'mail_admins': {
            'level': 'ERROR',
            'class': 'django.utils.log.AdminEmailHandler'
        },
    },
    'loggers': {
        'django': {
            'handlers': ['file'],
            'level': 'WARNING',
            'propagate': True,
        },
    },
}

Key Reminder #

Whichever option you choose, double-check that the Scrapyd process has write permissions to the target directory. If you get permission errors after fixing the path, you may need to adjust directory ownership or permissions with commands like chown or chmod.

内容的提问来源于stack exchange，提问作者Chiefir