Scrapy CrawlSpider仅爬取首个域名，多域名爬取异常求助

Hey there! I’ve run into this exact issue before with Scrapy’s CrawlSpider—let’s walk through the most likely culprits and get your multi-domain crawl back on track.

1. You’re Missing Domains in allowed_domains #

This is the most common root cause. If your allowed_domains list only includes the first domain, Scrapy will automatically filter out all requests to other domains. Double-check that you’ve added every target domain here, and avoid including http:// or trailing slashes:

allowed_domains = ["domain1.com", "domain2.org", "domain3.net"]

Pro tip: If you need to include subdomains, use wildcards like *.domain1.com—just be careful not to over-broaden your crawl scope unnecessarily.

2. Your Rules Are Restricting to a Single Domain #

If you added allow_domains to your LinkExtractor in the Rule, you might’ve only specified the first domain. For example:

# ❌ This will only follow links from domain1.com
Rule(LinkExtractor(allow=r'.*', allow_domains="domain1.com"), callback='parse_item', follow=True)

Fix this by either removing the allow_domains parameter entirely (to crawl all domains in your allowed_domains list) or adding all target domains to the parameter:

# ✅ Allows links from any domain in allowed_domains
Rule(LinkExtractor(allow=r'.*'), callback='parse_item', follow=True)

# OR ✅ Explicitly list domains for tighter control
Rule(LinkExtractor(allow=r'.*', allow_domains=["domain1.com", "domain2.org"]), callback='parse_item', follow=True)

3. Robots.txt Is Blocking Other Domains #

Scrapy obeys robots.txt rules by default. If some target domains disallow crawlers in their robots.txt, Scrapy will skip them entirely. You can temporarily disable this to test (just remember to respect robots rules for production crawls):

# In settings.py
ROBOTSTXT_OBEY = False

4. Scheduler or Dupe Filter Misconfiguration #

Rarely, the scheduler might prioritize the first domain’s requests over others, or the duplicate filter might incorrectly flag other domain URLs. To debug this:

• Enable debug logging to see why requests are being filtered:

  # In settings.py
  LOG_LEVEL = 'DEBUG'
  

  Look for lines like Filtered offsite request to 'domain2.org'—this will tell you exactly why requests are being skipped.
• Stick to the default scrapy.dupefilters.RFPDupeFilter unless you have a custom dupe filter that might interfere with multi-domain URLs.

Example Working Multi-Domain CrawlSpider #

Here’s a stripped-down version of what your spider should look like to handle multiple domains and track URL counts per domain:

from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor

class MultiDomainURLCountSpider(CrawlSpider):
    name = 'multi_domain_url_counter'
    allowed_domains = ["domain1.com", "domain2.org", "domain3.net"]
    start_urls = [
        "https://domain1.com",
        "https://domain2.org",
        "https://domain3.net"
    ]

    rules = (
        Rule(LinkExtractor(), callback='track_url_count', follow=True),
    )

    def track_url_count(self, response):
        # Extract the domain from the response URL
        domain = response.url.split('/')[2]
        # Increment the count for this domain using Scrapy's built-in stats
        self.crawler.stats.inc_value(f'domain_urls/{domain}')
        # Optional: Print progress to the console
        self.logger.info(f"Counted URL for {domain}: {self.crawler.stats.get_value(f'domain_urls/{domain}')}")

After applying these fixes, run your spider again and monitor the logs—you should see requests going out to all your target domains instead of just the first one.

内容的提问来源于stack exchange，提问作者Milano