如何阻止Scrapy爬虫爬取外部域名？配置后仍异常求排查

What's Going Wrong Here? #

Looking at your logs and code, it’s clear your crawler isn’t behaving as expected—it’s crawling domains like thalia.de even though your allowed_domains is set to ['eaton.com']. The weird part is some off-site requests (like rtbhouse.com) are being filtered, but others aren’t. Here’s why that might be happening:

• Missing explicit filtering in LinkExtractor:
  Right now, your Rule uses an empty LinkExtractor() which pulls all links from pages. While Scrapy’s default OffsiteMiddleware should filter out off-domain requests, relying solely on this can lead to edge cases. Without an explicit allow rule, the crawler still extracts off-site links first, and sometimes this filtering breaks due to custom middleware or unexpected page behavior.

• Custom middleware interference:
  Your DOWNLOADER_MIDDLEWARES includes RandomProxyForReDirectedUrls and FilterResponses. It’s possible one of these custom tools is bypassing Scrapy’s off-site checks—for example, if the proxy middleware handles redirects without respecting allowed_domains, it could let off-site requests slip through.

• Unfiltered manual requests:
  If your parse_item method generates new scrapy.Request objects manually, you need to make sure those are filtered against allowed_domains too. Skipping this check would let off-site requests bypass the OffsiteMiddleware.

Step-by-Step Fixes #

Let’s get your crawler back on track to only crawl eaton.com:

1. Add Explicit allow Rule to LinkExtractor #

Stop off-site links from being extracted in the first place by defining a clear rule for LinkExtractor:

from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule

class WebSpider(CrawlSpider):
    name = 'web'
    allowed_domains = ['eaton.com']
    start_urls= ['https://www.eaton.com/us/']
    exclude_patterns = ['.*\.(css|js|gif|jpg|jpeg|png)']
    response_type_whitelist = ['text/html']
    rules = [
        Rule(
            LinkExtractor(
                allow=r'https?://(www\.)?eaton\.com/',  # Only pull eaton.com links
                deny=exclude_patterns
            ),
            callback='parse_item',
            follow=True
        )
    ]

2. Confirm OffsiteMiddleware is Enabled #

Even though it’s enabled by default, explicitly set it in SPIDER_MIDDLEWARES to ensure it runs with the correct priority:

SPIDER_MIDDLEWARES = {
    'scrapy.spidermiddlewares.offsite.OffsiteMiddleware': 500,
}

This eliminates any chance of accidental disabling from custom middleware configurations.

3. Test Custom Middleware #

Temporarily disable your custom middlewares one by one to isolate the issue. Start with RandomProxyForReDirectedUrls:

DOWNLOADER_MIDDLEWARES = {
    # 'smartspider.middlewares.RandomProxyForReDirectedUrls': 650,
    'smartspider.middlewares.FilterResponses': 543,
}

If thalia.de requests stop after disabling this middleware, you’ll know it was the culprit—update it to check if the target domain is in allowed_domains before processing requests.

4. Filter Manual Requests in parse_item #

If you’re creating requests manually, add a domain check to stay within your allowed list:

from scrapy.utils.url import urlparse

def parse_item(self, response):
    # ... your existing code ...
    next_url = response.xpath('//a/@href').get()
    if next_url:
        parsed_url = urlparse(next_url)
        domain = parsed_url.netloc.lower()
        # Verify domain matches allowed list
        if domain.endswith(tuple(self.allowed_domains)):
            yield scrapy.Request(next_url, callback=self.parse_item)

Final Verdict #

Your crawler is not working as intended—it should never crawl domains outside allowed_domains. The inconsistent filtering (some domains blocked, others not) points to a breakdown in the filtering chain, most likely from missing explicit link rules or custom middleware interference.

Follow the fixes above, and you’ll have a reliable crawler that sticks to your target domain while staying adaptable to other sites (just update allowed_domains and the LinkExtractor allow rule when switching targets).

内容的提问来源于stack exchange，提问作者Gaurav