Scrapy、Crawlera与Splash访问特定网站超时问题求助

Great question! It's frustrating when tools like Splash underperform compared to Selenium, especially when you want to stick with Scrapy's ecosystem. Let's break down the possible fixes to get your Splash setup working reliably (and faster):

1. Bump Up Timeout Parameters #

Your local Splash instance takes 20+ seconds to load the site, which means the default timeout limits on Scrapinghub's small Splash instance (usually around 15s) are definitely too tight. Adjust both the overall timeout and per-resource resource_timeout in your Splash request:

yield SplashRequest(
    url=your_target_url,
    callback=self.parse,
    args={
        'timeout': 30,  # Give Splash enough time to fully render
        'resource_timeout': 25,  # Prevent hanging on slow external resources
        'wait': 5  # Wait a few seconds after page load for JS to execute
    }
)

2. Optimize Resource Loading to Speed Up Rendering #

Splash loads every resource by default (images, ads, third-party scripts) which adds unnecessary delay. Disable non-critical resources to cut down load time:

• Disable images: Add 'images': 0 to the args dict
• Block third-party domains: Use a Lua script to filter requests (example below)
• Disable unused plugins: Add 'plugins': 0 if the site doesn't need Flash/Java

Here's a quick Lua script to block unwanted domains:

function main(splash)
    splash:on_request(function(request)
        if string.find(request.url, 'ad-domain.com') or string.find(request.url, 'analytics-domain.com') then
            request:abort()
        end
    end)
    splash:go(splash.args.url)
    splash:wait(splash.args.wait)
    return splash:html()
end

3. Adjust Splash Instance Resources #

• Scrapinghub Small Instance: Small Splash instances have limited CPU/RAM, which causes timeouts for heavy sites. Upgrade to a medium or large instance—this alone might fix the timeout issue.
• Local Aquarium Instance: Allocate more resources to your local Splash container (if using Docker). For example, run it with --cpus=2 --memory=4g to give it more processing power, which should reduce that 20s load time closer to Firefox's 10s.

4. Fix Crawlera + Splash Integration #

If you're combining Crawlera with Splash, make sure the proxy is configured inside Splash, not just in Scrapy. Pass Crawlera credentials to Splash via the proxy argument:

yield SplashRequest(
    url=your_target_url,
    callback=self.parse,
    args={
        'timeout': 30,
        'proxy': 'http://USERNAME:PASSWORD@proxy.crawlera.com:8010/',
        'wait': 5
    }
)

This ensures all Splash requests route through Crawlera, avoiding any anti-bot blocks that might be causing timeouts.

5. Mimic Firefox's Fingerprint #

Splash's default browser fingerprint might be detected by the site's anti-bot system. Match Firefox's user agent and browser settings to make Splash look more like a real browser:

yield SplashRequest(
    url=your_target_url,
    callback=self.parse,
    args={
        'timeout': 30,
        'user_agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/115.0',
        'viewport': '1920x1080',
        'wait': 5
    }
)

You can also add small interactions in your Lua script (like scrolling or a mouse click) to mimic human behavior:

function main(splash)
    splash:set_user_agent(splash.args.user_agent)
    splash:go(splash.args.url)
    splash:wait(2)
    -- Simulate scrolling to trigger lazy-loaded content
    splash:runjs('window.scrollTo(0, document.body.scrollHeight);')
    splash:wait(3)
    return splash:html()
end

6. Check for Site-Specific JS Dependencies #

Some sites rely on specific JS features that Splash's WebKit engine handles poorly. If the site uses modern JS APIs, try enabling Splash's --enable-javascript flag (it's enabled by default, but double-check) or use the latest Splash version in Aquarium/Scrapinghub.

内容的提问来源于stack exchange，提问作者Szabolcs