Splash 3.0执行Lua脚本时特定URL出现http403错误的技术求助

This looks like a classic anti-scraping block—your Splash instance is being flagged as a non-human visitor by the target site's server, hence the 403 error. Since other URLs work fine, the issue is specific to how this particular site detects and blocks automated tools like Splash. Let's walk through actionable fixes tailored to your setup:

1. Spoof a Realistic User-Agent #

Most sites flag requests with generic or bot-specific User-Agent headers. Update your Lua script to use a modern browser's UA string:

function main(splash, args)
    -- Use a current Chrome User-Agent
    splash:set_user_agent("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36")
    splash.resource_timeout = 20.0
    splash:go(args.url)
    splash:wait(10.0)
    splash.scroll_position={0, 600}
    return splash:html()
end

2. Add Complete Request Headers #

Real browsers send a full set of headers beyond just User-Agent. Use splash:set_custom_headers() to replicate this:

function main(splash, args)
    splash:set_user_agent("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36")
    -- Mimic real browser headers
    splash:set_custom_headers({
        ["Accept"] = "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
        ["Accept-Language"] = "en-US,en;q=0.5",
        ["Accept-Encoding"] = "gzip, deflate, br",
        ["Connection"] = "keep-alive",
        ["Upgrade-Insecure-Requests"] = "1"
    })
    splash.resource_timeout = 20.0
    splash:go(args.url)
    splash:wait(10.0)
    splash.scroll_position={0, 600}
    return splash:html()
end

3. Mimic Human-like Interactions #

Static waits and predictable actions are dead giveaways for bots. Add random delays and mouse movements to make the request look natural:

function main(splash, args)
    splash:set_user_agent("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36")
    splash:set_custom_headers({
        ["Accept"] = "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
        ["Accept-Language"] = "en-US,en;q=0.5",
        ["Accept-Encoding"] = "gzip, deflate, br",
        ["Connection"] = "keep-alive",
        ["Upgrade-Insecure-Requests"] = "1"
    })
    splash.resource_timeout = 20.0
    
    splash:go(args.url)
    splash:wait(math.random(8, 12)) -- Random wait between 8-12s instead of fixed 10s
    
    -- Simulate random mouse movements
    splash:mouse_move(math.random(100, 300), math.random(200, 400))
    splash:wait(0.5)
    splash:mouse_move(math.random(400, 600), math.random(100, 300))
    
    splash.scroll_position={0, 600}
    splash:wait(1) -- Short pause after scrolling
    return splash:html()
end

4. Enable Proper Session & JS Handling #

Ensure Splash allows cookie persistence and full JavaScript execution (many sites use JS to validate visitors):

function main(splash, args)
    splash.js_enabled = true -- Explicitly enable JS (default is true, but explicit helps)
    splash.private_mode_enabled = false -- Allow cookies to persist between requests
    splash:set_user_agent("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36")
    -- ... rest of headers and actions ...
end

5. Check for IP Blocking #

If the above fixes don't work, your server's IP might be blocked by the target site. You can add a proxy to your Lua script:

function main(splash, args)
    splash:set_proxy{
        host = "your-proxy-host",
        port = 1234,
        -- Uncomment if proxy requires auth:
        -- username="proxy-username",
        -- password="proxy-password"
    }
    -- ... rest of the script ...
end

Or pass the proxy directly in your Scrapy SplashRequest:

yield SplashRequest(
    url=url,
    callback=self.parse,
    meta={'site': i},
    dont_filter=True,
    endpoint='execute',
    args={
        'lua_source': script,
        'timeout': 70,
        'proxy': 'http://your-proxy-host:1234'
    }
)

Troubleshooting Tip #

To debug exactly what headers Splash is sending, temporarily replace return splash:html() with return splash:har(). This returns a HAR (HTTP Archive) file showing all request/response details—compare this to a real browser's request headers (via Chrome DevTools) to spot missing or suspicious values.

内容的提问来源于stack exchange，提问作者user3520253