使用Scrapy+Splash爬取钢锭价格时无法操作日期下拉菜单求助

Hey there! Let's tackle this dropdown issue you're facing with Scrapy + Splash for fetching steel ingot prices. I've dealt with similar dynamic dropdowns on industrial pricing sites before, so here's a step-by-step solution tailored to your needs.

Most date dropdowns on sites like SteelMint fall into two categories: native HTML <select> elements, or custom JS-rendered dropdowns (div-based). Let's cover both scenarios with working Lua scripts, since your current script is incomplete.

1. Handling Native <select> Dropdowns #

If the date picker is a standard <select> element, we can use JavaScript to set the desired value directly (faster than simulating clicks) and trigger the page to load corresponding prices.

Complete Lua Script for Native Select #

function main(splash, args)
    -- Mimic a real browser to avoid anti-scraping blocks
    splash:set_user_agent("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36")
    splash.private_mode_enabled = false
    splash:init_cookies(args.cookies)

    -- Load the page and wait for JS to render elements
    assert(splash:go(args.url))
    assert(splash:wait(2))

    -- Target the date dropdown (replace with actual CSS selector from your page)
    local date_select = splash:select("#date-selector-id")
    if date_select then
        -- Set target date value and trigger change event to load prices
        splash:runjs([[
            const selectElement = document.querySelector('#date-selector-id');
            selectElement.value = '2024-05-20'; // Replace with your target date's value
            selectElement.dispatchEvent(new Event('change'));
        ]])

        -- Wait for price data to refresh after date change
        assert(splash:wait(1.5))
    else
        return {error = "Date dropdown element not found"}
    end

    -- Extract date and price data (update selectors to match your page structure)
    local selected_date = splash:evaljs([[
        const select = document.querySelector('#date-selector-id');
        select.options[select.selectedIndex].text
    ]])
    local price = splash:select(".ingot-price-value"):text()

    -- Return data to Scrapy
    return {
        selected_date = selected_date,
        price = price,
        html = splash:html() -- Optional, for debugging
    }
end

2. Handling Custom JS-Rendered Dropdowns #

If the date picker is a custom dropdown (e.g., a clickable div that expands into a list of dates), we need to simulate the full click sequence to select a date.

Complete Lua Script for Custom Dropdown #

function main(splash, args)
    splash:set_user_agent("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36")
    splash.private_mode_enabled = false
    splash:init_cookies(args.cookies)

    assert(splash:go(args.url))
    assert(splash:wait(2))

    -- Step 1: Click the dropdown trigger to expand options
    local dropdown_trigger = splash:select(".date-dropdown-trigger")
    if dropdown_trigger then
        dropdown_trigger:click()
        assert(splash:wait(0.5)) -- Wait for options to appear
    else
        return {error = "Dropdown trigger element not found"}
    end

    -- Step 2: Click the target date option (update selector/text to match your date)
    local target_date = splash:select(".date-option-item:contains('May 20, 2024')")
    if target_date then
        target_date:click()
        assert(splash:wait(1.5)) -- Wait for price data to load
    else
        return {error = "Target date option not found"}
    end

    -- Extract date and price
    local selected_date = splash:select(".selected-date-display"):text()
    local price = splash:select(".ingot-price-value"):text()

    return {
        selected_date = selected_date,
        price = price
    }
end

3. Integrating with Scrapy #

In your Scrapy spider, use SplashRequest to execute the Lua script and retrieve the data:

import scrapy
from scrapy_splash import SplashRequest

class SteelIngotSpider(scrapy.Spider):
    name = "steelmint_ingot"
    start_urls = ["https://www.steelmint.com/ingot-prices-indian"]

    def start_requests(self):
        for url in self.start_urls:
            yield SplashRequest(
                url,
                callback=self.parse,
                endpoint='execute',
                args={
                    'lua_source': open('your_lua_script.lua').read(),
                    # Optional: pass target date as a dynamic argument
                    'target_date': '2024-05-20'
                }
            )

    def parse(self, response):
        # Extract data from Splash's response
        selected_date = response.data['selected_date'].strip()
        price = response.data['price'].strip()
        yield {
            'date': selected_date,
            'price': price
        }

Key Debugging Tips #

• Use splash:html() in your Lua script to return the page source after interactions—this helps verify if the dropdown was triggered correctly.
• Use browser dev tools (F12) to inspect dropdown elements and get accurate CSS selectors (never guess them!).
• Adjust splash:wait() times based on how fast the page loads—some industrial sites are slower to render dynamic content.
• Keep your user agent updated and enable cookies to avoid being flagged as a bot.

内容的提问来源于stack exchange，提问作者Himanshu Sharma