使用Selenium和Scrapy爬取站点返回None，无法提取下拉框与表格数据

核心问题定位 #

• 点击查询按钮后未等待表格完成渲染就直接提取页面源码，此时页面还未加载出表格数据，因此提取结果为空
• 直接点击<option>元素的交互方式稳定性差，下拉框选择应使用Selenium官方提供的Select工具类实现
• Scrapy Selector执行XPath查询后未调用取值方法，打印内容为Selector对象而非实际文本值，看起来和返回None的效果一致
• 缺少driver资源清理逻辑，浏览器进程会残留后台占用资源

修复后代码 #

import scrapy
from scrapy import Selector
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
# 导入Select工具类处理下拉框
from selenium.webdriver.support.ui import Select

class CanaraBankSpider(scrapy.Spider):
    """This class defines Canara Bank Spider."""
    name = "canara_bank_ins"
    URL = "https://canarabank.com/locator.aspx"
    start_urls = (URL,)
    brand_name = "Canara Bank"
    spider_type = "chain"

    def parse(self, response, **kwargs):
        driver = webdriver.Chrome()
        try:
            driver.get(self.URL)
            # 1. 用Select类处理下拉框选择，稳定性更高
            locator_select = Select(WebDriverWait(driver, 20).until(
                EC.element_to_be_clickable((By.NAME, "ctl00$ContentPlaceHolder1$ddllocater"))
            ))
            locator_select.select_by_visible_text("Branch Details")

            state_select = Select(WebDriverWait(driver, 20).until(
                EC.element_to_be_clickable((By.NAME, "ctl00$ContentPlaceHolder1$ddlstate"))
            ))
            state_select.select_by_visible_text("Bihar")

            district_select = Select(WebDriverWait(driver, 20).until(
                EC.element_to_be_clickable((By.NAME, "ctl00$ContentPlaceHolder1$ddldistrict"))
            ))
            district_select.select_by_visible_text("Gaya")

            # 点击查询按钮
            WebDriverWait(driver, 20).until(
                EC.element_to_be_clickable((By.NAME, "ctl00$ContentPlaceHolder1$btnview"))
            ).click()

            # 2. 新增等待：等待表格渲染完成再提取页面源码
            WebDriverWait(driver, 20).until(
                EC.visibility_of_element_located((By.ID, "ContentPlaceHolder1_gvlocatordata"))
            )

            selector = Selector(text=driver.page_source)
            # 3. 调用.get()方法提取实际文本值，要取多个值用.getall()
            target_text = selector.xpath('//*[@id="ContentPlaceHolder1_gvlocatordata"]//tr[2]/td[3]/text()').get()
            print(target_text)
        finally:
            # 4. 无论是否报错都关闭driver，释放资源
            driver.quit()

动态获取下拉框选项方法 #

无需硬编码选项值，可通过Select类的options属性批量获取所有可选值：

# 示例：获取所有州的选项值
state_select = Select(driver.find_element(By.NAME, "ctl00$ContentPlaceHolder1$ddlstate"))
all_state_options = [opt.text.strip() for opt in state_select.options if opt.text.strip()]

内容的提问来源于stack exchange，提问作者AAYUSH JAIN