如何用Python BeautifulSoup抓取下拉选项选择后加载的数据

一、无头模式能否用Selenium选择选项？ #

是

二、两种可行抓取方案 #

方案1：直接模拟POST请求（高效推荐） #

1. 核心参数分析
   打开浏览器F12开发者工具，选择下拉选项点击「Get Results」，在「网络」标签中找到POST请求，可看到关键表单参数：

   • ctl00$ContentPlaceHolder1$orgtype：对应下拉选项的value值（如5,12对应「Public School District」）
   • 需附带页面隐藏字段__VIEWSTATE、__VIEWSTATEGENERATOR、__EVENTVALIDATION，这些可从初始页面HTML中提取

2. 代码示例

   import requests
   from bs4 import BeautifulSoup
   import time
   
   base_url = "https://profiles.doe.mass.edu/search/search.aspx?leftNavId=11238"
   session = requests.Session()
   
   # 获取初始页面的隐藏参数
   response = session.get(base_url)
   soup = BeautifulSoup(response.text, "html.parser")
   
   viewstate = soup.find("input", {"id": "__VIEWSTATE"})["value"]
   viewstategenerator = soup.find("input", {"id": "__VIEWSTATEGENERATOR"})["value"]
   eventvalidation = soup.find("input", {"id": "__EVENTVALIDATION"})["value"]
   
   # 定义要遍历的所有下拉选项
   options = [
       ("5,12", "Public School District"),
       ("6,13", "Public School"),
       ("26,25", "Alternative Education"),
       ("13", "Charter Public School"),
       ("3", "Collaborative"),
       # 剩余选项按此格式补充
   ]
   
   for option_value, option_name in options:
       # 构造POST表单数据
       form_data = {
           "__VIEWSTATE": viewstate,
           "__VIEWSTATEGENERATOR": viewstategenerator,
           "__EVENTVALIDATION": eventvalidation,
           "ctl00$ContentPlaceHolder1$orgtype": option_value,
           "ctl00$ContentPlaceHolder1$btnSearch": "Get Results"
       }
   
       # 发送请求并解析结果
       result_response = session.post(base_url, data=form_data)
       result_soup = BeautifulSoup(result_response.text, "html.parser")
   
       # 提取学校数据（需根据页面实际结构调整选择器）
       school_list = result_soup.find("div", id="innerWrapper").find_all("a")
       print(f"=== {option_name} 数据 ===")
       for school in school_list:
           print(school.get_text(strip=True))
       
       # 添加请求间隔，避免反爬
       time.sleep(2)
   

方案2：使用Selenium（适配复杂动态场景） #

1. 核心操作逻辑
   初始化无头浏览器，遍历所有下拉选项，逐个选择后触发查询，等待数据加载完成后提取内容

2. 代码示例

   from selenium import webdriver
   from selenium.webdriver.support.ui import Select
   from selenium.webdriver.common.by import By
   from selenium.webdriver.support.ui import WebDriverWait
   from selenium.webdriver.support import expected_conditions as EC
   from bs4 import BeautifulSoup
   import time
   
   # 无头模式配置
   chrome_options = webdriver.ChromeOptions()
   chrome_options.add_argument("--headless=new")
   driver = webdriver.Chrome(options=chrome_options)
   
   driver.get("https://profiles.doe.mass.edu/search/search.aspx?leftNavId=11238")
   
   # 获取下拉框元素
   select_box = Select(driver.find_element(By.ID, "ctl00_ContentPlaceHolder1_orgtype"))
   
   # 遍历所有选项（跳过默认的--Select--）
   for option in select_box.options[1:]:
       option_value = option.get_attribute("value")
       option_name = option.text
       select_box.select_by_value(option_value)
   
       # 点击查询按钮
       driver.find_element(By.ID, "ctl00_ContentPlaceHolder1_btnSearch").click()
   
       # 等待结果加载完成
       WebDriverWait(driver, 10).until(
           EC.presence_of_element_located((By.ID, "ctl00_ContentPlaceHolder1_lblSearchResults"))
       )
   
       # 解析页面数据
       result_soup = BeautifulSoup(driver.page_source, "html.parser")
       school_list = result_soup.find_all("div", class_="school-info")  # 按实际结构调整
       print(f"=== {option_name} 数据 ===")
       for school in school_list:
           print(school.get_text(strip=True))
       
       time.sleep(2)
   
   driver.quit()
   

三、注意事项 #

• 频繁请求易触发反爬，需添加请求间隔
• POST请求的隐藏参数可能过期，需每次请求前重新从页面获取
• Selenium模式下，需确保ChromeDriver与浏览器版本匹配

内容的提问来源于stack exchange，提问作者theycallmepix