如何通过Python的Selenium关闭弹窗以继续网页爬取

核心思路 #

点击"View on Map"链接后，先判断当前弹出的是正常详情弹窗还是需点击OK的提示弹窗，针对性处理后再继续爬取流程。

具体解决方案 #

1. 增加弹窗检测逻辑：点击元素后先尝试捕获特殊弹窗的OK按钮，若存在则点击关闭；若不存在则正常爬取详情。
2. 多维度定位OK按钮：根据弹窗实际结构，可选择以下几种定位方式：
   • 按按钮文本（兼容空格）：//button[normalize-space()='OK']
   • 按CSS类名：button.btn.btn-default（若按钮有对应样式类）
   • 按aria属性：button[aria-label='OK']

修改后的完整代码 #

from selenium import webdriver
from selenium.webdriver.support.wait import WebDriverWait    
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.select import Select
import pandas as pd
import time

driver = webdriver.Chrome()
driver.get('https://mspotrace.org.my/Sccs_list')
time.sleep(20)

# 选择最大条目数
elem = driver.find_element_by_css_selector('select[name=dTable_length]')
select = Select(elem)
select.select_by_value('500')
time.sleep(15)

# 获取所有可点击元素
elements = WebDriverWait(driver, 20).until(EC.presence_of_all_elements_located((By.XPATH, "//a[@title='View on Map']")))

# 初始化DataFrame
df = pd.DataFrame(columns=['facility_name','other_details'])
pos = 0

for element in elements:
    data = []
    element.click()
    time.sleep(2)  # 缩短固定等待，优先用显式等待
    
    try:
        # 检测是否出现需点击OK的特殊弹窗
        ok_button = WebDriverWait(driver, 5).until(
            EC.element_to_be_clickable((By.XPATH, "//button[normalize-space()='OK']"))
        )
        ok_button.click()
        print(f"跳过第{pos}条：无地理信息")
        pos += 1
        time.sleep(1)
        continue
    except:
        # 无特殊弹窗，正常爬取详情
        pass
    
    try:
        # 等待详情弹窗加载完成
        facility_name = WebDriverWait(driver, 10).until(
            EC.presence_of_element_located((By.XPATH, '//h4[@class="modal-title"]'))
        ).text
        other_details = WebDriverWait(driver, 10).until(
            EC.presence_of_element_located((By.XPATH, '//div[@class="modal-body"]'))
        ).text
        
        data.append(facility_name)
        data.append(other_details)
        df.loc[pos] = data
        
        # 关闭详情弹窗
        close_button = WebDriverWait(driver, 10).until(
            EC.element_to_be_clickable((By.CSS_SELECTOR, "button[aria-label='Close'] > span"))
        )
        close_button.click()
        time.sleep(1)
        pos += 1
    except Exception as e:
        print(f"处理第{pos}条时出错：{str(e)}")
        # 尝试关闭当前弹窗，避免阻塞后续流程
        try:
            driver.find_element_by_css_selector("button[aria-label='Close'] > span").click()
        except:
            pass
        pos += 1
        continue

print(df)
driver.quit()

关键优化点 #

• 用显式等待替代固定time.sleep，提升代码稳定性与执行效率
• 增加分层异常捕获，分别处理特殊弹窗、正常爬取两种场景
• 出错后自动尝试关闭弹窗，避免后续流程被中断

内容的提问来源于stack exchange，提问作者Funkeh-Monkeh