Selenium爬虫无法定位Messenger窗口，寻求解决方法

问题背景 #

我正在测试首个Web爬虫，目标从dsei.co.uk/exhibitor-list抓取展商数据。爬虫逻辑为点击展商名称进入详情页提取数据，点击关闭按钮返回后继续处理下一个展商。但运行约15秒后，Messenger窗口弹出遮挡关闭按钮，触发“不可点击”错误。我尝试在循环执行前添加代码关闭该窗口，却一直提示无法找到Messenger窗口元素，需要解决思路。

相关代码 #

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time

# Initialize the web driver
driver = webdriver.Chrome()

# Load the webpage
url = 'https://www.dsei.co.uk/exhibitor-list'
driver.get(url)

# Wait for the page to load and accept cookies
time.sleep(1)
accept_cookies_button = driver.find_element(By.ID, 'onetrust-accept-btn-handler')
accept_cookies_button.click()
time.sleep(20)

# Wait for the overlay element to become visible
try:
    overlay_element = WebDriverWait(driver, 30).until(
        EC.visibility_of_element_located((By.CLASS_NAME, 'css-1ecjg1h-MessageBubble-overlay-fillArea-shadowBefore-shadow'))
    )
    print("Overlay element found!")

    # Perform the required actions on the overlay element, e.g., move the cursor to trigger button visibility
    ActionChains(driver).move_to_element(overlay_element).perform()
    time.sleep(2)  # Wait for the overlay button to become visible
    
    # Click the overlay button
    overlay_button = driver.find_element(By.CLASS_NAME, 'css-oxvllm-overlay-small-button-button')
    overlay_button.click()
    print("Overlay button clicked!")

except Exception as e:
    print("Overlay element not found:", e)

# Continue with the rest of your code, including the loop

# Select the alphabet filter buttons
alphabet_buttons = driver.find_elements(By.CSS_SELECTOR, '.libraryaz__list__item a')

# Loop through the alphabet buttons
for button in alphabet_buttons:
    button.click()
    time.sleep(2)  # Wait for the page to load

    # Capture the company links before iterating over them
    company_links = driver.find_elements(By.CLASS_NAME, 'm-exhibitors-list__items__item__header__title__link')

    # Loop through the captured company links
    for i in range(len(company_links)):
        # Click on the company link
        company_links[i].click()
        time.sleep(5)  # Wait for the modal to load

        # Extract company attributes
        # ... (rest of your code for extracting company attributes)

        # Close the modal with retry using specific class names
        close_button = driver.find_element(By.CLASS_NAME, 'mfp-close')
        close_button.click()
        time.sleep(1)  # Give the modal a moment to close

# Close the browser
driver.quit()

解决思路 #

• 动态监听弹窗，替代固定延迟
  不要用time.sleep(20)提前等待弹窗，这类固定延迟极不稳定。写一个弹窗检查函数，在每次点击详情页关闭按钮前调用，用WebDriverWait动态监听Messenger弹窗的出现，找到后立即关闭。

• 优化元素定位规则
  你当前用的类名是动态生成的，页面更新后容易失效。换用更通用的定位方式，比如通过aria-label属性、按钮图标或包含的文本定位：

  def close_messenger(driver):
      try:
          # 多规则匹配关闭按钮，提高容错性
          close_btn = WebDriverWait(driver, 5).until(
              EC.element_to_be_clickable(
                  (By.XPATH, "//button[contains(@aria-label, '关闭') or contains(@class, 'overlay-close') or .//*[name()='svg' and @aria-label='Close']]")
              )
          )
          close_btn.click()
      except Exception:
          # 无弹窗则跳过，不中断流程
          pass
  

• 关键操作前强制检查弹窗
  在点击详情页的关闭按钮mfp-close之前，先调用上面的弹窗关闭函数，确保不会出现遮挡：

  # 提取完数据后，先处理可能出现的Messenger弹窗
  close_messenger(driver)
  # 等待详情页关闭按钮可点击，再执行点击
  close_button = WebDriverWait(driver, 10).until(
      EC.element_to_be_clickable((By.CLASS_NAME, 'mfp-close'))
  )
  close_button.click()
  

• 从根源阻止弹窗加载
  通过Chrome配置拦截Messenger相关请求，直接避免弹窗出现：

  from selenium.webdriver.chrome.options import Options
  
  chrome_options = Options()
  driver = webdriver.Chrome(options=chrome_options)
  # 拦截Messenger相关域名请求
  driver.execute_cdp_cmd('Network.setBlockedURLs', {
      "urls": ["*messenger.com*", "*facebook.com/plugins/messenger*"]
  })
  driver.execute_cdp_cmd('Network.enable', {})
  

• 绕过元素遮挡的点击方式
  如果弹窗偶尔还是会遮挡，用JavaScript直接触发关闭按钮的点击事件，绕过Selenium的元素可见性检查：

  close_button = WebDriverWait(driver, 10).until(
      EC.presence_of_element_located((By.CLASS_NAME, 'mfp-close'))
  )
  driver.execute_script("arguments[0].click();", close_button)
  

内容的提问来源于stack exchange，提问作者peterpeter5