使用Selenium无法从Google Maps提取元素的技术求助

尝试遍历列表提取Google Maps中的商家名称、地址、URL，脚本能正确点击首个搜索结果，但无法提取任何信息。已尝试显式等待及多种选择器，作为Selenium新手寻求技术帮助。当前代码如下：

import csv
from selenium import webdriver
from selenium.webdriver.common.by import By
from bs4 import BeautifulSoup
#from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait as wait

driver = webdriver.Chrome() 


url = 'https://www.google.com/maps/search/Ace+Hardware'
driver.get(url)
#wait = WebDriverWait(driver, 10)
# Locate and click on the link to the first search result
result = driver.find_element(By.XPATH,'//*[@id="QA0Szd"]/div/div/div[1]/div[2]/div/div[1]/div/div/div[2]/div[1]/div[3]/div/a')
result.click()

page_content = driver.page_source
wait(driver, 30).until(EC.presence_of_element_located((By.XPATH,'/html/body/div[3]/div[8]/div[9]/div/div/div[1]/div[2]/div/div[1]/div/div/div[9]/div[3]/button/div/div[2]/div[1]')))
url_element = driver.find_element(By.XPATH,'/html/body/div[3]/div[8]/div[9]/div/div/div[1]/div[2]/div/div[1]/div/div/div[9]/div[3]/button/div/div[2]/div[1]')
url = url_element.text
address_element = driver.find_element(By.XPATH,'//*[@id="QA0Szd"]/div/div/div[1]/div[3]/div/div[1]/div/div/div[2]/div[9]/div[3]/button/div/div[2]/div[1]')
address = address_element.text
print(url)
print(address)
driver.quit()

核心问题分析 #

• 绝对XPATH过于脆弱：Google Maps的DOM结构会频繁更新，你检查时的元素路径在实际运行时可能已发生变化
• 等待逻辑不准确：仅等待元素存在无法确保元素已加载完成并可交互
• URL提取方式错误：商家官网URL并非按钮文本，需要从对应元素的属性中获取

修正后的代码 #

import csv
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Chrome()
driver.maximize_window()  # 避免窗口过小导致元素被遮挡

url = 'https://www.google.com/maps/search/Ace+Hardware'
driver.get(url)

wait = WebDriverWait(driver, 20)

# 点击第一个搜索结果：用相对定位提升稳定性
first_result = wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, 'div[role="feed"] div[jsaction*="pane.rws"] a')))
first_result.click()

# 等待详情页加载完成：以商家名称元素作为加载完成标志
business_name = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, 'h1.fontHeadlineLarge'))).text

# 提取地址：通过地址按钮的子元素定位
address = wait.until(EC.visibility_of_element_located((By.XPATH, '//button[@data-item-id="address"]//div[@class="fontBodyMedium"]'))).text

# 提取商家官网URL：从a标签的href属性获取
try:
    website_url = wait.until(EC.visibility_of_element_located((By.XPATH, '//a[@data-item-id="authority"]'))).get_attribute('href')
except:
    website_url = "无官网信息"

# 提取Google Maps商家页面链接
maps_url = driver.current_url

print("商家名称:", business_name)
print("地址:", address)
print("商家官网:", website_url)
print("Google Maps链接:", maps_url)

driver.quit()

关键改进点 #

• 改用相对定位：基于元素的角色、属性而非绝对路径定位，适配Google Maps的DOM变化
• 优化等待条件：等待元素可见/可点击，确保元素已完全加载可交互
• 正确提取URL：商家官网从<a>标签的href属性获取，Google Maps自身链接直接取当前页面URL
• 增加异常处理：应对部分商家无官网的情况，避免脚本崩溃

内容的提问来源于stack exchange，提问作者Aidan Rosamond