如何用Selenium实现Google Maps多页循环爬取？

原代码存在的核心问题 #

• 下一页按钮定位错误：原代码找的是"Mais"，但Google Maps葡萄牙语界面的下一页按钮文本是Próxima，导致无法触发翻页
• 电话缺失时的变量错误：异常分支里错误地追加未定义的tel，应该用sem_telefone
• 未重新获取翻页后的商家列表：每次循环只爬取第一页的商家，翻页后没有刷新元素集合
• 依赖固定sleep：稳定性差，容易因页面加载慢导致元素定位失败

修改后的完整代码 #

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import pandas as pd
import time

# 初始化浏览器
driver = webdriver.Chrome()
wait = WebDriverWait(driver, 20)

procurar = "contabilidade em Curitiba"
# 初始化存储列表
Nome = []
Endereco = []
Telefone = []
# 记录已爬取商家，避免重复
crawled_names = set()

# 打开初始搜索页面
url = "https://www.google.com/search?q=contabilidade+em+manaus&tbm=lcl"
driver.get(url)
driver.maximize_window()

# 替换搜索关键词
wait.until(EC.presence_of_element_located((By.XPATH, "//input[@value='contabilidade em manaus']"))).clear()
input_buscar = wait.until(EC.presence_of_element_located((By.XPATH, "//input[@aria-label='Pesquisar']")))
input_buscar.send_keys(procurar, Keys.ENTER)
time.sleep(2)

while True:
    try:
        # 等待当前页商家列表加载完成
        wait.until(EC.presence_of_all_elements_located((By.XPATH, "//div[@class='rllt__details']")))
        classe_empresas = driver.find_elements(By.XPATH, "//div[@class='rllt__details']")
        
        for empresa in classe_empresas:
            # 滚动到商家元素位置，避免点击失败
            driver.execute_script("arguments[0].scrollIntoView();", empresa)
            time.sleep(1)
            empresa.click()
            time.sleep(2)
            
            # 获取商家名称
            nome = wait.until(EC.presence_of_element_located((By.XPATH, "//h2[@data-attrid='title']"))).text
            # 跳过已爬取的商家
            if nome in crawled_names:
                continue
            crawled_names.add(nome)
            Nome.append(nome)
            print(f"Nome: {nome}")
            
            # 获取地址
            endereco = wait.until(EC.presence_of_element_located((By.XPATH, "//span[@class='LrzXr']"))).get_attribute("innerHTML")
            Endereco.append(endereco)
            print(f"Endereço: {endereco}")
            
            # 获取电话，处理无电话情况
            try:
                tel = wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, ".LrzXr.zdqRlf.kno-fv"))).text
                Telefone.append(tel)
                print(f"Telefone: {tel}")
            except:
                sem_telefone = "Não Tem Telefone Cadastrado"
                Telefone.append(sem_telefone)
                print(f"Telefone: {sem_telefone}")
        
        # 点击下一页按钮
        next_button = wait.until(EC.element_to_be_clickable((By.XPATH, "//span[normalize-space()='Próxima']")))
        # 滚动到按钮位置
        driver.execute_script("arguments[0].scrollIntoView();", next_button)
        time.sleep(1)
        next_button.click()
        time.sleep(3)
        
    except Exception as e:
        print(f"爬取结束或遇到错误: {str(e)}")
        break

# 保存数据到Excel
data = {'Nome': Nome, 'Endereço': Endereco, 'Telefone': Telefone} 
df = pd.DataFrame(data)
df.to_excel('GoogleMaps_Curitiba.xlsx', engine='xlsxwriter', index=False)
print("\n爬取完成，数据已保存到GoogleMaps_Curitiba.xlsx")
print(df)

# 关闭浏览器
driver.quit()

关键修改说明 #

1. 修正下一页按钮定位：将原代码的"Mais"改为葡萄牙语界面的"Próxima"，同时用WebDriverWait确保按钮可点击
2. 增加去重逻辑：用集合crawled_names记录已爬取的商家名称，避免重复数据
3. 修复电话异常处理：将错误的Telefone.append(tel)改为Telefone.append(sem_telefone)
4. 替换固定sleep为显式等待：用WebDriverWait等待元素加载，提升爬取稳定性
5. 增加滚动操作：在点击元素前滚动到目标位置，避免因元素在视窗外导致点击失败
6. 优化初始URL：简化初始URL，避免携带过期的参数

内容的提问来源于stack exchange，提问作者Felipe