如何使用Selenium Python从Yelp平台抓取商户名称地址等公开数据

前置准备 #

• 安装依赖包：pip install selenium webdriver-manager
• 本地安装Chrome浏览器即可，无需手动下载对应版本驱动

实现逻辑 #

• 先访问目标搜索页，提取第一页所有商户详情页链接，避免页面跳转回退后元素定位失效问题
• 逐一遍历详情页，按需求提取字段，缺失字段直接跳过不输出
• 提取完成后按示例格式输出结果

完整可运行代码 #

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.by import By
import time

# 初始化浏览器
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()))
driver.implicitly_wait(10)

# 访问目标搜索页
search_url = "https://www.yelp.com/search?find_desc=nails+salons&find_loc=San+Francisco%2C+CA&ns=1"
driver.get(search_url)
time.sleep(3)

# 提取第一页所有商户详情页链接
biz_links = []
biz_name_elems = driver.find_elements(By.XPATH, '//a[@name="business-name" and contains(@href, "/biz/")]')
for elem in biz_name_elems:
    link = elem.get_attribute("href")
    if link not in biz_links:
        biz_links.append(link)

# 遍历所有商户详情页提取字段
for link in biz_links:
    driver.get(link)
    time.sleep(2)
    
    # 提取商户名称
    try:
        name = driver.find_element(By.XPATH, '//h1[contains(@class, "css-1se8maq")]').text.strip()
        print(name)
    except:
        # 无商户名直接跳过该商户
        continue
    
    # 提取地址
    try:
        address = driver.find_element(By.XPATH, '//p[contains(@class, "css-qyp8bo")]').text.strip()
        print(address)
    except:
        pass
    
    # 提取官网
    try:
        website = driver.find_element(By.XPATH, '//a[contains(@href, "biz_redir")]').text.strip()
        print(website)
    except:
        pass
    
    # 提取联系电话
    try:
        contact = driver.find_element(By.XPATH, '//p[contains(text(), "(") and contains(text(), "-")]').text.strip()
        print(contact)
    except:
        pass
    
    # 提取评价数量
    try:
        review_num = driver.find_element(By.XPATH, '//span[contains(@class, "css-1fdy0l5")]').text.strip()
        print(review_num)
    except:
        pass
    
    # 不同商户输出空行分隔
    print("\n")

# 关闭浏览器
driver.quit()

注意事项 #

• Yelp有反爬机制，代码中已添加访问延迟，若仍被拦截可适当加长sleep时间，或者添加代理IP
• 若运行时出现字段提取失败，可打开F12开发者工具更新对应元素的Xpath选择器，Yelp前端结构会不定期更新
• 爬取数据仅可用于个人学习，需遵守Yelp平台的使用规则

内容的提问来源于stack exchange，提问作者Muazam