如何用Selenium实现页面横向滚动，爬取谷歌应用商店全量应用URL

问题背景 #

我尝试通过以下代码爬取Google Play应用商店（https://play.google.com/store/apps?device=）中不同设备分类下的所有应用URL：

from selenium import webdriver
from selenium.webdriver.chrome.service import Service as ChromeService
from selenium.webdriver.common.by import By
from webdriver_manager.chrome import ChromeDriverManager
import time
from tqdm import tqdm
options = webdriver.ChromeOptions()
options.add_argument('--no-sandbox')
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option("useAutomationExtension", False)
driver = webdriver.Chrome(ChromeDriverManager().install(), options=options)
driver.maximize_window()

items = ['phone','tablet','tv','chromebook','watch','car']
target_url = "https://play.google.com/store/apps?device="

all_apps = []
for cat in tqdm(items):
    driver.get(target_url+cat)
    time.sleep(2)
    
    new_height = 0
    last_height = 0
    while True:
        # Scroll down to bottom
        driver.execute_script("window.scrollTo(0, document.body.scrollHeight)")
        # Wait to load page
        time.sleep(4)

        # Calculate new scroll height and compare with last scroll height
        new_height = driver.execute_script("return document.body.scrollHeight")


        # break condition
        if new_height == last_height:
            break
        last_height = new_height
        
    for i in driver.find_elements(By.XPATH,"//a[contains(@href,'/store/apps/details')]"):
        all_apps.append(i.get_attribute('href'))

代码可实现纵向滚动并获取应用URL，但页面存在横向滚动区域（区域右侧有箭头按钮，点击可查看更多内容）。尝试点击箭头按钮时触发ElementNotInteractableException错误，即使使用ActionChains模拟鼠标悬停也无法解决。

解决方案 #

方法1：直接通过JavaScript滚动横向容器 #

Google Play的横向滚动区域是独立的DOM容器，直接定位容器后用JS控制滚动，比点击箭头更可靠，还能避免元素交互性问题：

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

# 在纵向滚动完成后，添加横向滚动处理
# 定位页面中所有横向滚动容器
horizontal_containers = WebDriverWait(driver, 10).until(
    EC.presence_of_all_elements_located((By.XPATH, "//div[contains(@class, 'VfPpkd-EScbFb-JIbuQc')]"))
)

for container in horizontal_containers:
    last_scroll_pos = 0
    while True:
        # 每次向右侧滚动500像素
        driver.execute_script("arguments[0].scrollLeft += 500;", container)
        time.sleep(1)
        # 获取当前滚动位置
        current_scroll_pos = driver.execute_script("return arguments[0].scrollLeft;", container)
        # 滚动到最右侧时停止
        if current_scroll_pos == last_scroll_pos:
            break
        last_scroll_pos = current_scroll_pos

# 最后抓取所有应用URL
for i in driver.find_elements(By.XPATH,"//a[contains(@href,'/store/apps/details')]"):
    all_apps.append(i.get_attribute('href'))

方法2：正确定位并触发箭头按钮 #

如果坚持使用点击箭头的方式，需要确保元素处于可视区域，并用JS点击绕过交互限制：

from selenium.webdriver.common.action_chains import ActionChains

# 等待所有右侧箭头按钮加载完成
arrow_buttons = WebDriverWait(driver, 10).until(
    EC.presence_of_all_elements_located((By.XPATH, "//button[@aria-label='滚动到右边']"))
)

for button in arrow_buttons:
    # 先滚动到按钮所在区域，确保可视
    driver.execute_script("arguments[0].scrollIntoView({block: 'center'});", button)
    # 使用JS点击，避免元素不可交互的问题
    driver.execute_script("arguments[0].click();", button)
    time.sleep(2)

额外优化建议 #

• 替换time.sleep()为WebDriverWait显式等待，提升代码稳定性，减少不必要的等待时间。
• 对抓取到的URL去重，避免重复存储：all_apps = list(set(all_apps))

内容的提问来源于stack exchange，提问作者Lalit Joshi