Python Selenium滚动后无法抓取谷歌搜索新加载链接问题

问题核心是滚动页面后立即执行抓取操作，此时浏览器还未完成新加载内容的渲染，导致Selenium无法定位到新出现的链接元素。以下是几种可行的修复方案：

方案1：添加固定等待时间 #

滚动后给浏览器预留足够的加载时间，再执行抓取操作：

from selenium import webdriver
import time

driver = webdriver.Firefox()
driver.set_window_size(673, 720)
query = "jobs for back-end dev"
driver.get(f"https://google.com/search?q={query}")

# 滚动到底部
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
# 等待3秒让页面加载新内容
time.sleep(3)

# 抓取所有链接
elems = driver.find_elements("xpath", "//a[@jsname='UWckNb']")
for elem in elems:
    link = elem.get_attribute("href")
    if "https://www.google.com" not in link:
        print(link)

方案2：使用显式等待（更可靠） #

通过显式等待监听元素数量变化，确保新内容加载完成后再抓取，避免固定等待的不确定性：

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By

driver = webdriver.Firefox()
driver.set_window_size(673, 720)
query = "jobs for back-end dev"
driver.get(f"https://google.com/search?q={query}")

# 获取初始链接数量
initial_link_count = len(driver.find_elements(By.XPATH, "//a[@jsname='UWckNb']"))

# 滚动到底部
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

# 等待链接数量增加（最多等待10秒）
WebDriverWait(driver, 10).until(
    lambda d: len(d.find_elements(By.XPATH, "//a[@jsname='UWckNb']")) > initial_link_count
)

# 抓取所有链接
elems = driver.find_elements(By.XPATH, "//a[@jsname='UWckNb']")
for elem in elems:
    link = elem.get_attribute("href")
    if "https://www.google.com" not in link:
        print(link)

额外处理：应对"显示更多结果"按钮 #

部分谷歌搜索结果滚动到底部后会出现"显示更多结果"按钮（而非自动加载），此时需要添加按钮点击逻辑：

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
import time

driver = webdriver.Firefox()
driver.set_window_size(673, 720)
query = "jobs for back-end dev"
driver.get(f"https://google.com/search?q={query}")

# 滚动到底部
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

# 尝试点击"显示更多结果"按钮
try:
    show_more_btn = WebDriverWait(driver, 5).until(
        EC.element_to_be_clickable((By.XPATH, "//input[@value='显示更多结果']"))
    )
    show_more_btn.click()
    time.sleep(3)
except:
    # 没有按钮则跳过
    pass

# 抓取所有链接
elems = driver.find_elements(By.XPATH, "//a[@jsname='UWckNb']")
for elem in elems:
    link = elem.get_attribute("href")
    if "https://www.google.com" not in link:
        print(link)

内容的提问来源于stack exchange，提问作者eternalodballl