使用Python Selenium爬取LinkedIn时无法定位下一页按钮的问题求助

问题根源 #

• 静态sleep等待时间不稳定，元素未完全加载就执行定位操作
• 原XPATH定位范围太宽泛，无法精准匹配目标按钮
• 按钮处于页面底部未滚动至可见区域，Selenium无法识别

解决方案 #

1. 用显式等待替代固定sleep

显式等待会在元素满足可交互条件后立即执行操作，比固定时长sleep更适配页面加载的不确定性。

2. 优化XPATH定位表达式

结合按钮的父容器特征或自身class属性缩小定位范围，确保精准匹配：

• 结合分页容器：//div[contains(@class, 'search-pagination')]//button[@aria-label='Далее']
• 结合按钮class：//button[@aria-label='Далее' and contains(@class, 'artdeco-button')]

3. 滚动到按钮可见区域

如果按钮在页面底部，需先滚动至该位置，保证Selenium能捕捉到元素。

修改后的代码示例 #

from selenium import webdriver
from selenium.webdriver import Keys
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC


chrome_driver_path = Service("E:\programming\chromedriver_win32\chromedriver.exe")
driver = webdriver.Chrome(service=chrome_driver_path)
url = "https://www.linkedin.com/feed/"
driver.get(url)
SEARCH_QUERY = "python developer"
LOGIN = "EMAIL"
PASSWORD = "PASSWORD"

# 等待登录链接加载并点击
sign_in_link = WebDriverWait(driver, 20).until(
    EC.element_to_be_clickable((By.XPATH, '/html/body/div[1]/main/p[1]/a'))
)
sign_in_link.click()

# 输入登录信息
login_input = WebDriverWait(driver, 10).until(
    EC.presence_of_element_located((By.ID, "username"))
)
login_input.send_keys(LOGIN)

password_input = WebDriverWait(driver, 10).until(
    EC.presence_of_element_located((By.ID, "password"))
)
password_input.send_keys(PASSWORD)

enter_button = WebDriverWait(driver, 10).until(
    EC.element_to_be_clickable((By.XPATH, '//*[@id="organic-div"]/form/div[3]/button'))
)
enter_button.click()

# 等待主页搜索按钮加载并点击
WebDriverWait(driver, 30).until(
    EC.element_to_be_clickable((By.XPATH, '//*[@id="global-nav-search"]/div/button'))
).click()

# 执行搜索操作
search_input = WebDriverWait(driver, 10).until(
    EC.presence_of_element_located((By.XPATH, '//*[@id="global-nav-typeahead"]/input'))
)
search_input.send_keys(SEARCH_QUERY)
search_input.send_keys(Keys.ENTER)

# 切换到人员标签页
WebDriverWait(driver, 15).until(
    EC.element_to_be_clickable((By.XPATH, '//*[@id="search-reusables__filters-bar"]/ul/li[1]/button'))
).click()

# 定位并点击下一页按钮
page_button = WebDriverWait(driver, 20).until(
    EC.element_to_be_clickable((By.XPATH, "//div[contains(@class, 'search-pagination')]//button[@aria-label='Далее']"))
)
# 滚动到按钮可见位置
driver.execute_script("arguments[0].scrollIntoView();", page_button)
page_button.click()

driver.quit()

额外检查项 #

• 打开浏览器开发者工具，确认目标按钮的aria-label确实为Далее，无拼写或空格差异
• 用driver.find_elements(By.XPATH, "//button[@aria-label='Далее']")获取所有匹配元素，检查是否存在多个相同标签按钮，再通过索引或其他属性筛选目标元素

内容的提问来源于stack exchange，提问作者Sergey