Selenium爬取LinkedIn职位href仅返回前7条，后续结果无法获取

使用Selenium爬取LinkedIn职位列表的href链接时，仅能获取前7条结果，第8条及以后的结果无法获取，即便指定范围也无效。已添加睡眠等待确保页面加载，但问题仍存在。疑问：这是代码bug、网站安全措施还是代码存在问题？

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
from icecream import ic

PATH = "C:\\Program Files (x86)\\chromedriver.exe"
service = Service(executable_path=PATH)
driver = webdriver.Chrome(service=service)
user = "*********"
pswd = "***********"

driver.get("https://www.linkedin.com/jobs")
driver.implicitly_wait(10)
userLogin = driver.find_element('id', "session_key")
pswdLogin = driver.find_element('id', "session_password")
userLogin.send_keys(user)
pswdLogin.send_keys(pswd)
pswdLogin.send_keys(Keys.RETURN)
driver.implicitly_wait(600)
searchKey = driver.find_element(By.XPATH, '//input[contains(@id, "jobs-search-box-keyword-id-ember")]')
searchLocation = driver.find_element(By.XPATH, '//input[contains(@id, "jobs-search-box-location-id-ember")]')

searchKey.send_keys("python developer")
searchLocation.send_keys("27560")
time.sleep(0.25)
searchLocation.send_keys(Keys.RETURN)
WebDriverWait(driver, 3)
time.sleep(2)
jobs = []
for i in range(1, 16):
   ic(i)
   the_url = driver.find_element(by=By.XPATH,
                                 value='/html/body/div[5]/div[3]/div[4]/div/div/main/div/div[2]/div[1]/div/ul/li[' + str(
                                     i) + ']/div/div/div[1]/div[2]/div[1]/a')
   print(the_url.get_attribute('href'))

1. 绝对XPATH不可靠：你使用的是从根节点开始的绝对XPATH，LinkedIn的页面DOM结构会动态变化，且后续职位元素未出现在视口时，可能尚未渲染，导致定位失败。
2. 未处理滚动加载：LinkedIn职位列表采用滚动加载机制，只有当元素进入浏览器视口后才会加载DOM节点，直接循环定位第8条及以后的元素时，这些元素还未被渲染。
3. 等待方式不合理：WebDriverWait(driver, 3)没有指定等待条件，只是空等；固定time.sleep无法适配页面实际加载速度，可能导致元素未加载完成就执行定位。

1. 使用相对定位替代绝对XPATH #

通过职位列表容器和职位卡片的特征定位元素，避免依赖不稳定的绝对路径。

2. 实现滚动加载逻辑 #

每次滚动到当前列表的底部，触发新职位加载，等待新元素出现后再继续获取。

3. 用显式等待确保元素可访问 #

针对每个操作设置明确的等待条件，比如等待元素可见、可点击。

修改后的代码 #

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time

PATH = "C:\\Program Files (x86)\\chromedriver.exe"
service = Service(executable_path=PATH)
driver = webdriver.Chrome(service=service)
user = "*********"
pswd = "***********"

driver.get("https://www.linkedin.com/jobs")
# 等待登录元素加载
WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.ID, "session_key")))

userLogin = driver.find_element(By.ID, "session_key")
pswdLogin = driver.find_element(By.ID, "session_password")
userLogin.send_keys(user)
pswdLogin.send_keys(pswd)
pswdLogin.send_keys(Keys.RETURN)

# 等待搜索框加载
WebDriverWait(driver, 15).until(EC.visibility_of_element_located((By.XPATH, '//input[contains(@id, "jobs-search-box-keyword-id-ember")]')))
searchKey = driver.find_element(By.XPATH, '//input[contains(@id, "jobs-search-box-keyword-id-ember")]')
searchLocation = driver.find_element(By.XPATH, '//input[contains(@id, "jobs-search-box-location-id-ember")]')

searchKey.send_keys("python developer")
searchLocation.send_keys("27560")
time.sleep(0.25)
searchLocation.send_keys(Keys.RETURN)

# 等待职位列表加载
WebDriverWait(driver, 15).until(EC.visibility_of_element_located((By.CLASS_NAME, "jobs-search-results-list")))

jobs = []
target_count = 15  # 要获取的职位数量
last_height = driver.execute_script("return document.body.scrollHeight")

while len(jobs) < target_count:
    # 获取当前页面所有职位卡片的链接
    job_cards = WebDriverWait(driver, 10).until(
        EC.presence_of_all_elements_located((By.XPATH, '//ul[@class="jobs-search-results-list"]/li//div[@class="job-card-list__title"]/a'))
    )
    # 去重并收集链接
    for card in job_cards:
        href = card.get_attribute('href')
        if href not in jobs:
            jobs.append(href)
    
    # 滚动到底部加载新职位
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    time.sleep(2)  # 等待加载
    new_height = driver.execute_script("return document.body.scrollHeight")
    # 如果高度不再变化，说明没有更多职位
    if new_height == last_height:
        break
    last_height = new_height

# 打印结果
for idx, url in enumerate(jobs[:target_count], 1):
    print(f"{idx}: {url}")

driver.quit()

关键改进说明 #

• 使用WebDriverWait结合expected_conditions等待元素可见/存在，替代固定sleep和无意义的等待。
• 采用相对XPATH定位职位链接，依赖职位列表和卡片的类名，更稳定。
• 实现滚动加载逻辑，每次滚动到底部获取新职位，直到达到目标数量或无更多职位。
• 对链接去重，避免重复抓取同一职位。

内容的提问来源于stack exchange，提问作者user167707