如何爬取带有无限滚动功能的网站？现有LinkedIn爬虫仅能抓取前25条数据

LinkedIn公开职位搜索页首次加载仅返回25条职位数据，后续数据依赖用户滚动页面触发异步接口加载，直接使用requests请求静态页面只能获取到首屏资源，因此只能拿到25条结果。

这个方案不需要启动浏览器，请求速度快，资源消耗低，是最优解。
你可以直接调用LinkedIn的职位分页接口，接口核心参数start控制分页偏移量，每次偏移25即可获取下一页数据，示例代码如下：

import requests
from bs4 import BeautifulSoup
import time
import pandas as pd

start_time = time.time()
s = requests.Session()
# 添加UA请求头避免被反爬拦截
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36"
}
s.headers.update(headers)

# 要爬取的总数量可自行调整，未登录状态下公开接口最多可获取约1000条
total_target = 100
page_size = 25
job_titles = []
job_companies = []
job_locations = []
job_links = []

for start in range(0, total_target, page_size):
    # 分页接口地址，搜索参数和你原需求保持一致
    url = f"https://www.linkedin.com/jobs-guest/jobs/api/seeMoreJobPostings/search?keywords=It%20Business%20Analyst&location=Boston%2C%20Massachusetts%2C%20United%20States&geoId=102380872&trk=public_jobs_jobs-search-bar_search-submit&position=1&pageNum=0&start={start}"
    response = s.get(url)
    # 添加延时避免触发反爬策略
    time.sleep(2)
    if response.status_code != 200:
        print(f"第{start}条起始的数据获取失败，状态码{response.status_code}")
        break
    soup = BeautifulSoup(response.text, 'html.parser')
    items = soup.find_all('div', {'class': 'base-search-card__info'})
    if not items:
        print("已获取完所有可公开访问的职位数据")
        break
    # 解析逻辑和原代码保持一致，增加空值判断避免报错
    for item in items:
        title = item.find('h3', {'class': 'base-search-card__title'}).text.strip('\n ') if item.find('h3', {'class': 'base-search-card__title'}) else ""
        company = item.find('h4', {'class': 'base-search-card__subtitle'}).text.strip('\n ') if item.find('h4', {'class': 'base-search-card__subtitle'}) else ""
        location = item.find('span', {'class': 'job-search-card__location'}).text.strip('\n ') if item.find('span', {'class': 'job-search-card__location'}) else ""
        link = item.find_parent('a', {'class': 'base-card__full-link'})["href"].strip('\n ') if item.find_parent('a', {'class': 'base-card__full-link'}) else ""
        job_titles.append(title)
        job_companies.append(company)
        job_locations.append(location)
        job_links.append(link)

# 后续统计逻辑和原代码保持一致
a = pd.DataFrame({'Job Titles': job_titles})
b = pd.DataFrame({'Job Companies': job_companies})
c = pd.DataFrame({'Job Locations': job_locations})

value_counts1 = a['Job Titles'].value_counts()
value_counts2 = b['Job Companies'].value_counts()
value_counts3 = c['Job Locations'].value_counts()

l1 = [f"{key} - {value_counts1[key]}" for key in value_counts1.keys()]
l2 = [f"{key} - {value_counts2[key]}" for key in value_counts2.keys()]
l3 = [f"{key} - {value_counts3[key]}" for key in value_counts3.keys()]

data = l1, l2, l3
df = pd.DataFrame(
    data, index=['Job Titles', 'Job Companies', 'Job Locations'])
df = df.T

print(df)
print("--- %s seconds ---" % (time.time() - start_time))

如果不想调试接口，也可以使用Selenium模拟用户滚动页面的操作，等待数据加载后再解析，适合动态渲染逻辑复杂的场景，核心实现逻辑如下：

from selenium import webdriver
from selenium.webdriver.common.by import By
import time

driver = webdriver.Chrome()
driver.get('https://www.linkedin.com/jobs/search?keywords=It%20Business%20Analyst&location=Boston%2C%20Massachusetts%2C%20United%20States&geoId=102380872&trk=public_jobs_jobs-search-bar_search-submit&position=1&pageNum=0')
# 滚动次数可根据需要爬取的数量调整
scroll_num = 5
for i in range(scroll_num):
    # 执行JS滚动到页面底部
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    time.sleep(3)
    # 部分场景滚动到底部会出现加载更多按钮，增加点击逻辑
    try:
        load_more_btn = driver.find_element(By.CSS_SELECTOR, 'button[data-tracking-control-name="infinite-scroller_show-more"]')
        load_more_btn.click()
        time.sleep(2)
    except:
        pass
# 滚动完成后拿页面源码用BeautifulSoup按原逻辑解析即可
page_source = driver.page_source

• 每次请求添加2秒左右的延时，避免请求频率过高触发反爬被封IP
• 未登录状态下LinkedIn公开职位搜索最多只能获取约1000条数据，需要更多数据需要登录账号后调用接口

内容的提问来源于stack exchange，提问作者intermarketics