如何爬取Transfermarkt中URL相同的分页表格数据？

爬取Transfermarkt网站的表格数据时遇到问题：仅能获取第一页的数据，点击第2页等分页标签后，页面显示26-50号球员，但URL并未发生变化，导致无法爬取后续页面的数据。请问如何修改现有代码以获取所有分页的表格数据？

原代码 #

import requests
from bs4 import BeautifulSoup
import pandas as pd

headers = {'User-Agent': 
           'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36'}

page = "https://www.transfermarkt.us/premier-league/transferrekorde/wettbewerb/GB1/plus/1/galerie/0?saison_id=2021&land_id=alle&ausrichtung=alle&spielerposition_id=alle&altersklasse=alle&leihe=&w_s=s&zuab=0"
pageTree = requests.get(page, headers=headers)
pageSoup = BeautifulSoup(pageTree.content, 'html.parser')



TransferPrice = pageSoup.find_all("td",{"class","rechts hauptlink"})

transfer_prices = []
cleaned_transfer_prices = []
for i in TransferPrice:
    transfer_prices.append(i.text)
for i in transfer_prices:
    i = i[1:-1]
    i = float(i)
    cleaned_transfer_prices.append(i)
cleaned_transfer_prices

some_list = []
#Players = pageSoup.find_all("td",{"class", "hauptlink"})

for td_tag in pageSoup.find_all("td",{"class", "hauptlink"}):
    a_tag = td_tag.find('a')
    if a_tag == None:
        pass
    else:
        some_list.append(a_tag.text)
players = []
team_left = []
team_gone_to = []
for i in range(0,len(some_list),3):
    players.append(some_list[i])
for i in range(1,len(some_list),3):
    team_left.append(some_list[i])
for i in range(2,len(some_list),3):
    team_gone_to.append(some_list[i])

df_2 = pd.DataFrame()
df_2['Player Name'] = players
df_2['Team Left'] = team_left
df_2['New Team'] = team_gone_to
df_2['Transfer Price'] = cleaned_transfer_prices
df_2.index+=1

df_2

问题原因 #

这种分页属于前端异步加载（AJAX）：点击分页按钮时，页面不会刷新或改变URL，而是悄悄发送HTTP请求获取对应页的数据，所以直接请求原URL只能拿到第一页内容。

修改思路 #

1. 先获取总页数：解析页面的分页栏，确定需要爬取的总页数
2. 构造分页请求URL：Transfermarkt的分页通过page参数控制，在原URL后追加&page=页码即可请求对应页面
3. 封装数据提取逻辑：把单页数据的提取代码封装成函数，减少重复代码
4. 循环爬取所有页面：依次请求每一页，合并所有数据

修改后的代码 #

import requests
from bs4 import BeautifulSoup
import pandas as pd

headers = {
    'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36',
    'X-Requested-With': 'XMLHttpRequest'  # 模拟AJAX请求标识，避免被拦截
}

base_url = "https://www.transfermarkt.us/premier-league/transferrekorde/wettbewerb/GB1/plus/1/galerie/0?saison_id=2021&land_id=alle&ausrichtung=alle&spielerposition_id=alle&altersklasse=alle&leihe=&w_s=s&zuab=0"

# 获取总页数
page_tree = requests.get(base_url, headers=headers)
page_soup = BeautifulSoup(page_tree.content, 'html.parser')
last_page_tag = page_soup.find("li", class_="paginate-item last")
total_pages = int(last_page_tag.find("a")["data-page"]) if last_page_tag else 1

# 初始化全局数据存储列表
all_players = []
all_team_left = []
all_team_gone_to = []
all_transfer_prices = []

def extract_page_data(soup):
    """提取单页的表格数据"""
    # 提取转会价格
    transfer_price_tags = soup.find_all("td", {"class": "rechts hauptlink"})
    prices = []
    for tag in transfer_price_tags:
        price_text = tag.text.strip()[1:-1]  # 去除$符号和多余空格
        prices.append(float(price_text))
    
    # 提取球员、转出/转入球队
    some_list = []
    for td_tag in soup.find_all("td", {"class": "hauptlink"}):
        a_tag = td_tag.find('a')
        if a_tag:
            some_list.append(a_tag.text.strip())
    
    players = some_list[::3]
    team_left = some_list[1::3]
    team_gone_to = some_list[2::3]
    
    return players, team_left, team_gone_to, prices

# 爬取第一页
players, team_left, team_gone_to, prices = extract_page_data(page_soup)
all_players.extend(players)
all_team_left.extend(team_left)
all_team_gone_to.extend(team_gone_to)
all_transfer_prices.extend(prices)

# 爬取后续页面
for page_num in range(2, total_pages + 1):
    page_url = f"{base_url}&page={page_num}"
    page_tree = requests.get(page_url, headers=headers)
    page_soup = BeautifulSoup(page_tree.content, 'html.parser')
    
    players, team_left, team_gone_to, prices = extract_page_data(page_soup)
    all_players.extend(players)
    all_team_left.extend(team_left)
    all_team_gone_to.extend(team_gone_to)
    all_transfer_prices.extend(prices)

# 生成最终DataFrame
df = pd.DataFrame({
    'Player Name': all_players,
    'Team Left': all_team_left,
    'New Team': all_team_gone_to,
    'Transfer Price': all_transfer_prices
})
df.index += 1

print(df)

关键改动说明 #

• 添加X-Requested-With: XMLHttpRequest请求头，模拟前端AJAX请求，避免被网站的反爬机制拦截
• 新增总页数获取逻辑，确保能爬取所有分页内容
• 封装extract_page_data函数，将单页数据提取逻辑模块化，提升代码可读性和可维护性
• 通过循环+page参数构造分页URL，批量获取所有页面数据并合并

内容的提问来源于stack exchange，提问作者Buftek