Python网页爬虫：遍历玩家列表动态生成URL抓取CS选手数据

我有一段Python代码可以从固定URL抓取Liquipedia上CS选手的nickname、Role:、Born:数据并生成Excel，但现在想改成遍历玩家名称列表，动态拼接URL批量获取数据并整理到DataFrame里，自己尝试循环实现失败了。

原固定URL代码：

import requests
from bs4 import BeautifulSoup
import pandas as pd

headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36'}
url = 'https://liquipedia.net/counterstrike/S1mple'
soup = BeautifulSoup(requests.get(url, headers=headers).text, 'html.parser')

player_list = ['M0NESY','S1mple']

listdf = pd.DataFrame(player_list)
listdf.index = listdf.index + 1
print(listdf)

stammdaten = dict( (e.text, e.find_next_sibling('div').text) for e in soup.select('.infobox-description'))
stammdaten['nickname'] = url.split('/')[-1]

# 创建DataFrame
index = [1]
stammdatendf = pd.DataFrame(stammdaten, columns = ['nickname', 'Role:','Born:'], index=index)
print(stammdatendf)

# 导出到Excel
stammdatendf.to_excel('test.xlsx')

我想要实现的效果类似下面这段代码，但希望把urls数组替换成已有的玩家名称列表：

import requests
from bs4 import BeautifulSoup
import pandas as pd

headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36'}
urls = ['https://liquipedia.net/counterstrike/S1mple', 'https://liquipedia.net/counterstrike/M0NESY', 'https://liquipedia.net/counterstrike/Dexter']
players = []
for url in urls:
    soup = BeautifulSoup(requests.get(url, headers=headers).text, 'html.parser')
    stammdaten = dict( (e.text, e.find_next_sibling('div').text) for e in soup.select('.infobox-description'))
    stammdaten['nickname'] = url.split('/')[-1]
    players.append(stammdaten)
    # print(players)

# 创建选手DataFrame
playerdf = pd.DataFrame(players, columns =['nickname','Born:','Role:'])
print(playerdf)

直接遍历玩家名称列表，动态拼接URL即可，核心是把玩家名拼接到基础URL后。修改后的代码如下：

import requests
from bs4 import BeautifulSoup
import pandas as pd

headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36'}
# 基础URL模板，后续拼接玩家昵称生成完整地址
base_url = 'https://liquipedia.net/counterstrike/'
player_list = ['M0NESY','S1mple', 'Dexter']
players = []

for nickname in player_list:
    # 动态拼接目标URL
    url = base_url + nickname
    soup = BeautifulSoup(requests.get(url, headers=headers).text, 'html.parser')
    # 抓取选手信息字典
    stammdaten = dict( (e.text, e.find_next_sibling('div').text) for e in soup.select('.infobox-description'))
    # 直接使用遍历的昵称，无需从URL拆分
    stammdaten['nickname'] = nickname
    players.append(stammdaten)

# 整理成DataFrame并指定列顺序
playerdf = pd.DataFrame(players, columns=['nickname','Born:','Role:'])
print(playerdf)

# 导出到Excel，去掉冗余索引
playerdf.to_excel('cs_players_data.xlsx', index=False)

关键说明 #

• 定义base_url作为固定前缀，遍历玩家列表时直接拼接昵称生成完整URL，无需手动维护每个URL
• 直接用循环中的nickname赋值，替代从URL拆分的方式，代码更简洁高效
• 导出Excel时添加index=False，避免把DataFrame的索引写入表格，让导出的文件更整洁

内容的提问来源于stack exchange，提问作者quarantinho