BeautifulSoup+Selenium动态爬取NBA球员名单重复问题求助

我尝试从NBA官网球员页面爬取前2页共100名球员姓名（每页展示50名），结合Selenium与BeautifulSoup编写循环爬取代码后，结果重复爬取了第一页数据，尽管Selenium已完成页面跳转。

我的代码如下：

url = 'https://www.nba.com/players'
driver.get(url)
page = requests.get(url)
print(page)
soup = BeautifulSoup(page.text, 'html.parser')
print(soup)

table = soup.table
table = soup.find(class_='players-list')
names[]
for pages in range(2):
    for names in table.find_all('tbody'):
        name_rows = names.find_all('tr')
        for name_row in name_rows:
            player_names = name_row.find('td', class_ = 'primary text RosterRow_primaryCol__1lto4').text
    names.append(player_names)

输出结果重复包含第一页球员姓名：

['PreciousAchiuwa',
 'StevenAdams',
 'BamAdebayo',
 'OchaiAgbaji',
 'SantiAldama',
 'NickeilAlexander-Walker',
 'GraysonAllen',
 'JarrettAllen',
 'JoseAlvarado',
 'KyleAnderson',
 'GiannisAntetokounmpo',
 'KostasAntetokounmpo',
 'ThanasisAntetokounmpo',
 'ColeAnthony',
 'O.G.Anunoby',
 'RyanArcidiacono',
 'DeniAvdija',
 'DeandreAyton',
 'UdokaAzubuike',
 'IbouBadji',
 'MarvinBagley III',
 'PatrickBaldwin Jr.',
 'LaMeloBall',
 'LonzoBall',
 'MoBamba',
 'PaoloBanchero',
 'DesmondBane',
 'DalanoBanton',
 'DominickBarlow',
 'HarrisonBarnes',
 'ScottieBarnes',
 'RJBarrett',
 'WillBarton',
 'CharlesBassey',
 'KeitaBates-Diop',
 'NicolasBatum',
 'DariusBazley',
 'BradleyBeal',
 'MalikBeasley',
 'MarJonBeauchamp',
 'DavisBertans',
 'PatrickBeverley',
 'SaddiqBey',
 'KhemBirch',
 'GogaBitadze',
 'BismackBiyombo',
 'BuddyBoeheim',
 'BogdanBogdanovic',
 'BojanBogdanovic',
 'BolBol',
 'PreciousAchiuwa',
 'StevenAdams',
 'BamAdebayo',
 'OchaiAgbaji',
 'SantiAldama',
 'NickeilAlexander-Walker',
 'GraysonAllen',
 'JarrettAllen',
 'JoseAlvarado',
 'KyleAnderson',
 'GiannisAntetokounmpo',
 'KostasAntetokounmpo',
 'ThanasisAntetokounmpo',
 'ColeAnthony',
 'O.G.Anunoby',
 'RyanArcidiacono',
 'DeniAvdija',
 'DeandreAyton',
 'UdokaAzubuike',
 'IbouBadji',
 'MarvinBagley III',
 'PatrickBaldwin Jr.',
 'LaMeloBall',
 'LonzoBall',
 'MoBamba',
 'PaoloBanchero',
 'DesmondBane',
 'DalanoBanton',
 'DominickBarlow',
 'HarrisonBarnes',
 'ScottieBarnes',
 'RJBarrett',
 'WillBarton',
 'CharlesBassey',
 'KeitaBates-Diop',
 'NicolasBatum',
 'DariusBazley',
 'BradleyBeal',
 'MalikBeasley',
 'MarJonBeauchamp',
 'DavisBertans',
 'PatrickBeverley',
 'SaddiqBey',
 'KhemBirch',
 'GogaBitadze',
 'BismackBiyombo',
 'BuddyBoeheim',
 'BogdanBogdanovic',
 'BojanBogdanovic',
 'BolBol'
]

1. 静态请求与动态页面脱节：requests.get(url)获取的是初始页面的静态内容，和Selenium的浏览器会话完全独立。后续Selenium跳转页面后，没有重新获取当前页面的HTML，始终使用第一次解析的soup对象，导致每次循环都处理第一页数据。
2. 变量命名冲突：names既是存储结果的列表，又是遍历tbody的循环变量，覆盖了列表引用，导致逻辑混乱。
3. 循环逻辑错误：未在每一页处理完成后重新抓取当前页面HTML并解析，也没有添加页面跳转后的等待逻辑。

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from bs4 import BeautifulSoup

url = 'https://www.nba.com/players'
driver = webdriver.Chrome()  # 根据你的浏览器选择对应驱动
driver.get(url)

player_names_list = []

for _ in range(2):
    # 等待页面加载完成，确保球员列表渲染完毕
    WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.CLASS_NAME, 'players-list'))
    )
    # 获取当前页面的HTML内容
    page_source = driver.page_source
    soup = BeautifulSoup(page_source, 'html.parser')
    
    # 解析当前页的球员姓名
    table = soup.find(class_='players-list')
    tbody = table.find('tbody')
    name_rows = tbody.find_all('tr')
    for row in name_rows:
        name_td = row.find('td', class_='primary text RosterRow_primaryCol__1lto4')
        if name_td:
            player_name = name_td.text.strip()
            player_names_list.append(player_name)
    
    # 如果不是最后一页，点击下一页按钮
    if _ < 1:
        try:
            next_button = WebDriverWait(driver, 10).until(
                EC.element_to_be_clickable((By.CLASS_NAME, 'Pagination_button__sqGoX'))
            )
            next_button.click()
            # 等待页面跳转完成
            WebDriverWait(driver, 10).until(
                EC.staleness_of(table)
            )
        except Exception as e:
            print(f"跳转下一页失败: {e}")
            break

driver.quit()
print(player_names_list)

• 每次循环从Selenium当前页面获取page_source，确保解析的是当前页最新内容。
• 用player_names_list专门存储结果，避免变量名冲突。
• 添加显式等待，确保页面元素加载完成后再解析和点击，避免动态加载导致的元素找不到问题。
• 完善页面跳转逻辑，确保第一页解析完成后成功跳转到第二页。

内容的提问来源于stack exchange，提问作者novicecoder131