Selenium遍历NBA官网下拉菜单时出现list index out of range报错如何解决？

错误根因 #

• 隐式等待的作用是全局设置DOM元素查找的最长等待时间，仅需初始化浏览器后声明一次，放在循环中不会生效
• 切换赛季选项后页面表格会先销毁旧数据再渲染新数据，隐式等待无法识别「表格内容更新完成」的状态，会提前拿到还没渲染完表格的页面源码，导致findAll返回的table列表长度不足，取下标[1]时触发越界报错

解决方案 #

替换隐式等待为显式等待，每次切换赛季后主动等待目标表格加载完成再解析页面，需要先导入显式等待依赖模块：

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver import Chrome

修改后的完整代码如下：

from selenium.webdriver.support.ui import Select
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver import Chrome
from bs4 import BeautifulSoup
import pandas as pd

player_id = str(1629216)
    
url = 'https://www.nba.com/stats/player/' + player_id + "/shooting/"
browser = Chrome(executable_path='/usr/local/bin/chromedriver')
# 隐式等待仅需全局设置一次，也可删除完全用显式等待替代
browser.implicitly_wait(3)
browser.get(url)

select = Select(browser.find_element_by_xpath('/html/body/main/div/div/div/div[4]/div/div/div/div/div[1]/div[1]/div/div/label/select'))
options = select.options
# 显式等待最长等待时间设置为10秒，可根据网络情况调整
wait = WebDriverWait(browser, 10)

for index in range(0, len(options)):
    select.select_by_index(index)
    # 显式等待：直到页面加载出至少2个class为nba-stat-table__overflow的元素再继续
    wait.until(
        lambda x: len(x.find_elements(By.CLASS_NAME, "nba-stat-table__overflow")) >=2
    )
    # 可选：额外等待页面渲染完成，避免偶发性的旧数据被读取
    browser.execute_script("return document.readyState == 'complete'")

    src = browser.page_source
    parser = BeautifulSoup(src, "lxml") 
    table = parser.findAll("div", attrs = {"class":"nba-stat-table__overflow"})

    headers = table[1].findAll('th')
    headerlist = [h.text.strip() for h in headers[1:]]
    headerlist = [a for a in headerlist if not '\n' in a]
    headerlist.append('AST%')
    headerlist.append('UAST%')

    row_labels = table[1].findAll("td", {"class": "first"})
    row_labels_list = [r.text.strip() for r in row_labels[0:]]

    rows = table[1].findAll('tr')[1:]
    player_stats = [[td.getText().strip() for td in rows[i].findAll('td')[1:]] for i in range(len(rows))]
    
    df = pd.DataFrame(data=player_stats, columns=headerlist, index = row_labels_list)

    print(df)

browser.quit()

补充说明 #

• 显式等待会每间隔500ms检查一次条件是否满足，满足就立刻继续执行，比固定时长的隐式等待效率更高，也更适配动态渲染的页面场景
• 如果遇到偶发的表格内容为空的情况，可以在等待条件中额外增加表格行数量的判断，确保数据已经渲染完成

内容的提问来源于stack exchange，提问作者steezebutter