Selenium循环多URL爬取遇索引错误，无法完成循环

我尝试用Selenium循环遍历一组名人名言页面URL，爬取内容后生成对应词云PNG，但循环仅完成第一次迭代就终止，报错索引越界。

原代码 #

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

from WordCloudCreator import WordCloudGeneration

service = Service()
options = webdriver.ChromeOptions()
browser = webdriver.Chrome(service=service, options=options)

wait = WebDriverWait(browser, 5)
base_url_1, base_url_2 = 'https://www.brainyquote.com/authors/', '-quotes'
figures = {1: 'george-washington', 2: 'thomas-jefferson', 3: 'abraham-lincoln', 4: 'john-kennedy', 5: 'george-h-w-bush', 6: 'barack-obama', 7: 'donald-trump', 8: 'joe-biden', 9: 'martin-luther-king-jr'}
urls = [base_url_1+figure+base_url_2 for figure in list(figures.values())]


def wordCouldCreator(input_text, figure_name):
    wordcloud_generator = WordCloudGeneration()
    wordcloud_generator.create_word_cloud(input_text, figure_name)


for fitureId, figure, url in zip(list(figures.keys()), list(figures.values()), urls):
    browser.get(url)
    window_before = browser.window_handles[fitureId-1]
    elementExistence = EC.presence_of_all_elements_located((By.XPATH, '//a[@title="view quote"]'))
    quotes = [x.text for x in wait.until(elementExistence) if len(x.text) > 5]
    figureQuote = ' '.join(quotes)
    wordCouldCreator(figureQuote, fitureId)
    window_after = browser.window_handles[fitureId]
    browser.switch_to.window(window_after)
    browser.switch_to.window(window_before)

报错信息 #

[nltk_data] Downloading package stopwords to
[nltk_data]     C:\Users\username\AppData\Roaming\nltk_data...
[nltk_data]   Package stopwords is already up-to-date!
[nltk_data] Downloading package punkt to
[nltk_data]     C:\Users\username\AppData\Roaming\nltk_data...
[nltk_data]   Package punkt is already up-to-date!

DevTools listening on ws://******/devtools/browser/******
Traceback (most recent call last):
  File "c:\Users\username\OneDrive\Desktop\Projects\WebScraper\WebScraper.py", line 56, in <module>
    window_after = browser.window_handles[fitureId]
                   ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
IndexError: list index out of range

解决方法 #

问题根源

报错出在window_after = browser.window_handles[fitureId]：

• 你用browser.get(url)是在当前窗口加载新页面，并没有打开新窗口，所以browser.window_handles始终只有1个元素（索引为0）。
• 第一次循环时fitureId=1，尝试取索引1的元素自然会触发索引越界错误。
• 整个代码里的窗口切换逻辑完全多余，因为你根本没有打开多窗口的操作。

修改后的代码

删除所有不必要的窗口切换代码，直接在当前窗口循环访问每个URL即可：

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

from WordCloudCreator import WordCloudGeneration

service = Service()
options = webdriver.ChromeOptions()
browser = webdriver.Chrome(service=service, options=options)

wait = WebDriverWait(browser, 5)
base_url_1, base_url_2 = 'https://www.brainyquote.com/authors/', '-quotes'
figures = {1: 'george-washington', 2: 'thomas-jefferson', 3: 'abraham-lincoln', 4: 'john-kennedy', 5: 'george-h-w-bush', 6: 'barack-obama', 7: 'donald-trump', 8: 'joe-biden', 9: 'martin-luther-king-jr'}


def wordCouldCreator(input_text, figure_name):
    wordcloud_generator = WordCloudGeneration()
    wordcloud_generator.create_word_cloud(input_text, figure_name)


# 简化循环逻辑，去掉多余的窗口操作
for fitureId, figure in figures.items():
    url = base_url_1 + figure + base_url_2
    browser.get(url)
    # 等待名言元素加载完成
    elementExistence = EC.presence_of_all_elements_located((By.XPATH, '//a[@title="view quote"]'))
    quotes = [x.text for x in wait.until(elementExistence) if len(x.text) > 5]
    figureQuote = ' '.join(quotes)
    wordCouldCreator(figureQuote, fitureId)

# 循环结束后关闭浏览器，避免资源泄漏
browser.quit()

额外优化说明

• 用figures.items()直接遍历键值对，省去构造urls列表和zip多列表的冗余操作，代码更简洁。
• 加入browser.quit()确保循环结束后关闭浏览器，释放系统资源。

内容的提问来源于stack exchange，提问作者Rebel