如何提升Selenium爬虫运行速度？目标5-10秒完成数据爬取

问题现状 #

当前基于Selenium的爬取代码耗时60-80秒，远超出5-10秒的目标，核心原因是Selenium启动浏览器、渲染页面的开销过大，且原代码仅针对单一行数据爬取（示例场景）。以下是具体提速方案：

原代码（供参考） #

from selenium import webdriver
from selenium.webdriver.common.by import By
import pandas as pd
import time

start = time.time()
driver = webdriver.Chrome()
driver.get("https://www.investing.com/economic-calendar/")
# Scrape the data
events = []

# Locate the rows in the table
rows = driver.find_elements(By.XPATH, '/html/body/div[6]/section/div[6]/table/tbody/tr[14]')

for row in rows:
    try:
        actual = row.find_element(By.XPATH, './td[5]').text
        previous = row.find_element(By.XPATH, './td[7]').text

        events.append([actual, previous])
    except Exception as e:
        print(f"Error processing row: {e}")

driver.quit()

df = pd.DataFrame(events, columns=[ 'Actual', 'Previous'])

df.head()

df.to_csv('economic_calendar.csv', index=False)

end = time.time()

print(end - start)

方案1：改用Requests+BeautifulSoup直接爬取接口数据（最快，推荐） #

直接抓取网站加载数据的API接口，跳过浏览器渲染环节，速度可控制在5秒内。

步骤说明 #

1. 打开浏览器开发者工具（F12），切换到Network标签，刷新页面后筛选XHR请求；
2. 找到含economicCalendar关键词的请求，复制请求URL和Headers；
3. 用requests发送请求，解析JSON数据后存入CSV。

示例代码 #

import requests
import pandas as pd
import time

start = time.time()

# 替换为实际抓包得到的API URL和Headers
url = "https://www.investing.com/economic-calendar/Service/getCalendarFilteredData"
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36",
    "X-Requested-With": "XMLHttpRequest",
    "Referer": "https://www.investing.com/economic-calendar/"
}

# 构造请求参数（根据抓包结果调整，示例参数）
params = {
    "country[]": ["25", "32", "72", "22", "17", "37", "6", "39", "10", "110", "11"],
    "dateFrom": "2024-05-20",
    "dateTo": "2024-05-20",
    "timeZone": "8",
    "timeFilter": "timeRemain",
    "currentTab": "today",
    "limit_from": "0"
}

response = requests.post(url, headers=headers, data=params)
data = response.json()

# 提取所需字段（Actual和Previous）
events = []
for item in data["data"]:
    try:
        actual = item.get("actual", "")
        previous = item.get("previous", "")
        events.append({"Actual": actual, "Previous": previous})
    except Exception as e:
        print(f"处理数据出错: {e}")

df = pd.DataFrame(events)
df.to_csv('economic_calendar.csv', index=False)

end = time.time()
print(f"耗时: {end - start:.2f}秒")

方案2：优化Selenium配置（保留Selenium场景） #

若必须使用Selenium，通过以下配置减少页面加载时间：

优化点 #

• 启用无头模式（不显示浏览器窗口）；
• 禁用图片、CSS、不必要的JS加载；
• 改用相对选择器（避免绝对XPATH）。

示例代码 #

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
import pandas as pd
import time

start = time.time()

# 配置Chrome选项
chrome_options = Options()
chrome_options.add_argument("--headless=new")  # 无头模式
chrome_options.add_argument("--disable-gpu")
chrome_options.add_argument("--disable-images")  # 禁用图片加载
chrome_options.add_argument("--disable-css")  # 禁用CSS加载
chrome_options.add_argument("--disable-javascript")  # 若JS不影响数据加载可禁用
chrome_options.add_argument("--no-sandbox")
chrome_options.add_argument("--disable-dev-shm-usage")

driver = webdriver.Chrome(options=chrome_options)
driver.get("https://www.investing.com/economic-calendar/")

events = []
# 改用相对选择器，避免绝对XPATH
rows = driver.find_elements(By.CSS_SELECTOR, "#economicCalendarData tbody tr")

# 示例仅处理第14行，实际可遍历所有行
if len(rows) >=14:
    row = rows[13]  # 索引从0开始
    try:
        actual = row.find_element(By.CSS_SELECTOR, "td:nth-child(5)").text
        previous = row.find_element(By.CSS_SELECTOR, "td:nth-child(7)").text
        events.append([actual, previous])
    except Exception as e:
        print(f"处理行出错: {e}")

driver.quit()

df = pd.DataFrame(events, columns=['Actual', 'Previous'])
df.to_csv('economic_calendar.csv', index=False)

end = time.time()
print(f"耗时: {end - start:.2f}秒")

方案3：改用Playwright替代Selenium #

Playwright的页面渲染效率更高，内置等待机制更智能，速度比Selenium快2-3倍。

示例代码 #

from playwright.sync_api import sync_playwright
import pandas as pd
import time

start = time.time()

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    # 禁用图片和CSS加速加载
    page.route("**/*", lambda route: route.abort() if route.request.resource_type in ["image", "stylesheet"] else route.continue_())
    
    page.goto("https://www.investing.com/economic-calendar/")
    # 等待表格加载完成
    page.wait_for_selector("#economicCalendarData tbody tr")
    
    events = []
    # 获取第14行数据
    row = page.locator("#economicCalendarData tbody tr").nth(13)
    try:
        actual = row.locator("td:nth-child(5)").text_content()
        previous = row.locator("td:nth-child(7)").text_content()
        events.append([actual, previous])
    except Exception as e:
        print(f"处理行出错: {e}")
    
    browser.close()

df = pd.DataFrame(events, columns=['Actual', 'Previous'])
df.to_csv('economic_calendar.csv', index=False)

end = time.time()
print(f"耗时: {end - start:.2f}秒")

内容的提问来源于stack exchange，提问作者Ismoiljon Jo'rayev