Selenium Python：如何无需等待页面完全加载即可开始抓取

云原生阿岚

2026-8-8

优化Selenium网页抓取速度：跳过无关内容，仅等待目标元素加载

核心问题

网页抓取耗时过长（预计360小时），根源是目标网站服务器速度慢，且Selenium默认会等待页面所有资源（包括广告、图片等无关内容）加载完成，而实际需要抓取的元素早已加载完毕。

最优解决方案

1. 调整页面加载策略，减少全局等待

Selenium默认的normal加载策略会等待页面完全加载完成，改为eager策略后，浏览器仅等待DOM结构加载完成（DOMContentLoaded事件触发），无需等待图片、广告等静态资源，直接缩短全局等待时间：

options = webdriver.ChromeOptions()
# 改为eager加载策略
options.page_load_strategy = 'eager'
# 保留原有配置
options.add_experimental_option('excludeSwitches', ['enable-logging'])
# 可选：禁用图片加载进一步提速
options.add_experimental_option("prefs", {"profile.managed_default_content_settings.images": 2})

2. 显式等待：仅等待目标元素加载

使用Selenium的WebDriverWait配合expected_conditions，针对每个需要抓取的元素设置精准等待，元素一加载完成就立即执行抓取，完全跳过无关内容的等待。

首先导入依赖模块：

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

然后替换原代码中直接调用browser.find_element()的逻辑，以抓取state为例：

try:
    # 最多等待10秒，直到目标元素出现在DOM中
    state_element = WebDriverWait(browser, 10).until(
        EC.presence_of_element_located((By.XPATH, '(.//span[@id = "ContentPlaceHolder2_rpParent_lblheaderCheild_0"])'))
    )
    state = state_element.text.strip()
    state = state if state else None
except:
    state = None

同理，将其他元素（city、group、sub_group等）的抓取逻辑都替换为这种显式等待方式，确保只等待需要的元素加载完成。

3. 其他优化点

避免重复请求列表页：原代码中每次循环都重新调用browser.get()打开列表页，可改为进入列表页后直接翻页，减少重复加载的时间浪费。
简化文件操作：原代码中文件读写逻辑存在冗余，可直接用Pandas写入CSV，无需中间文本文件。

修改后完整参考代码

#C:\Users\keibo\PycharmProjects\emergency ahanonline project
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from selenium import webdriver
import pandas as pd
from webdriver_manager.chrome import ChromeDriverManager
import time
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

t = time.localtime()
current_time = time.strftime("%H:%M:%S", t)
print(f'[{current_time}] Started.')

options = webdriver.ChromeOptions()
options.add_experimental_option('excludeSwitches', ['enable-logging'])
# 设置eager加载策略
options.page_load_strategy = 'eager'
# 禁用图片加载
options.add_experimental_option("prefs", {"profile.managed_default_content_settings.images": 2})
# options.add_argument("--headless")

output_data = []
browser = webdriver.Chrome(options=options,service=Service(ChromeDriverManager().install()))

def tir():
    global output_data
    browser.get('https://senf.ir/ListCompany/75483/%D8%A2%D9%87%D9%86-%D8%A2%D9%84%D8%A7%D8%AA-%D9%88-%D8%B6%D8%A7%DB%8C%D8%B9%D8%A7%D8%AA')
    # 等待分页元素加载后点击
    WebDriverWait(browser, 10).until(
        EC.element_to_be_clickable((By.ID, "ContentPlaceHolder2_rptPager_lnkPage_11"))
    ).click()
    
    # 获取总页数
    pages_element = WebDriverWait(browser, 10).until(
        EC.presence_of_element_located((By.ID, "ContentPlaceHolder2_rptPager_lnkPage_9"))
    )
    pages = int(pages_element.text)
    print(f'There are {pages} pages of 20 names which means there is {pages*20} people to save.')
    
    for page in range(pages):
        # 等待当前页的人员列表加载完成
        WebDriverWait(browser, 10).until(
            EC.presence_of_element_located((By.ID, "ContentPlaceHolder2_grdProduct_HpCompany_0"))
        )
        
        for person in range(20):
            try:
                # 点击人员链接
                WebDriverWait(browser, 10).until(
                    EC.element_to_be_clickable((By.ID, f"ContentPlaceHolder2_grdProduct_HpCompany_{person}"))
                ).click()
                
                # 抓取各项数据
                state = None
                try:
                    state_elem = WebDriverWait(browser, 10).until(
                        EC.presence_of_element_located((By.XPATH, './/span[@id = "ContentPlaceHolder2_rpParent_lblheaderCheild_0"]'))
                    )
                    state = state_elem.text.strip() or None
                except:
                    pass
                
                city = None
                try:
                    city_elem = WebDriverWait(browser, 10).until(
                        EC.presence_of_element_located((By.XPATH, './/span[@id = "ContentPlaceHolder2_rpParent_lblheaderCheild_1"]'))
                    )
                    city = city_elem.text.strip() or None
                except:
                    pass
                
                group = None
                try:
                    group_elem = WebDriverWait(browser, 10).until(
                        EC.presence_of_element_located((By.XPATH, './/span[@id = "ContentPlaceHolder2_rpParent_lblheaderCheild_2"]'))
                    )
                    group = group_elem.text.strip() or None
                except:
                    pass
                
                sub_group = None
                try:
                    sub_group_elem = WebDriverWait(browser, 10).until(
                        EC.presence_of_element_located((By.XPATH, './/span[@id = "ContentPlaceHolder2_rpParent_lblheaderCheild_3"]'))
                    )
                    sub_group = sub_group_elem.text.strip() or None
                except:
                    pass
                
                address = None
                try:
                    address_elem = WebDriverWait(browser, 10).until(
                        EC.presence_of_element_located((By.XPATH, './/span[@id = "ContentPlaceHolder2_txtAddress"]'))
                    )
                    address = address_elem.text.strip() or None
                except:
                    pass
                
                website = None
                try:
                    website_elem = WebDriverWait(browser, 10).until(
                        EC.presence_of_element_located((By.XPATH, './/a[@id = "ContentPlaceHolder2_hfWebsait"]'))
                    )
                    website = website_elem.text.strip() or None
                except:
                    pass
                
                description = None
                try:
                    desc_elem = WebDriverWait(browser, 10).until(
                        EC.presence_of_element_located((By.XPATH, './/span[@id = "ContentPlaceHolder2_lblDesc"]'))
                    )
                    description = desc_elem.text.strip() or None
                except:
                    pass
                
                views = None
                try:
                    views_elem = WebDriverWait(browser, 10).until(
                        EC.presence_of_element_located((By.XPATH, './/span[@id = "ContentPlaceHolder2_lblVisit"]'))
                    )
                    views = views_elem.text.strip() or None
                except:
                    pass
                
                # 将数据加入列表
                output_data.append({
                    'State': state,
                    'City': city,
                    'Group': group,
                    'Sub_Group': sub_group,
                    'Address': address,
                    'Website': website,
                    'Description': description,
                    'Views': views
                })
                print(f'已抓取第{page+1}页，第{person+1}条数据')
                print('--------------------------------------------')
                
                # 返回列表页
                browser.back()
            except Exception as e:
                print(f'抓取第{page+1}页第{person+1}条数据失败: {str(e)}')
                browser.back()
                continue
        
        # 翻页（最后一页无需翻页）
        if page < pages - 1:
            try:
                WebDriverWait(browser, 10).until(
                    EC.element_to_be_clickable((By.ID, "ContentPlaceHolder2_rptPager_lnkPage_12"))
                ).click()
            except:
                print('翻页失败')
                break

tir()
print("End")

# 直接用Pandas写入CSV
df = pd.DataFrame(output_data)
df.to_csv('Output.csv', index=False)

try:
    browser.close()
except Exception as e:
    print(f'关闭浏览器失败: {str(e)}')

内容的提问来源于stack exchange，提问作者Keyvan Abedini