如何在div内循环获取有序数据并滚动页面抓取全量数据

问题描述 #

我需要在div循环中获取数据，确保各字段值能按正确行排列，同时获取页面全部数据而非仅可见部分，具体问题如下：

1. 如何从div[@class='sc-kbGplQ bCRLdc']中提取Firm_name、Remediation_status等字段？
2. 下方代码仅能获取不到20行数据，但企业总数达1800+，如何通过滚动页面获取全量数据？

现有代码 #

ruby = driver.find_elements(By.XPATH, "//div[@class='sc-kbGplQ bCRLdc']")
for i in ruby:    
#    actions.move_to_element(i).perform()
    driver.execute_script("arguments[0].scrollIntoView();", i)
    time.sleep(INTERVAL)    
    
    try:
        Firm_name = [Firm_name.text for Firm_name in i.find_elements(By.XPATH, "//div[1]/h2[@class='sc-idjmjb jDJltL']")]        
        Remediation_status = [Remediation_status.text for Remediation_status in i.find_elements(By.XPATH, "//div[1]/span[2][@class='sc-iKpIOp iKvkEG']")]
        Safety_training = [Safety_training.text for Safety_training in i.find_elements(By.XPATH, "//div[2]/span[2][@class = 'sc-iKpIOp iKvkEG']" )]
        Worker_number = [Worker_number.text for Worker_number in i.find_elements(By.XPATH, "//div[1]/h2[@class='sc-bsVVwV gnfeLF']")]
        Progress_rate = [Progress_rate.text for Progress_rate in i.find_elements(By.XPATH, "//div[2]/h2[@class= 'sc-bsVVwV gnfeLF']")]        
    except:
        print("na")
#driver.execute_script("window.scrollBy(0,500)","")
time.sleep(INTERVAL)
df1 = pd.DataFrame(data=list(zip(Firm_name, Remediation_status, Safety_training, Progress_rate, Worker_number)), columns=['Firm_name', 'Remediation_status', 'Safety_training', 'Progress_rate', 'Worker_number'])
df1.to_csv('namefirm.csv')

一、正确提取单个企业节点内的字段 #

你的代码核心问题是：循环每个企业节点i时，XPATH用了//开头，这会从整个文档根节点搜索，而非当前节点i的子节点，导致每次循环都捞取页面所有符合条件的元素，字段对应关系完全混乱。

修改方法： #

1. 将XPATH开头的//改为.，表示从当前节点的子元素中查找
2. 每个字段对应单个企业的信息，无需用列表推导式，直接取单个元素的文本
3. 初始化一个空列表存储每条数据，后续统一生成DataFrame

修改后的循环内代码：

# 循环外先初始化数据列表
data = []

for i in ruby:    
    driver.execute_script("arguments[0].scrollIntoView();", i)
    time.sleep(INTERVAL)    
    
    try:
        # 用.//限定在当前节点i下查找子元素
        firm_name = i.find_element(By.XPATH, ".//div[1]/h2[@class='sc-idjmjb jDJltL']").text
        remediation_status = i.find_element(By.XPATH, ".//div[1]/span[2][@class='sc-iKpIOp iKvkEG']").text
        safety_training = i.find_element(By.XPATH, ".//div[2]/span[2][@class='sc-iKpIOp iKvkEG']").text
        worker_number = i.find_element(By.XPATH, ".//div[1]/h2[@class='sc-bsVVwV gnfeLF']").text
        progress_rate = i.find_element(By.XPATH, ".//div[2]/h2[@class='sc-bsVVwV gnfeLF']").text
        
        # 将单条数据存入列表
        data.append([firm_name, remediation_status, safety_training, progress_rate, worker_number])
    except Exception as e:
        print(f"提取失败: {e}")
        # 缺失数据用空值填充
        data.append(["", "", "", "", ""])

# 生成DataFrame并保存
df1 = pd.DataFrame(data, columns=['Firm_name', 'Remediation_status', 'Safety_training', 'Progress_rate', 'Worker_number'])
df1.to_csv('namefirm.csv', index=False)

二、滚动页面获取全量1800+条数据 #

原代码先获取所有节点再循环，但页面是懒加载模式（仅加载可见区域），所以初始只能拿到20条左右。正确做法是循环滚动页面直到无新数据加载，步骤如下：

1. 初始化数据列表和上次获取的节点数
2. 循环滚动到页面底部，等待新数据加载
3. 对比每次滚动后的节点数，直到数量不再变化（说明全量数据已加载）
4. 最后一次性提取所有节点的字段

示例完整代码：

import time
from selenium import webdriver
from selenium.webdriver.common.by import By
import pandas as pd

driver = webdriver.Chrome()
driver.get("你的目标页面URL")
INTERVAL = 2  # 可根据页面加载速度调整
data = []

last_count = 0
while True:
    # 滚动到页面底部
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    time.sleep(INTERVAL)
    
    # 获取当前所有企业节点
    current_nodes = driver.find_elements(By.XPATH, "//div[@class='sc-kbGplQ bCRLdc']")
    current_count = len(current_nodes)
    
    # 节点数不再增加，说明全量数据已加载
    if current_count == last_count:
        break
    last_count = current_count
    print(f"已加载{current_count}条数据")

# 提取所有节点的字段
for node in current_nodes:
    try:
        firm_name = node.find_element(By.XPATH, ".//div[1]/h2[@class='sc-idjmjb jDJltL']").text
        remediation_status = node.find_element(By.XPATH, ".//div[1]/span[2][@class='sc-iKpIOp iKvkEG']").text
        safety_training = node.find_element(By.XPATH, ".//div[2]/span[2][@class='sc-iKpIOp iKvkEG']").text
        worker_number = node.find_element(By.XPATH, ".//div[1]/h2[@class='sc-bsVVwV gnfeLF']").text
        progress_rate = node.find_element(By.XPATH, ".//div[2]/h2[@class='sc-bsVVwV gnfeLF']").text
        data.append([firm_name, remediation_status, safety_training, progress_rate, worker_number])
    except Exception as e:
        print(f"节点提取失败: {e}")
        data.append(["", "", "", "", ""])

# 保存数据
df1 = pd.DataFrame(data, columns=['Firm_name', 'Remediation_status', 'Safety_training', 'Progress_rate', 'Worker_number'])
df1.to_csv('namefirm.csv', index=False)
driver.quit()

额外优化建议 #

• 若页面有"加载更多"按钮，可改为模拟点击按钮，比滚动更可靠
• 可根据网络情况调整INTERVAL时长（网络慢设3-5秒）
• 字段定位可结合文本内容等属性，避免前端类名变动导致定位失败

内容的提问来源于stack exchange，提问作者Ruby