现有Selenium爬取员工姓名邮箱代码漏爬，求更优方案及排查方法

使用Selenium爬取Brick Township High School官网的员工姓名及邮箱信息时，现有代码仅返回少量数据，大量数据漏爬。尝试更换定位器后问题仍存在，需排查代码问题并获取更优爬取方案。

附原有代码：

################

#   Brick Township High School

# Tuesday, February 7, 2023

################

# imports
import json
from turtle import pd
from seleniumwire import webdriver
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.by import By

from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

from selenium.common import NoSuchElementException
import pandas

# Driver & url
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()))
url = "https://www.brickschools.org/bths/home-default/departments/#1624628243020-4e557b29-3c1a"  # 1494
driver.get(url)
driver.maximize_window()
all_contacts = {}
# contacts = []
fileName_csv = "Brick_teachers.csv"
fileName_json = "Brick_teachers.json"

total_pages = 1

print(f"Total pages as per the website : {total_pages}")
startingPage = 1
endPage = total_pages + 1

# Function to extract info within each contact card

def extract_fields(subject, order, m):
    myXpath = f'/html/body/div[3]/div/div/div/div/div/div/div/div/div/div/div[{subject}]/div[2]/div/div/table/tbody/tr[{order}]/td[{m}]'

    if m == 2:
        myXpath = f'/html/body/div[3]/div/div/div/div/div/div/div/div/div/div/div[{subject}]/div[2]/div/div/table/tbody/tr[{order}]/td[{m}]/a'

    try:
        fieldName = driver.find_element(
            By.XPATH, myXpath).text
        print("found it", fieldName, ".....", "subject..",
              subject, "order..", order, "m..", m)
    except:
        fieldName = "NA"
    return fieldName



new_contact = {}
for subject in range(1, 13):
    for order in range(2, 20):
        staffName = extract_fields(subject, order, 1)
        staffEmail = extract_fields(subject, order, 2)

        # assign the values to the dictionary
        new_contact['staffName'] = staffName
        new_contact['staffEmail'] = staffEmail

        # add the page number to the dictionary
        new_contact['pageNumber'] = subject
        all_contacts[staffEmail] = new_contact
        new_contact = {}


#### done with data extraction ####

total = len(all_contacts)
print(
    f"We're done with all {total_pages} pages and there are {total} contacts")
# convert the all_contacts to a json file
with open(fileName_json, 'w') as file:
    json.dump(all_contacts, file)

# Step 4 - close the driver
driver.quit()
data = all_contacts

# Convert JSON File to CSV File
# pandas read JSON File
df = pandas.read_json(fileName_json)
df_transposed = df.T
df_transposed.to_csv(fileName_csv)

print(
    "Done! -- with all the pages and JSON & csv files are created")

1. 硬编码的定位范围与路径：代码用range(1,13)和range(2,20)硬编码部门数量和员工行数，实际页面各部门员工数量不一致，且绝对XPATH依赖固定页面结构，一旦元素位置变动就会失效，导致大量元素被判定为"NA"。
2. 字典键去重错误：使用staffEmail作为all_contacts的键，当多个员工无邮箱（值为"NA"）时，后出现的记录会覆盖前一个，直接丢失数据。
3. 无页面加载等待：页面未完全渲染就开始爬取，部分元素尚未加载，导致返回"NA"。
4. 无效数据未过滤：即使姓名为"NA"（对应行不存在），仍会将该记录加入字典，造成无效数据占比高、有效数据被覆盖。

改用相对定位遍历元素，用列表存储数据，添加显式等待，过滤无效数据：

import json
import pandas as pd
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

# 初始化驱动与页面
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()))
url = "https://www.brickschools.org/bths/home-default/departments/#1624628243020-4e557b29-3c1a"
driver.get(url)
driver.maximize_window()

# 显式等待页面加载完成，等待部门容器出现
wait = WebDriverWait(driver, 10)
wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, "div.department-container")))

all_contacts = []
fileName_csv = "Brick_teachers.csv"
fileName_json = "Brick_teachers.json"

# 获取所有部门容器
departments = driver.find_elements(By.CSS_SELECTOR, "div.department-container")

for dept_idx, dept in enumerate(departments, 1):
    # 查找当前部门下的员工表格
    try:
        table = dept.find_element(By.CSS_SELECTOR, "table.staff-table")
        rows = table.find_elements(By.TAG_NAME, "tr")[1:]  # 跳过表头行
    except:
        # 无员工表格的部门直接跳过
        continue

    for row in rows:
        try:
            # 提取姓名和邮箱
            staff_name = row.find_element(By.TAG_NAME, "td").text.strip()
            email_elem = row.find_element(By.CSS_SELECTOR, "td a")
            staff_email = email_elem.get_attribute("href").replace("mailto:", "").strip() if email_elem else "NA"
            
            if staff_name:  # 过滤空姓名的无效行
                all_contacts.append({
                    "staffName": staff_name,
                    "staffEmail": staff_email,
                    "departmentIndex": dept_idx
                })
                print(f"抓取到: {staff_name} - {staff_email}")
        except:
            # 跳过解析失败的行
            continue

# 数据导出
print(f"共抓取到 {len(all_contacts)} 条有效数据")

# 保存JSON
with open(fileName_json, 'w', encoding='utf-8') as f:
    json.dump(all_contacts, f, indent=2, ensure_ascii=False)

# 保存CSV
df = pd.DataFrame(all_contacts)
df.to_csv(fileName_csv, index=False, encoding='utf-8')

driver.quit()
print("数据导出完成")

• 相对定位：通过CSS选择器定位部门容器和员工表格，不依赖固定元素位置，适配页面结构变化。
• 列表存储：用列表存储所有记录，避免邮箱重复导致的数据覆盖。
• 显式等待：确保页面元素加载完成后再爬取，减少元素未找到的情况。
• 无效数据过滤：跳过无姓名的行和无员工表格的部门，只保留有效数据。
• 邮箱提取优化：直接获取mailto:链接中的邮箱地址，避免仅提取文本可能出现的错误。

内容的提问来源于stack exchange，提问作者sas70