Python Selenium爬取企业详情页数据失败问题排查

• 元素查找作用域错误：跳转至企业详情页后，你所有详情字段的定位都是基于列表页获取的item（列表行tr元素）执行的，该元素仅存在于列表页DOM结构中，详情页不存在对应节点，自然无法定位到目标数据。
• Selenium API混用问题：代码同时使用了已被Selenium 4+完全废弃的find_element_by_xpath旧接口和新版find_element(By.XPATH)接口，旧接口在高版本Selenium中会直接抛出属性不存在异常。
• 等待逻辑失效：implicitly_wait是driver全局配置项，仅需初始化driver后设置一次即可，反复调用修改等待值没有实际效果；页面跳转后未等待详情页核心元素加载完成就执行查找，极易返回空结果。
• 循环逻辑冗余且鲁棒性差：外层按列表长度做for循环、内层又遍历所有列表行，会导致重复遍历列表N次；代码未对find_elements返回的列表做空判断，直接取link[0]、对返回的元素列表直接调用send_keys，一旦元素未加载就会触发索引错误、属性错误。
• 数据提取逻辑错误：find_elements返回的是WebElement对象列表，直接打印得到的是对象内存地址，不是实际文本内容；电话、邮箱字段为<a>标签，需要提取文本或href属性才能拿到有效值。
• 页面跳转后元素失效问题：从详情页点击返回按钮回到列表页后，之前缓存的所有列表页WebElement都会失效，必须重新定位才能正常操作。
• 路径转义问题：ChromeDriver路径字符串未做转义，Windows环境下反斜杠会被识别为转义字符，导致驱动加载失败。

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import Select
import time

# 路径用原始字符串避免转义问题
PATH = r"C:\Program Files (x86)\chromedriver.exe"
driver = webdriver.Chrome(PATH)
# 全局隐式等待只设置一次
driver.implicitly_wait(10)
wait = WebDriverWait(driver, 15)

driver.get('https://www.outils.ffbatiment.fr/federation-francaise-du-batiment/laffb/annuaire.html')

# 统一用新版API做选择操作
selectOne = Select(driver.find_element(By.XPATH, '//*[@id="ctl00_ctl00_ContentPlaceHolderGlobal_ContentPlaceHolderContenu_DDL_SectAct"]'))
selectOne.select_by_value('1')
time.sleep(1)

selectTwo = Select(driver.find_element(By.XPATH, '//*[@id="ctl00_ctl00_ContentPlaceHolderGlobal_ContentPlaceHolderContenu_DDL_Act"]'))
selectTwo.select_by_value('704')
time.sleep(1)

selectThree = Select(driver.find_element(By.XPATH, '//*[@id="ctl00_ctl00_ContentPlaceHolderGlobal_ContentPlaceHolderContenu_DDL_Departement"]'))
selectThree.select_by_value('85')
time.sleep(1)

button = driver.find_element(By.XPATH, '//*[@id="ctl00_ctl00_ContentPlaceHolderGlobal_ContentPlaceHolderContenu_Button1"]')
button.click()

# 等待列表加载完成
wait.until(EC.presence_of_element_located((By.XPATH, '//*[@id="ctl00_ctl00_ContentPlaceHolderGlobal_ContentPlaceHolderContenu_RadGrid1_ctl00"]/tbody/tr')))

# 首次获取列表行
tr_list = driver.find_elements(By.XPATH, '//*[@id="ctl00_ctl00_ContentPlaceHolderGlobal_ContentPlaceHolderContenu_RadGrid1_ctl00"]/tbody/tr')
page_total = len(tr_list)

for idx in range(page_total):
    # 每次操作前重新获取当前页的所有行，避免元素失效
    tr_list = driver.find_elements(By.XPATH, '//*[@id="ctl00_ctl00_ContentPlaceHolderGlobal_ContentPlaceHolderContenu_RadGrid1_ctl00"]/tbody/tr')
    current_tr = tr_list[idx]
    # 定位当前行的企业链接
    link = current_tr.find_element(By.XPATH, './td[1]/a')
    link.send_keys(Keys.RETURN)

    # 等待详情页企业名称加载完成，确认进入详情页
    wait.until(EC.presence_of_element_located((By.XPATH, '//*[@id="ctl00_ctl00_ContentPlaceHolderGlobal_ContentPlaceHolderContenu_FormView1_Ent_NomLabel"]')))
    
    # 所有详情元素从driver根节点查找，不要用之前的列表行元素
    company = driver.find_element(By.XPATH, '//*[@id="ctl00_ctl00_ContentPlaceHolderGlobal_ContentPlaceHolderContenu_FormView1_Ent_NomLabel"]').text.strip()
    address = driver.find_element(By.XPATH, '//*[@id="ctl00_ctl00_ContentPlaceHolderGlobal_ContentPlaceHolderContenu_PanelDetails"]/div[1]/div[2]/div/div[1]/p[3]').text.strip()
    postal = driver.find_element(By.XPATH, '//*[@id="ctl00_ctl00_ContentPlaceHolderGlobal_ContentPlaceHolderContenu_FormView1_Ent_CpLabel"]').text.strip()
    city = driver.find_element(By.XPATH, '//*[@id="ctl00_ctl00_ContentPlaceHolderGlobal_ContentPlaceHolderContenu_FormView1_Ent_VilleLabel"]').text.strip()
    # 电话、邮箱取文本即可，需要链接的话可以替换成get_attribute('href')
    phone = driver.find_element(By.XPATH, '//*[@id="ctl00_ctl00_ContentPlaceHolderGlobal_ContentPlaceHolderContenu_FormView1_Ent_TelHyperLink"]').text.strip()
    email = driver.find_element(By.XPATH, '//*[@id="ctl00_ctl00_ContentPlaceHolderGlobal_ContentPlaceHolderContenu_FormView1_Ent_EmailHyperLink"]').text.strip()

    print(f"企业名称：{company}，地址：{address}，邮编：{postal}，城市：{city}，电话：{phone}，邮箱：{email}")

    # 点击返回按钮回到列表页
    back_btn = driver.find_element(By.XPATH, '//*[@id="ctl00_ctl00_ContentPlaceHolderGlobal_ContentPlaceHolderContenu_FormView1_LButtonRetour"]')
    back_btn.send_keys(Keys.RETURN)
    # 等待列表重新加载完成
    wait.until(EC.presence_of_element_located((By.XPATH, '//*[@id="ctl00_ctl00_ContentPlaceHolderGlobal_ContentPlaceHolderContenu_RadGrid1_ctl00"]/tbody/tr')))
    time.sleep(1)

time.sleep(3)
driver.quit()

内容的提问来源于stack exchange，提问作者Tanvir Hossain Dihan