使用Selenium爬取Angular数据时循环提取字段为空的解决办法

问题说明 #

手动运行Python代码可正常获取所有字段，但在循环提取数据时，Owner、邮寄地址等字段为空，其他字段不受影响且无报错，存入数据库时这些值为空。

目标HTML代码 #

<td colspan="2"><strong>Owner</strong>
                <div ng-class="{'owner-overflow' : property.ownersInfo.length > 4}">
                    <!-- ngRepeat: owner in property.ownersInfo --><div ng-repeat="owner in property.ownersInfo" class="ng-scope">
                        <div class="ng-binding">ERROL P BROWN LLC 
                            <!-- &nbsp;&nbsp; <span ng-if="owner.shortDescription != null && owner.shortDescription.length > 0">({{owner.shortDescription}})</span> -->
                        </div>
                    </div><!-- end ngRepeat: owner in property.ownersInfo -->
                </div>
            </td>    

<td colspan="2" class="pi_mailing_address"><strong>Mailing Address</strong>
                <div>
                    <span class="ng-binding">1784 NE 163 ST </span>
                    <span ng-show="property.mailingAddress.address2" class="ng-binding ng-hide"></span>
                    <span ng-show="property.mailingAddress.address3" class="ng-binding ng-hide"></span>
                    <span ng-show="property.mailingAddress.city" ng-class="{'inline':property.mailingAddress.city}" class="ng-binding inline">NORTH MIAMI,</span>
                    <span class="inline ng-binding">FL</span>
                    <span class="inline ng-binding">33162</span>
                    <span ng-hide="isCountryUSA(property.mailingAddress.country)" class="ng-binding ng-hide">USA</span>
                </div>
            </td>

原Python代码 #

Owner = driver.find_element(By.XPATH, "//strong[text()='Owner']//following::div[1]").text
SubDivision = driver.find_element(By.XPATH, "//strong[text()='Sub-Division:']//following::div[1]").text
Address1 = driver.find_element(By.XPATH, "//strong[text()='Mailing Address']//following::div[1]//following::span[1]").text
Address2 = driver.find_element(By.XPATH, "//strong[text()='Mailing Address']//following::div[1]//following::span[2]").text
Address3 = driver.find_element(By.XPATH, "//strong[text()='Mailing Address']//following::div[1]//following::span[3]").text
city = driver.find_element(By.XPATH, "//strong[text()='Mailing Address']//following::div[1]//following::span[4]").text.replace(",", "")
state = driver.find_element(By.XPATH, "//strong[text()='Mailing Address']//following::div[1]//following::span[5]").text
zipcode = driver.find_element(By.XPATH, "//strong[text()='Mailing Address']//following::div[1]//following::span[6]").text

解决措施 #

1. 添加显式等待，确保元素加载完成 #

循环时页面渲染速度可能变慢，直接提取会拿到未加载完成的空元素。改用显式等待，等待元素可见后再提取：

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

# 提取Owner
owner_elem = WebDriverWait(driver, 10).until(
    EC.visibility_of_element_located((By.XPATH, "//strong[text()='Owner']/following-sibling::div//div[@class='ng-binding']"))
)
Owner = owner_elem.text

# 提取邮寄地址示例（Address1）
address1_elem = WebDriverWait(driver, 10).until(
    EC.visibility_of_element_located((By.XPATH, "//td[@class='pi_mailing_address']//div/span[1]"))
)
Address1 = address1_elem.text

2. 优化XPATH定位，避免依赖元素索引 #

原XPATH用following::span[n]依赖元素顺序，循环时页面元素的显示状态变化会导致定位错误。改为根据元素属性或父子关系定位：

• Owner定位：//strong[text()='Owner']/following-sibling::div//div[@class='ng-binding']
• 城市定位：//td[@class='pi_mailing_address']//span[@ng-show='property.mailingAddress.city']
• 州定位：//td[@class='pi_mailing_address']//span[@class='inline ng-binding'][2]

3. 处理动态隐藏的字段 #

邮寄地址中的Address2、Address3是ng-hide状态，直接提取会得到空值，用try-except捕获异常，为空时赋值空字符串：

try:
    address2_elem = WebDriverWait(driver, 5).until(
        EC.visibility_of_element_located((By.XPATH, "//td[@class='pi_mailing_address']//span[@ng-show='property.mailingAddress.address2']"))
    )
    Address2 = address2_elem.text
except:
    Address2 = ""

try:
    address3_elem = WebDriverWait(driver, 5).until(
        EC.visibility_of_element_located((By.XPATH, "//td[@class='pi_mailing_address']//span[@ng-show='property.mailingAddress.address3']"))
    )
    Address3 = address3_elem.text
except:
    Address3 = ""

4. 循环跳转后确保页面完全加载 #

如果循环涉及页面跳转，每次跳转后等待页面加载完成：

WebDriverWait(driver, 15).until(
    lambda d: d.execute_script('return document.readyState') == 'complete'
)

内容的提问来源于stack exchange，提问作者Leo Torres