使用Python在PyCharm爬取网页，导出CSV为空的问题排查

问题场景 #

使用Python在PyCharm环境中通过Selenium爬取中原地产愉景新城的交易数据页面，期望导出房产交易数据到CSV文件，但最终生成的文件为空。原代码如下：

import time

import pandas as pd
from selenium import webdriver
from selenium.webdriver import Chrome
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from webdriver_manager.chrome import ChromeDriverManager

options = webdriver.ChromeOptions()
options.headless = True

options.page_load_strategy = 'none'

chrome_path = ChromeDriverManager().install()
chrome_service = Service(chrome_path)

driver = Chrome(options=options, service=chrome_service)
driver.implicitly_wait(5)

url= "https://hk.centanet.com/findproperty/list/transaction/%E6%84%89%E6%99%AF%E6%96%B0%E5%9F%8E_3-DMHSZHHRHD?q=TiDxvVGMUUeutVzA0g1JlQ"

driver.get(url)
time.sleep(10)

contents = driver.find_element(By.CSS_SELECTOR,"div[class*='bx--structured-list-tbody']")

properties = contents.find_elements(By.TAG_NAME,"data-v-af617cf2")

def extract_data(element):
    Date = element.find_elements(By.CSS_SELECTOR, "div[class*='infodate']>span")
    Dev = element.find_elements(By.XPATH, "//*[text() = '愉景新城'")
    Price = element.find_elements(By.CSS_SELECTOR, "div[class*='tranPrice']>span")
    RiseBox = element.find_elements(By.CSS_SELECTOR, "div[class*='riseBox']")
    Area = element.find_elements(By.XPATH, "//*[text() = '呎'")

    return{
        "Date": Date,
        "Development": Dev,
        "Consideration": Price,
        "Change": RiseBox,
        "Area": Area
    }

data = []

for property in properties:
    extracted_data = extract_data(property)
    data.append(extracted_data)

df = pd.DataFrame(data)
df.to_csv("result.csv", index=False)

错误分析 #

1. 房产条目定位完全错误
   properties = contents.find_elements(By.TAG_NAME,"data-v-af617cf2")中的data-v-af617cf2是Vue框架生成的作用域属性，并非HTML标签名，这条语句无法找到任何元素，导致properties为空列表，后续循环不执行，最终CSV无数据。

2. 数据提取逻辑错误

   • 所有find_elements调用返回的是元素对象列表，不是文本内容，直接存入字典会导致DataFrame无法识别有效数据；且每个交易条目对应单个数据，应使用find_element（单数）而非find_elements。
   • XPath语法错误：//*[text() = '愉景新城'和//*[text() = '呎'都缺少闭合括号，属于无效语法。
   • 全局XPath问题：使用//*会在整个页面搜索元素，而非当前交易条目下，导致提取的数据不对应；应使用相对路径.//限定搜索范围。

3. 页面加载策略不可靠
   options.page_load_strategy = 'none'会跳过页面加载完成的判断，硬等待time.sleep(10)不稳定，建议用显式等待确保目标元素加载完成。

修正后的代码 #

import pandas as pd
from selenium import webdriver
from selenium.webdriver import Chrome
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from webdriver_manager.chrome import ChromeDriverManager

options = webdriver.ChromeOptions()
options.headless = True
# 改为正常加载策略，配合显式等待更可靠
options.page_load_strategy = 'normal'

chrome_path = ChromeDriverManager().install()
chrome_service = Service(chrome_path)

driver = Chrome(options=options, service=chrome_service)
driver.implicitly_wait(10)

url= "https://hk.centanet.com/findproperty/list/transaction/%E6%84%89%E6%99%AF%E6%96%B0%E5%9F%8E_3-DMHSZHHRHD?q=TiDxvVGMUUeutVzA0g1JlQ"

driver.get(url)

# 显式等待交易列表加载完成
wait = WebDriverWait(driver, 15)
contents = wait.until(EC.presence_of_element_located((By.CSS_SELECTOR,"div[class*='bx--structured-list-tbody']")))

# 定位每个交易条目（页面中实际条目类为bx--structured-list-row）
properties = contents.find_elements(By.CSS_SELECTOR, "div[class*='bx--structured-list-row']")

def extract_data(element):
    # 提取日期文本
    date_elem = element.find_element(By.CSS_SELECTOR, "div[class*='infodate']>span")
    date = date_elem.text.strip() if date_elem else ""
    
    # 提取小区名称（当前条目下的小区元素）
    dev_elem = element.find_element(By.CSS_SELECTOR, "div[class*='info']>div:first-child")
    dev = dev_elem.text.strip() if dev_elem else ""
    
    # 提取成交价
    price_elem = element.find_element(By.CSS_SELECTOR, "div[class*='tranPrice']>span")
    price = price_elem.text.strip() if price_elem else ""
    
    # 提取涨跌幅度
    rise_elem = element.find_element(By.CSS_SELECTOR, "div[class*='riseBox']")
    rise = rise_elem.text.strip() if rise_elem else ""
    
    # 提取面积（包含呎的文本）
    area_elem = element.find_element(By.XPATH, ".//*[contains(text(), '呎')]")
    area = area_elem.text.strip() if area_elem else ""

    return{
        "Date": date,
        "Development": dev,
        "Consideration": price,
        "Change": rise,
        "Area": area
    }

data = []

for prop in properties:
    extracted_data = extract_data(prop)
    data.append(extracted_data)

df = pd.DataFrame(data)
# 编码设置确保中文正常显示
df.to_csv("result.csv", index=False, encoding='utf-8-sig')
driver.quit()

关键修改说明 #

• 修正交易条目定位方式，使用实际CSS类bx--structured-list-row获取每个交易项。
• 所有数据提取改为获取元素的text内容，处理可能的空元素情况。
• 将全局XPath改为相对路径.//，确保只在当前交易条目下搜索。
• 使用显式等待替代硬等待，提高页面加载可靠性。
• 添加encoding='utf-8-sig'确保CSV中中文正常显示。

内容的提问来源于stack exchange，提问作者YPT