如何从汽车网站提取车辆规格列表到DataFrame？附示例

问题背景 #

我在做个人项目时，需要从汽车网站爬取车辆数据，网站的车辆产品卡片HTML结构如下：

<section class="product-card-details">
    <h3 class="product-card-details__title">
Mercedes-Benz A-Class
    </h3>

    <p class="product-card-details__subtitle">
1.3 A 200 AMG LINE 5d 161 BHP | 14-DAYS MONEY BACK GUARANTEE*
    </p>

        <p class="product-card-details__attention-grabber">
***FREE 3 MONTHS WARRANTY***
        </p>

    <ul class="listing-key-specs">

            <li class="atc-type-picanto--medium">2018 (68 reg)</li>
        
            <li class="atc-type-picanto--medium">Hatchback</li>
        
            <li class="atc-type-picanto--medium">39,009 miles</li>
        
            <li class="atc-type-picanto--medium">1.3L</li>
        
            <li class="atc-type-picanto--medium">161BHP</li>
        
            <li class="atc-type-picanto--medium">Automatic</li>
        
            <li class="atc-type-picanto--medium">Petrol</li>
        
            <li class="atc-type-picanto--medium">1 owner</li>
        
            <li class="atc-type-picanto--medium">ULEZ</li>
        

    </ul>
</section>

我已经实现了标题和副标题的循环提取：

# Find Elements by Class Name. Create array of all cards
car_list = driver.find_elements(By.CLASS_NAME, "product-card-details")

titles = []
subtitles = []

for car in car_list:
    title = car.find_element(By.CLASS_NAME, "product-card-details__title").text
    subtitle = car.find_element(By.CLASS_NAME, "product-card-details__subtitle").text

但提取车辆规格时遇到问题：

• 使用XPath会获取所有车辆的规格，而非当前单辆车的：

specs = car.find_elements(By.XPATH,"//li[contains(@class, 'atc-type-picanto--medium')]")
for spec in specs:
    print(spec.get_attribute('innerHTML'))

• 使用get_attribute('innerHTML')会得到整个HTML块，无法提取单个项：

specs = car.find_element(By.CLASS_NAME, "listing-key-specs").get_attribute('innerHTML')
print(specs)

我需要生成嵌套列表格式的规格数据：

all_specs = [[car1spec1, car1spec2, ...], [car2spec1, car2spec2, ...]]

解决方案 #

方法1：使用相对路径XPath #

问题出在XPath的绝对路径写法，改用.开头的相对路径，限定在当前车辆卡片范围内查找：

all_specs = []
titles = []
subtitles = []

car_list = driver.find_elements(By.CLASS_NAME, "product-card-details")

for car in car_list:
    # 提取并清理标题、副标题文本
    title = car.find_element(By.CLASS_NAME, "product-card-details__title").text.strip()
    subtitle = car.find_element(By.CLASS_NAME, "product-card-details__subtitle").text.strip()
    titles.append(title)
    subtitles.append(subtitle)
    
    # 提取当前车辆的规格列表
    spec_elements = car.find_elements(By.XPATH, ".//li[contains(@class, 'atc-type-picanto--medium')]")
    car_specs = [spec.text.strip() for spec in spec_elements]
    all_specs.append(car_specs)

说明：//会从文档根节点开始查找，而.//表示从当前元素节点开始搜索，确保只获取当前车辆卡片下的<li>元素。

方法2：先定位规格容器再提取子元素 #

先找到当前车辆的规格列表容器，再从中遍历获取所有列表项，逻辑更直观：

all_specs = []
titles = []
subtitles = []

car_list = driver.find_elements(By.CLASS_NAME, "product-card-details")

for car in car_list:
    title = car.find_element(By.CLASS_NAME, "product-card-details__title").text.strip()
    subtitle = car.find_element(By.CLASS_NAME, "product-card-details__subtitle").text.strip()
    titles.append(title)
    subtitles.append(subtitle)
    
    # 先锁定规格容器，再提取内部的li元素
    spec_container = car.find_element(By.CLASS_NAME, "listing-key-specs")
    spec_elements = spec_container.find_elements(By.TAG_NAME, "li")
    car_specs = [spec.text.strip() for spec in spec_elements]
    all_specs.append(car_specs)

说明：通过先定位listing-key-specs容器，再查找其下的<li>元素，避免了路径范围错误的问题，同时用strip()清理文本中的多余空格和换行，让数据更规整。

两种方法最终都会生成你需要的嵌套列表格式all_specs。

内容的提问来源于stack exchange，提问作者Curious Student