如何用Python抓取网页下拉菜单中的车企与车型信息

需求说明 #

• 目标：抓取www.autocar.co.uk页面中下拉菜单的车企及对应车型
• 规则：跳过“All models”选项，若车企无对应车型则标注#Skip due to no model
• 期望输出格式：

Auto OEM (e.g, Tesla)
   All Models of Tesla (e.g. Model 3, Y...) 

Example:
    Abarth
       595
       595 Competi...
       124 Spider...
       695 Bopisto...
    AC Cars
       #Skip due to no model 

现有代码 #

from bs4 import BeautifulSoup
import requests

#Inputs/URLs to scrape: 
URL = ('https://www.autocar.co.uk/car-review/tesla')
(response := requests.get(URL)).raise_for_status()
soup = BeautifulSoup(response.text, 'lxml')
overview = soup.find()

oems = soup.select('select.car-finder-make-chooser option')                      
for oem_loop in oems[1:]:
    print(oem_loop.text)
    models = soup.select('select.car-finder-model-chooser option')  
    for model_loop in models:
        print(model_loop)

当前输出 #

Abarth
<option value="0">Model</option>
AC Cars
<option value="0">Model</option>
AC Schnitzer
<option value="0">Model</option>
Aiways
<option value="0">Model</option>
Allard
<option value="0">Model</option>
Alfa Romeo
<option value="0">Model</option>
Alpina
<option value="0">Model</option>
Alpine
<option value="0">Model</option>
Ariel
<option value="0">Model</option>
Ascari
<option value="0">Model</option>
Aston Martin
<option value="0">Model</option>
Audi
<option value="0">Model</option>
BAC
<option value="0">Model</option>
Bentley
<option value="0">Model</option>
Bizzarrini
<option value="0">Model</option>
BMW
<option value="0">Model</option>
Borgward
<option value="0">Model</option>
Bowler
<option value="0">Model</option>
Bugatti
<option value="0">Model</option>
BYD
<option value="0">Model</option>
...

问题分析及修正方案 #

问题原因 #

1. 页面初始仅加载默认车企的车型列表，切换车企需触发AJAX请求，静态解析无法获取对应车型
2. 直接打印model_loop对象而非文本内容，导致输出HTML标签
3. 未处理无车型的场景

修正代码 #

from bs4 import BeautifulSoup
import requests

# 基础URL，用于拼接接口请求
BASE_URL = "https://www.autocar.co.uk"
# 车企列表所在页面
MAKE_URL = "https://www.autocar.co.uk/car-review"

# 获取所有车企选项
response = requests.get(MAKE_URL)
response.raise_for_status()
soup = BeautifulSoup(response.text, 'lxml')
oem_options = soup.select('select.car-finder-make-chooser option')[1:]  # 跳过第一个默认选项

for oem in oem_options:
    oem_name = oem.text.strip()
    oem_value = oem.get('value')
    print(f"    {oem_name}")
    
    # 调用AJAX接口获取对应车企的车型数据
    model_api_url = f"{BASE_URL}/api/car-finder/models?make={oem_value}"
    model_response = requests.get(model_api_url)
    model_response.raise_for_status()
    model_data = model_response.json()
    
    # 遍历车型并过滤规则
    has_valid_models = False
    for model in model_data:
        model_name = model['name']
        if model_name.lower() == 'all models':
            continue  # 跳过All models选项
        print(f"       {model_name}")
        has_valid_models = True
    
    if not has_valid_models:
        print("       #Skip due to no model")

关键说明 #

1. 从主页面获取车企的value属性，这是调用接口的关键参数
2. 利用网站内置的AJAX接口动态获取对应车企的车型数据，这是页面切换车企时实际触发的请求
3. 解析JSON格式的返回数据，跳过指定选项
4. 增加无有效车型时的标注逻辑，匹配期望输出格式

内容的提问来源于stack exchange，提问作者JAA