如何通过Python网页爬取从二级下拉菜单获取车型URL

问题背景 #

我需要爬取www.autocar.co.uk网站，通过下拉菜单遍历所有汽车厂商及对应车型，获取车型页面的链接，后续还要从这些页面提取信息（暂未实现）。目前代码输出存在多余的https://www.autocar.co.uk0无效链接，需要移除，同时希望得到代码改进建议。

当前代码 #

from bs4 import BeautifulSoup
import requests
import pandas as pd

#Inputs/URLs to scrape: 
url = "http://www.autocar.co.uk/"
s = requests.Session()

r = s.get(url)
soup = BeautifulSoup(r.text,'html.parser')
full_car_list = []

car_list = [(x.text, x.get("value"), f'https://www.autocar.co.uk/ajax/car-models/{x.get("value")}/0') for x in soup.select_one('#edit-make').select('option')]
for x in car_list:
    r = s.get(x[2])
    try:
        for item in r.json()['options'].items():
            #Car Model
            car_model_url = (f'https://www.autocar.co.uk{item[0]}')
            print(car_model_url)
            
    except Exception as e:
        full_car_list.append((x[0], 'no models', f'https://www.autocar.co.uk/vehicles/{x[0]}'))

当前输出 #

https://www.autocar.co.uk0
https://www.autocar.co.uk/car-review/abarth/595
https://www.autocar.co.uk/car-review/abarth/595-competizione
https://www.autocar.co.uk/car-review/abarth/124-spider-2016-2019
https://www.autocar.co.uk/car-review/abarth/695-biposto-2015-2016
https://www.autocar.co.uk0
https://www.autocar.co.uk/car-review/ac-schnitzer/acs3-sport
https://www.autocar.co.uk/car-review/ac-schnitzer/acs1
https://www.autocar.co.uk/car-review/ac-schnitzer/acs5-sport
https://www.autocar.co.uk0
https://www.autocar.co.uk/car-review/allard/j2x-mkii
https://www.autocar.co.uk0
https://www.autocar.co.uk/car-review/alfa-romeo/giulia
https://www.autocar.co.uk/car-review/alfa-romeo/tonale

期望输出 #

https://www.autocar.co.uk/car-review/abarth/595
https://www.autocar.co.uk/car-review/abarth/595-competizione
https://www.autocar.co.uk/car-review/abarth/124-spider-2016-2019
https://www.autocar.co.uk/car-review/abarth/695-biposto-2015-2016
https://www.autocar.co.uk/car-review/ac-schnitzer/acs3-sport
https://www.autocar.co.uk/car-review/ac-schnitzer/acs1
https://www.autocar.co.uk/car-review/ac-schnitzer/acs5-sport
https://www.autocar.co.uk/car-review/allard/j2x-mkii
https://www.autocar.co.uk/car-review/alfa-romeo/giulia
https://www.autocar.co.uk/car-review/alfa-romeo/tonale

问题分析与改进方案 #

问题根源 #

无效链接https://www.autocar.co.uk0的出现，是因为下拉菜单里存在**value为空的选项**（通常是默认提示选择的选项），请求这类空value对应的AJAX接口后，返回的JSON数据里会出现0作为无效的路径key，拼接后就生成了错误链接。

具体改进点 #

1. 过滤无效厂商选项：只保留value不为空的厂商选项，避免发起无效的AJAX请求
2. 校验车型路径有效性：确保拼接的路径以/开头，跳过无效的路径值
3. 优化异常处理：不要捕获所有异常，只处理JSON解析失败这类特定异常，方便排查问题
4. 代码可读性优化：增加关键步骤注释，拆分复杂逻辑

修正后的代码 #

from bs4 import BeautifulSoup
import requests

# 目标网站基础URL
BASE_URL = "https://www.autocar.co.uk"
s = requests.Session()

# 获取首页并解析下拉菜单
r = s.get(BASE_URL)
soup = BeautifulSoup(r.text, 'html.parser')
full_car_list = []

# 只保留value不为空的厂商选项，过滤默认提示项
car_makes = soup.select_one('#edit-make').select('option[value!=""]')
car_list = [
    (make.text, make.get("value"), f"{BASE_URL}/ajax/car-models/{make.get('value')}/0")
    for make in car_makes
]

for make_name, make_value, ajax_url in car_list:
    r = s.get(ajax_url)
    try:
        # 解析AJAX返回的车型数据
        model_options = r.json().get('options', {})
        for model_path, model_name in model_options.items():
            # 只处理以/开头的有效路径，跳过无效的0值
            if model_path.startswith('/'):
                car_model_url = f"{BASE_URL}{model_path}"
                print(car_model_url)
                # 把有效链接存入列表，方便后续爬取
                full_car_list.append((make_name, model_name, car_model_url))
    except ValueError:
        # 只捕获JSON解析失败的异常，其他异常正常抛出方便调试
        print(f"厂商{make_name}无有效车型数据")
        full_car_list.append((make_name, 'no models', f"{BASE_URL}/vehicles/{make_name}"))

代码说明 #

• 用option[value!=""]直接过滤空value的选项，避免无效请求
• 用startswith('/')校验路径有效性，跳过0这类无效值
• 异常处理只捕获ValueError（JSON解析失败），其他异常会正常抛出，便于定位问题
• 把有效链接存入full_car_list，方便后续批量爬取车型页面

内容的提问来源于stack exchange，提问作者JAA