使用BeautifulSoup提取无href的图片URL——Mecca护肤页爬取求助

我尝试爬取护肤品类页面的所有产品，提取每个产品的品牌、名称和图片URL，但在提取图片URL时卡住了——图片相关标签没有href属性。以某产品页面为例，图片的HTML结构如下：

<picture class="css-1azgcry-container" id="image-reponsive-container" data-testid="imageReponsiveContainer">  
  <img class="" title="Tatcha - Indigo Cleansing Balm" alt="Tatcha - Indigo   Cleansing Balm" src="https://www.mecca.com.au/on/demandware.static/-/Sites-mecca-online-catalog/default/dw0a0707d8/product/tatcha/hr/i-059912-indigo-cleansing-balm-7-1-940.jpg" data-testid="imageReponsiveImg">
</picture>

我需要提取<img>标签里src属性对应的URL，但下面的代码在提取图片URL部分存在问题，请帮忙修正：

import requests
from bs4 import BeautifulSoup

url = "https://www.mecca.com.au/skin-care/"
params = {"start": "0", "sz": "36", "format": "ajax"}
for params["start"] in range(0, 36 * 5, 36):
    productlinks = []
    cataloguelist = []
    soup = BeautifulSoup(requests.get('https://www.mecca.com.au/skin-care/', params=params).content, "html.parser")
    products = soup.find_all('div', class_="grid-product-info")

    for item in products:
        for link in item.find_all('a', href=True):
            productlinks.append(url + link['href'])

    for link in productlinks:
        response = requests.get(link)

        soup = BeautifulSoup(response.content, "html.parser")
        brand = soup.find('a', class_='css-1p371np-size5-size5-sansSerif-sansSerif-brandNameLink').text
        name = soup.find('span', class_='css-1noela6-paragraph-paragraph-sansSerif-sansSerif-productName').text
### 这里是我卡住的地方，我也试过用'picture'和class_='css-1azgcry-container'
        image = soup.find('picture', attrs={'css-1azgcry-container', 'img', 'alt', 'src'})
        
print(brand, name, image)

你的问题出在图片提取的代码逻辑上：

1. attrs={'css-1azgcry-container', 'img', 'alt', 'src'} 写法错误，attrs需要传入键值对（比如{'class': 'css-1azgcry-container'}），而不是集合。
2. 你需要先找到<picture>标签，再定位里面的<img>元素，最后提取src属性；或者直接通过<img>的data-testid定位（更稳定，因为class可能随网站更新变化）。

修正后的图片提取代码 #

可以用两种方式实现：

# 方式1：通过picture标签的class定位，再找内部img
picture = soup.find('picture', class_='css-1azgcry-container')
image_url = picture.find('img')['src'] if picture and picture.find('img') else None

# 方式2：直接通过img的data-testid定位（推荐，属性更稳定）
img = soup.find('img', data-testid='imageReponsiveImg')
image_url = img['src'] if img else None

完整修正后的代码（含异常处理，避免单个产品提取失败导致程序中断） #

import requests
from bs4 import BeautifulSoup

base_url = "https://www.mecca.com.au/skin-care/"
params = {"start": "0", "sz": "36", "format": "ajax"}
cataloguelist = []  # 移到循环外，避免每次重置

for params["start"] in range(0, 36 * 5, 36):
    productlinks = []
    response = requests.get(base_url, params=params)
    soup = BeautifulSoup(response.content, "html.parser")
    products = soup.find_all('div', class_="grid-product-info")

    for item in products:
        for link in item.find_all('a', href=True):
            # 拼接完整产品链接，避免重复添加base_url
            product_link = link['href'] if link['href'].startswith('http') else base_url + link['href']
            productlinks.append(product_link)

    for link in productlinks:
        try:
            response = requests.get(link)
            response.raise_for_status()  # 检查请求是否成功
            soup = BeautifulSoup(response.content, "html.parser")

            # 提取品牌
            brand_elem = soup.find('a', class_='css-1p371np-size5-size5-sansSerif-sansSerif-brandNameLink')
            brand = brand_elem.text.strip() if brand_elem else "未知品牌"

            # 提取产品名称
            name_elem = soup.find('span', class_='css-1noela6-paragraph-paragraph-sansSerif-sansSerif-productName')
            name = name_elem.text.strip() if name_elem else "未知名称"

            # 提取图片URL（推荐用data-testid定位）
            img_elem = soup.find('img', data-testid='imageReponsiveImg')
            image_url = img_elem['src'] if img_elem else "无图片URL"

            cataloguelist.append({
                "brand": brand,
                "name": name,
                "image_url": image_url
            })

            print(f"品牌：{brand}，名称：{name}，图片URL：{image_url}")
        except Exception as e:
            print(f"处理链接{link}时出错：{str(e)}")

# 可选：将结果保存到文件
# import json
# with open('mecca_skincare.json', 'w', encoding='utf-8') as f:
#     json.dump(cataloguelist, f, ensure_ascii=False, indent=2)

内容的提问来源于stack exchange，提问作者user22276310