Python爬虫请求AliExpress链接触发MissingSchema错误求助

问题场景 #

爬取AliExpress家居分类商品信息及图片时，遍历商品详情页链接发起requests.get请求时触发MissingSchema错误，报错显示获取的URL为//开头的格式，缺少http/https协议头。

完整代码 #

import os
import requests
from bs4 import BeautifulSoup
import pandas as pd
from urllib.parse import urlparse, parse_qs


folder = 'images_test'
os.makedirs(folder, exist_ok=True)

root_folder = os.getcwd()

baseurl = 'https://www.aliexpress.com/category/3710/home-decor.html?spm=a2g0o.home.107.9.650c2145X9NKoP/'

# 存储所有商品链接
product_links = [] 

# 遍历店铺所有页面
for x in range(1, 7):
    url = f'https://www.aliexpress.com/category/3710/home-decor.html?spm=a2g0o.home.107.9.650c2145X9NKoP/page/{x}'
    print(url)
    
    r = requests.get(url)
    soup = BeautifulSoup(r.content, 'lxml')

    product_list = soup.find_all('div', class_= "content--container--2dDeH1y")
    # 获取每个商品的详情页链接
    for item in product_list:
        for link in item.find_all('a', href=True, class_="manhattan--container--1lP57Ag cards--gallery--2o6yJVt"):
            product_links.append(link['href'])

product_items_list = []

i = 0
d = {}  # 用于去重图片链接

os.chdir(folder)

for link_test in product_links:
    

    r = requests.get(link_test)
    soup = BeautifulSoup(r.content, 'lxml')
    
    title = soup.find('h1', class_='product_title').text.strip()
    price = soup.find('p', class_ = 'price').text.strip()
    header = soup.find('div', class_ = 'woocommerce-product-details__short-description').text.strip()
    sku  = soup.find('span', class_ = 'sku').text.strip()
    categories = soup.find('span' , class_ = 'posted_in').text.strip()
    description = soup.find('div', class_ = 'wc-tab-inner').text.strip()
    brand = soup.find('div', id = 'tab-pwb_tab-content').text.strip()

    images = soup.select('img', class_ = "manhattan--img--36QXbtQ product-img")
    
    downloaded = []
    
    for image in images:
        link = image['src']
        if link in d:
            name = d[link]
            downloaded.append(name)
        else:
            i += 1
            name = str(i) +'img.jpg'
            d[link] = name
            print('link:', link)
            print('name:', name)
            print('---')
            # 保存图片
            with open(name, 'wb') as f:
                im = requests.get(link)
                f.write(im.content)
            downloaded.append(name)
            
    # 存储商品信息
    product = {
        'sku': sku,
        'categories': categories,
        'name' : title,
        'description': description,
        'short_description': header,
        'price' : price[0:5],
        'images': downloaded,
    }
    product_items_list.append(product)


os.chdir(root_folder)

df = pd.DataFrame(product_items_list)
print(df)
df.to_csv('aziz3.csv')

报错信息 #

File ~\anaconda3\lib\site-packages\spyder_kernels\py3compat.py:356 in compat_exec
    exec(code, globals, locals)

  File c:\users\aziz3\desktop\new folder (2)\paloma.py:43
    r = requests.get(link_test)

  File ~\anaconda3\lib\site-packages\requests\api.py:73 in get
    return request("get", url, params=params, **kwargs)

  File ~\anaconda3\lib\site-packages\requests\api.py:59 in request
    return session.request(method=method, url=url, **kwargs)

  File ~\anaconda3\lib\site-packages\requests\sessions.py:573 in request
    prep = self.prepare_request(req)

  File ~\anaconda3\lib\site-packages\requests\sessions.py:484 in prepare_request
    p.prepare(

  File ~\anaconda3\lib\site-packages\requests\models.py:368 in prepare
    self.prepare_url(url, params)

  File ~\anaconda3\lib\site-packages\requests\models.py:439 in prepare_url
    raise MissingSchema(

MissingSchema: Invalid URL '//www.aliexpress.com/item/1005004696733559.html?algo_pvid=0f8ad776-26e1-4ff3-b0db-d1a0d421fd03&algo_exp_id=0f8ad776-26e1-4ff3-b0db-d1a0d421fd03-0&pdp_npi=3%40dis%21MAD%2191.8%213.94%21%21%21%21%21%40211bf31716825292368193713d07e4%2112000030134332941%21sea%21MA%210&curPageLogUid=kBu6G370fwtQ': No scheme supplied. Perhaps you meant http:////www.aliexpress.com/item/1005004696733559.html?algo_pvid=0f8ad776-26e1-4ff3-b0db-d1a0d421fd03&algo_exp_id=0f8ad776-26e1-4ff3-b0db-d1a0d421fd03-0&pdp_npi=3%40dis%21MAD%2191.8%213.94%21%21%21%21%21%40211bf31716825292368193713d07e4%2112000030134332941%21sea%21MA%210&curPageLogUid=kBu6G370fwtQ?

错误原因 #

页面中提取的链接是相对协议URL（以//开头），这种格式在浏览器中会自动继承当前页面的HTTP/HTTPS协议，但requests库无法自动识别，必须明确指定协议头才能发起请求。

修复方案 #

方案1：直接拼接HTTPS协议头 #

在收集商品链接时，对//开头的URL补全协议：

# 修改收集链接的代码段
for item in product_list:
    for link in item.find_all('a', href=True, class_="manhattan--container--1lP57Ag cards--gallery--2o6yJVt"):
        href = link['href']
        # 补全协议头
        if href.startswith('//'):
            href = 'https:' + href
        product_links.append(href)

同时，图片的src也可能是//开头，需要同步处理：

# 修改图片请求的代码段
link = image['src']
if link.startswith('//'):
    link = 'https:' + link

方案2：使用urllib.parse.urljoin（更稳妥） #

利用urljoin基于基础URL自动补全相对链接，避免手动判断的遗漏：

from urllib.parse import urljoin  # 确保已导入

# 收集商品链接时
base_url = 'https://www.aliexpress.com'
for item in product_list:
    for link in item.find_all('a', href=True, class_="manhattan--container--1lP57Ag cards--gallery--2o6yJVt"):
        full_link = urljoin(base_url, link['href'])
        product_links.append(full_link)

# 处理图片链接时
for image in images:
    link = urljoin(base_url, image['src'])
    # 后续保存逻辑不变

额外注意事项 #

• AliExpress有反爬机制，建议添加请求头（如User-Agent）模拟浏览器请求，避免被封禁：

  headers = {
      'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36'
  }
  # 请求时传入headers
  r = requests.get(url, headers=headers)
  

• 代码中提取商品信息的部分（如product_title、price等类名）可能因AliExpress页面更新失效，需定期检查页面结构调整选择器。

内容的提问来源于stack exchange，提问作者Anass Belgour