使用BeautifulSoup爬取亚马逊土耳其站商品链接时遇AttributeError问题

问题原因 #

错误出现的核心原因是：部分匹配到的div.aok-relative元素中不存在span.rush-component标签，此时i.find()返回None，尝试访问link.a会触发AttributeError——因为None对象没有a属性。

修复方案 #

1. 增加空值检查：在访问元素属性前，先确认元素不为None，自动跳过不符合条件的元素。
2. 添加请求头：亚马逊会拦截无浏览器标识的请求，添加User-Agent确保获取完整的HTML内容。
3. 修正解析器：使用官方推荐的html.parser替代已废弃的html解析器。

修正后的代码（保留原遍历逻辑） #

from bs4 import BeautifulSoup
import requests

# 模拟浏览器请求头，避免被拦截
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36"
}

r = requests.get("https://www.amazon.com.tr/s?me=A2OCQ69K09P55W&marketplaceID=A33AVAJ2PDY3EV", headers=headers)
soup = BeautifulSoup(r.content, "html.parser")

urunler = soup.find_all("div", attrs={"class":"sg-col-inner"})

for urun in urunler:
    urun_linkleri = urun.find_all("div", attrs={"class":"aok-relative"})
    for i in urun_linkleri:
        link = i.find("span", attrs={"class":"rush-component"})
        if link:
            anchor_tag = link.find("a")
            if anchor_tag:
                link_devam = anchor_tag.get("href")
                if link_devam:
                    link_tamami = "https://www.amazon.com.tr" + link_devam
                    print(link_tamami)

更简洁的写法（直接定位商品链接） #

跳过冗余层级遍历，直接选择商品对应的锚点标签：

from bs4 import BeautifulSoup
import requests

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36"
}

r = requests.get("https://www.amazon.com.tr/s?me=A2OCQ69K09P55W&marketplaceID=A33AVAJ2PDY3EV", headers=headers)
soup = BeautifulSoup(r.content, "html.parser")

# 直接匹配商品链接的a标签
product_links = soup.find_all("a", class_="a-link-normal s-no-outline")

for link in product_links:
    href = link.get("href")
    if href:
        full_link = f"https://www.amazon.com.tr{href}"
        print(full_link)

内容的提问来源于stack exchange，提问作者Kadir Bağdatlı