亚马逊网页爬取遇AttributeError：NoneType无get_text属性

首次尝试爬取亚马逊商品页面获取标题时，遇到错误：通过id="productTitle"查找元素返回None，调用get_text()触发AttributeError: 'NoneType' object has no attribute 'get_text'。

执行代码 #

# import libaries
from bs4 import BeautifulSoup
import requests

# Connect to website 
URL = 'https://www.amazon.com/Hanes-Short-Sleeve-Beefy-T-Smoke/dp/B01KNM3EAM/ref=sr_1_4?crid=5ZT2DA0EILMA&keywords=t+shirts+for+man&qid=1692275014&sprefix=t+s%2Caps%2C422&sr=8-4'

headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36", 
    "X-Amzn-Trace-Id": "Root=1-64de10e0-49a614555bcfb2fc5d15488a"}

page = requests.get(URL, headers=headers)

soup1 = BeautifulSoup(page.content, "lxml")

soup2 = BeautifulSoup(soup1.prettify(), "lxml")

title = soup2.find(id="productTitle").get_text()

print(title)

报错信息 #

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Input In [42], in <cell line: 14>()
     10 soup1 = BeautifulSoup(page.content, "lxml")
     12 soup2 = BeautifulSoup(soup1.prettify(), "lxml")
---> 14 title = soup2.find(id="productTitle").get_text()
     16 print(title)

AttributeError: 'NoneType' object has no attribute 'get_text'

已确认信息 #

页面中确实存在目标元素，对应的HTML片段：

<span id="productTitle" class="a-size-large product-title-word-break">  "Hanes Mens Beefyt T-Shirt, Classic Heavyweight Cotton Crewneck Tee, Roomy Fit, 1 Or 2 Pack, Available in Tall" </span>

解决建议 #

• 移除重复解析步骤：代码中soup2 = BeautifulSoup(soup1.prettify(), "lxml")属于多余操作，prettify()会改变HTML结构，导致后续查找失败。直接使用soup1进行元素查找即可。
• 验证请求状态：先打印print(page.status_code)，确认返回值为200。如果是4xx/5xx，说明请求被亚马逊拦截，需要调整请求头或使用代理。
• 完善请求头：补充Accept-Language字段，模拟更真实的浏览器请求，示例：

  headers = {
      "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36",
      "Accept-Language": "zh-CN,zh;q=0.9,en;q=0.8",
      "X-Amzn-Trace-Id": "Root=1-64de10e0-49a614555bcfb2fc5d15488a"
  }
  

• 备用定位方式：若通过id查找仍失败，可尝试通过class属性定位：

  title = soup1.find("span", class_="a-size-large product-title-word-break").get_text(strip=True)
  

• 反爬应对：亚马逊反爬机制严格，若上述方法无效，可尝试添加浏览器Cookie（从实际访问的请求中复制），或使用代理IP分散请求来源。

内容的提问来源于stack exchange，提问作者izzka