爬取Flipkart商品评论无法获取全部内容，请求技术协助

作为数据科学新手，学习网页爬取时尝试抓取Flipkart特定商品的全部评论，先是代码触发AttributeError报错，修改后无报错但仍无法获取全部评论，以下是相关代码、错误信息及排查方案：

旧代码 #

def perform_scraping(link):
    response = requests.get(link)
    soup = BeautifulSoup(response.content,"html.parser")
    all_div = soup.find_all("div", class_="_1AtVbE col-12-12")
    del all_div[0]
    for review in range(1,len(all_div)):
        rating = all_div[review].find("div", class_ = "_3LWZlK _1BLPMq").text
        review_title = all_div[review].find("p","_2-N8zT").text
        all_reviews.append((rating,review_title))

all_reviews = []
for page_number in range(1,2):
    i = page_number
    link = f"https://www.flipkart.com/prowl-tiger-shroff-push-up-board-upper-body-workout-push-up-bar/product-reviews/itm0487671f4df34?pid=BAAGM82GUHTQ3KFZ&lid=LSTBAAGM82GUHTQ3KFZVCWDQ9&marketplace=FLIPKART&page={i}"
    perform_scraping(link)
all_reviews

错误信息 #

AttributeError                            Traceback (most recent call last)
Cell In[3], line 15
     13     i = page_number
     14     link = f"https://www.flipkart.com/prowl-tiger-shroff-push-up-board-upper-body-workout-push-up-bar/product-reviews/itm0487671f4df34?pid=BAAGM82GUHTQ3KFZ&lid=LSTBAAGM82GUHTQ3KFZVCWDQ9&marketplace=FLIPKART&page={i}"
---> 15     perform_scraping(link)
     16 all_reviews

Cell In[3], line 7, in perform_scraping(link)
      5 del all_div[0]
      6 for review in range(1,len(all_div)):
----> 7     rating = all_div[review].find("div", class_ = "_3LWZlK _1BLPMq").text
      8     review_title = all_div[review].find("p","_2-N8zT").text
      9     all_reviews.append((rating,review_title))

AttributeError: 'NoneType' object has no attribute 'text'

更新后的代码 #

def perform_scraping(link):
    response = requests.get(link)
    soup = BeautifulSoup(response.content, "html.parser")
    all_div = soup.find_all("div", class_="_1AtVbE col-12-12")
    del all_div[0]
    all_reviews = []
    for review in range(0, len(all_div)):
        rating_elem = all_div[review].find("div", class_="_3LWZlK _1BLPMq")
        review_title_elem = all_div[review].find("p", class_="_2-N8zT")

        if rating_elem is not None and review_title_elem is not None:
            rating = rating_elem.text
            review_title = review_title_elem.text
            all_reviews.append((rating, review_title))
    
    return all_reviews

all_reviews = []
for page_number in range(1, 2):
    link = f"https://www.flipkart.com/prowl-tiger-shroff-push-up-board-upper-body-workout-push-up-bar/product-reviews/itm0487671f4df34?pid=BAAGM82GUHTQ3KFZ&lid=LSTBAAGM82GUHTQ3KFZVCWDQ9&marketplace=FLIPKART&page={page_number}"
    all_reviews.extend(perform_scraping(link))

print(all_reviews)

问题分析与修复方案 #

1. 旧代码报错核心原因 #

直接调用.text前未判断元素是否存在：部分评论的评分类名并非_3LWZlK _1BLPMq（比如差评会用_3LWZlK _1rdVr6 _1BLPMq），导致find()返回None，触发AttributeError。

2. 更新后代码仍无法获取全部评论的原因 #

• 类名匹配不全：仅抓取了特定样式的评分元素，遗漏其他评分类型的评论
• 未处理反爬机制：无请求头的requests.get会被Flipkart识别为爬虫，返回不完整页面内容
• 容器筛选不精准：all_div包含分页栏、广告等非评论元素，直接遍历会漏掉有效评论
• 分页范围过小：当前仅爬取第1页数据

3. 具体修复措施 #

（1）通用化元素选择器

匹配所有评分元素，无需精准匹配完整类名：

rating_elem = item.find("div", class_="_3LWZlK")  # 只要包含_3LWZlK类名即可匹配所有评分

（2）添加请求头模拟浏览器

避免被反爬机制拦截：

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36"
}
response = requests.get(link, headers=headers)

（3）精准定位评论容器

直接从评论列表父容器抓取评论项，避免混入无关元素：

review_container = soup.find("div", class_="_16PBlm")
if review_container:
    all_review_items = review_container.find_all("div", class_="_1AtVbE col-12-12")
    for item in all_review_items:
        # 处理评论数据

（4）扩大分页范围

如需多页数据，调整range参数，比如爬取前5页：

for page_number in range(1, 6):

内容的提问来源于stack exchange，提问作者Atharva