如何抓取带独立div id的用户评论？Python批量提取整页评论

核心思路 #

之前的代码硬编码了具体的post_message_xxx ID，只能抓取单条评论。要获取整页评论，只需匹配所有ID以post_message_开头的<div>元素即可，这可以通过XPath或BeautifulSoup的筛选规则实现。

方案1：基于lxml实现（沿用你当前的库） #

修改XPath路径，使用starts-with()函数匹配所有目标元素，再遍历提取内容：

import requests
from lxml import html

url = 'https://www.team-bhp.com/forum/luxury-imports-niche/213083-looking-buying-bmw-x1-need-advice.html'

# 匹配所有ID以post_message_开头的div
xpath_path = '//div[starts-with(@id, "post_message_")]'
response = requests.get(url)
source_code = html.fromstring(response.content)

# 获取所有评论元素
comment_elements = source_code.xpath(xpath_path)

# 遍历输出每条评论
for idx, element in enumerate(comment_elements, 1):
    comment_text = element.text_content().strip()
    print(f"评论 {idx}:\n{comment_text}\n{'='*50}")

方案2：基于BeautifulSoup实现（你可接受的库） #

利用BeautifulSoup的lambda筛选器匹配目标ID，代码更直观：

import requests
from bs4 import BeautifulSoup

url = 'https://www.team-bhp.com/forum/luxury-imports-niche/213083-looking-buying-bmw-x1-need-advice.html'

response = requests.get(url)
soup = BeautifulSoup(response.content, 'lxml')

# 筛选所有ID以post_message_开头的div
comment_elements = soup.find_all('div', id=lambda x: x and x.startswith('post_message_'))

# 遍历输出每条评论
for idx, element in enumerate(comment_elements, 1):
    comment_text = element.get_text(strip=True)
    print(f"评论 {idx}:\n{comment_text}\n{'='*50}")

注意事项 #

• 建议给请求添加User-Agent头，模拟浏览器请求，避免被网站反爬机制拦截：

  headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36'}
  response = requests.get(url, headers=headers)
  

• 若目标页面存在分页，需额外处理分页逻辑（解析下一页URL并循环抓取）。

内容的提问来源于stack exchange，提问作者Jay21