使用BeautifulSoup编写的网页抓取代码未得到预期结果该如何排查?
问题排查结果
核心错误点
- 类名拼写不匹配:代码中查找特征字段的类名写为
feature_Group、feature_Name,实际页面对应元素的类名为featureGroup、featureName,无下划线、大小写也不匹配,导致Lot Size相关的循环完全不执行。 - 字段取值逻辑错误:仅卧室数量提取了
<b>标签内的数值,卫生间、面积等字段直接提取外层span文本,会输出带多余描述的内容(如"3 Full Baths"而非"3"),和预期的纯数值输出不符。
可选优化项
- 内置
html.parser解析容错性较低,若出现部分元素识别失败的情况,可替换为lxml解析器,需先执行pip install lxml安装依赖,再修改解析器参数即可。
修正后可运行代码
import requests from bs4 import BeautifulSoup headers = { 'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:55.0) Gecko/20100101 Firefox/55.0', } r = requests.get("http://pyclass.com/real-estate/rock-springs-wy/LCWYROCKSPRINGS/", headers=headers) c = r.content # 可替换为lxml解析器提升兼容性:BeautifulSoup(c, "lxml") soup = BeautifulSoup(c, "html.parser") all = soup.find_all("div", {"class": "propertyRow"}) for item in all: print(item.find("h4", {"class": "propPrice"}).text.replace("\n", "").replace(" ", "")) print(item.find_all("span", {"class": "propAddressCollapse"})[0].text) print(item.find_all("span", {"class": "propAddressCollapse"})[1].text) try: print(item.find("span", {"class": "infoBed"}).find("b").text) except: print(None) try: # 修正为提取b标签内的数值 print(item.find("span", {"class": "infoValueFullBath"}).find("b").text) except: print(None) try: # 修正为提取b标签内的数值 print(item.find("span", {"class": "infoSqFt"}).find("b").text) except: print(None) try: # 修正为提取b标签内的数值 print(item.find("span", {"class": "infoValueHalfBath"}).find("b").text) except: print(None) for column_group in item.find_all("div", {"class":"columnGroup"}): # 修正类名拼写 for feature_group, feature_name in zip(column_group.find_all("span", {"class":"featureGroup"}),column_group.find_all("span", {"class":"featureName"})): if "Lot Size" in feature_group.text: print(feature_name.text) print(" ")
内容的提问来源于stack exchange,提问作者Star Anurag
相关产品推荐
相关产品推荐

