使用Beautiful Soup爬取多URL时出现ResultSet无find_all属性报错求助
报错根因
- 变量名覆盖冲突
你导入时将BeautifulSoup类的别名设置为soup,但在URL遍历循环中,你又将解析后的页面实例赋值给了同名的soup变量:soup = soup(url_contents,"html.parser")。第一次循环执行完成后,soup已经不再是BeautifulSoup类,而是解析后的页面对象,第二次循环时再调用soup()构造解析对象,返回的是不符合预期的ResultSet类型,自然没有find_all方法,触发报错。 - 代码结构逻辑错误
你处理击球手、投球手数据写入CSV的循环,全部写在URL遍历循环的外部,就算修复了变量名问题,也只能拿到最后一个URL的爬取结果,前面的所有URL的爬取数据都会被覆盖,无法实现多URL批量爬取的需求。
修复后可运行代码
import csv import pandas as pd from bs4 import BeautifulSoup # 取消别名避免冲突,也可设置为和变量不重复的别名如bs from urllib.request import urlopen as ureq import numpy as np import re urls = ["https://www.espncricinfo.com/series/pepsi-indian-premier-league-2014-695871/chennai-super-kings-vs-royal-challengers-bangalore-42nd-match-734013/full-scorecard", "https://www.espncricinfo.com/series/pepsi-indian-premier-league-2014-695871/chennai-super-kings-vs-kolkata-knight-riders-21st-match-733971/full-scorecard"] # 提前打开CSV文件,避免循环中反复打开关闭浪费资源 with open('testcsv3.csv', 'a',newline='') as csvfile: f = csv.writer(csvfile) for url in urls: url_contents = ureq(url) # 用独立变量名存储解析后的页面对象,不和类名/别名重名 page_soup = BeautifulSoup(url_contents,"html.parser") batsmen = page_soup.find_all("table", { "class":["table batsman"]}) bowlers = page_soup.find_all("table", { "class":["table bowler"]}) # 数据处理逻辑缩进挪到URL循环内部 for batsman in batsmen[0]: rows = batsman.find_all('tr')[:-2] for tr in rows: data=[] cols = tr.find_all('td') for td in cols: data.append(td.text.strip()) f.writerow(data) print(data) for bowler in bowlers[1]: rows = bowler.find_all('tr') for tr in rows: data=[] cols = tr.find_all('td') for td in cols: data.append(td.text.strip()) f.writerow(data) print(data)
优化建议
你的实现思路是可行的,可做以下调整提升稳定性和效率:
- 增加请求异常捕获:避免某一个URL请求超时、页面结构变化导致整个程序崩溃
- 统一页面结构校验:爬取前先判断
batsmen和bowlers是否为空,避免索引取值报错 - 数据写入可以改用pandas的
to_csv方法,代码更简洁,也方便后续数据处理
内容的提问来源于stack exchange,提问作者WPB
相关产品推荐
相关产品推荐

