如何调整Selenium爬虫,仅抓取德甲页面指定区域的最新轮次数据
Selenium爬虫抓取范围调整方案
你只需要对现有代码做2处核心调整即可,无需重写原有逻辑:
1. 限定抓取范围为「Programma」表格
不要直接从整个页面查找元素,先定位到目标表格的父容器,后续所有元素查找都在该容器内执行,过滤其他3个无关表格的内容:
# 原有driver.get执行后新增这行,定位Programma对应的目标表格容器 target_table = driver.find_element(By.CSS_SELECTOR, "div.event-summary.event, div.leagues-static.event-summary-leagues") # 原来的all_rows查找修改为从目标容器内查找,不再从全局driver查找 all_rows = target_table.find_elements(By.CSS_SELECTOR, "div[class^='event__round'],div[class^='event__match']")
2. 仅保留最新一轮赛事数据
页面内轮次按时间倒序排列,第一个出现的event__round就是最新轮次,你可以加一个终止标志,抓取完最新轮的所有赛事后直接终止循环即可:
current_round = '?' # 新增标志位,标记是否已经抓取完最新一轮 latest_round_finished = False for bundesliga in all_rows: # 如果已经抓取完最新轮,直接退出循环 if latest_round_finished: break classes = bundesliga.get_attribute('class') if 'event__round' in classes: # 如果current_round已经有值,说明遇到了下一轮的标题,标记抓取完成 if current_round != '?': latest_round_finished = True continue current_round = bundesliga.text.split(" ")[-1] else: # 原有赛事数据解析逻辑完全不用改 datetime = bundesliga.find_element(By.CSS_SELECTOR, "[class^='event__time']") date, time = datetime.text.split(" ") date = date.rstrip('.') team_home = bundesliga.find_element(By.CSS_SELECTOR, "[class^='event__participant event__participant--home']") team_away = bundesliga.find_element(By.CSS_SELECTOR, "[class^='event__participant event__participant--away']") score_home = bundesliga.find_element(By.CSS_SELECTOR, "[class^='event__score event__score--home']") score_away = bundesliga.find_element(By.CSS_SELECTOR, "[class^='event__score event__score--away']") # 注意原有代码这里有笔误:不要把列表append到自身,建议单独定义结果列表存储 match_data = [current_round, date, time, team_home.text, team_away.text, score_home.text, score_away.text] print(match_data)
内容的提问来源于stack exchange,提问作者Erling Olsen
相关产品推荐
相关产品推荐

