Scrapy爬取HackerNews筛选300分以上文章的实现方案优化咨询
HackerNews Scrapy爬取需求实现方案
现有代码的核心问题是分开抓取文章基础信息与积分信息未做关联,无法实现积分匹配和过滤,以下是两种可直接运行的实现方案:
方案1:利用页面结构直接成对匹配(更简洁)
利用HackerNews页面每篇文章行与对应积分行相邻的结构特性,直接成对遍历无需额外做ID匹配:
import scrapy class ArticlesSpider(scrapy.Spider): name = 'articles' # 直接配置前2页的爬取地址 start_urls = [ 'https://news.ycombinator.com/news?p=1', 'https://news.ycombinator.com/news?p=2' ] def parse(self, response): article_rows = response.css('tr.athing') score_rows = response.css('tr.athing + tr td.subtext') # 成对遍历文章与对应积分信息 for article, score_info in zip(article_rows, score_rows): article_id = article.css('::attr(id)').get() title = article.css('td.title a.storylink::text').get() link = article.css('td.title a.storylink::attr(href)').get() # 处理积分格式与空值情况 score_text = score_info.css('span.score::text').get() if not score_text: continue points = int(score_text.replace(' points', '')) # 仅返回积分超300的内容 if points > 300: yield { 'id': article_id, 'title': title, 'link': link, 'points': points }
方案2:按ID映射匹配(符合你原本的实现思路)
如果需要用ID比对的逻辑实现,可以先构建积分ID映射字典,再匹配文章ID:
import scrapy class ArticlesSpider(scrapy.Spider): name = 'articles' start_urls = [ 'https://news.ycombinator.com/news?p=1', 'https://news.ycombinator.com/news?p=2' ] def parse(self, response): link = response.css('tr.athing') score = response.css('td.subtext') # 先构建积分映射字典 point_map = {} for points in score: post_id_str = points.css('span.score::attr(id)').get() if not post_id_str: continue # 提取score_前缀后的数字ID post_id = post_id_str.replace('score_', '') point_num = points.css('span.score::text').get() if point_num: point_map[post_id] = int(point_num.replace(' points', '')) # 遍历文章时匹配积分过滤 for website in link: article_id = website.css('tr::attr(id)').get() points = point_map.get(article_id, 0) if points > 300: yield { 'title': website.css('td.title a.storylink::text').get(), 'link': website.css('td.title a.storylink::attr(href)').get(), 'id': article_id, 'points': points }
两种方案都可以直接实现你的需求:爬取前2页内容,自动关联积分,过滤输出积分超过300的文章信息。
内容的提问来源于stack exchange,提问作者mharre
相关产品推荐
相关产品推荐

