Scrapy爬取IMDb速度极慢易停滞，求优化解决方法

使用Scrapy编写的IMDb爬虫脚本运行极慢且频繁停滞，已设置DOWNLOAD_DELAY=1但无效果。脚本及运行日志如下：

爬虫脚本 #

import scrapy
import json

class MoviesSpider(scrapy.Spider):
    name = 'movies'
    allowed_domains = ['www.imdb.com']
    start_urls = ['https://www.imdb.com/title/tt0096463/fullcredits/']

    custom_settings = {
        'DOWNLOAD_DELAY': 1  # add a delay of 1  seconds between requests
    }

    def parse(self, response):
        # Get movie information
        movie_year = response.css("img.poster::attr(alt)").extract_first().split("(")[1].split(")")[0]
        movie_name = response.css('meta[property="og:title"]::attr(content)').extract_first().split("(")[0]
        movie_id = response.url.split("/")[4]
        

        # Iterate over actors
        for i, actor in enumerate(response.css("table.cast_list tr")):
            actor_id = actor.css("a::attr(href)").extract_first()
            if actor_id:
                actor_id = actor_id.split("/")[-2]
                actor_name = actor.css("img::attr(title)").extract_first()

                # Get role name
                role_selector = f"table.cast_list tr:nth-child({i+1}) td.character a"
                actor_role = response.css(role_selector + "::text").extract_first()
                actor_role = actor_role.strip() if actor_role else None

                # Build movie data
                movie_data = {
                    "movie_id": movie_id,
                    "movie_name": movie_name,
                    "movie_year": movie_year,
                    "actor_id": actor_id,
                    "actor_name": actor_name,
                    "role_name": actor_role
                }

                # Follow actor page
                next_page = f"https://www.imdb.com/name/{actor_id}"
                yield response.follow(next_page, callback=self.parse_actor_bio,
                                      meta={'movie_data': movie_data})

    def parse_actor_bio(self, response):
        
        response.css(".ipc-metadata-list-item__list-content ::text").extract()

        movie_data = response.meta['movie_data']
        date_place_info = response.css('ul li:contains("Born") ::text').extract()[1:]

        born_date = "".join(date_place_info[0:3])
        born_place = "".join(date_place_info[3:])
        

        # Build result object
        result = {
            "movie_id": movie_data['movie_id'],
            "movie_name": movie_data['movie_name'],
            "movie_year": movie_data['movie_year'],
            "actor_id": movie_data['actor_id'],
            "actor_name": movie_data['actor_name'],
            "role_name": movie_data['role_name'],
            "born_date": born_date,
            "born_place": born_place
        }

        yield json.loads(json.dumps(result))

        movie_links = [x.split("/")[2] for x in response.css('a[href^="/title/"]::attr(href)').extract()]
        movie_links = list(set(movie_links))
        for movie_link in movie_links:
            yield response.follow(f"https://www.imdb.com/title/{movie_link}/fullcredits/", callback=self.parse)

运行日志 #

2023-04-11 18:54:23 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.imdb.com/name/nm14444245/>
{'movie_id': 'tt5640060', 'movie_name': 'Chicago Justice ', 'movie_year': 'TV Series 2017', 'actor_id': 'nm14444245', 'actor_name': 'Matt Abbott', 'role_name': 'Juror', 'born_date': '', 'born_place': ''}
2023-04-11 18:54:25 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.imdb.com/name/nm0811523/>
{'movie_id': 'tt0102975', 'movie_name': 'Star Trek VI: The Undiscovered Country ', 'movie_year': '1991', 'actor_id': 'nm0811523', 'actor_name': 'Michael Snyder', 'role_name': 'Crewman Dax', 'born_date': '', 'born_place': ''}
2023-04-11 18:54:39 [scrapy.extensions.logstats] INFO: Crawled 41 pages (at 10 pages/min), scraped 22 items (at 6 items/min)
2023-04-11 18:55:39 [scrapy.extensions.logstats] INFO: Crawled 41 pages (at 0 pages/min), scraped 22 items (at 0 items/min)
2023-04-11 18:56:39 [scrapy.extensions.logstats] INFO: Crawled 41 pages (at 0 pages/min), scraped 22 items (at 0 items/min)

1. 限制爬虫深度，避免无限递归 #

当前脚本从演员页面提取所有关联电影并重复调用parse，会导致爬虫无限遍历IMDb的电影-演员网络，很快耗尽请求队列或被反爬机制限制。

• 解决：在custom_settings中添加深度限制：

  custom_settings = {
      'DOWNLOAD_DELAY': 1,
      'DEPTH_LIMIT': 3
  }
  

• 或者维护已爬取电影ID集合，避免重复请求：
  在Spider类中初始化集合：

  def __init__(self, *args, **kwargs):
      super().__init__(*args, **kwargs)
      self.crawled_movies = set()
  

  在parse方法开头检查：

  def parse(self, response):
      movie_id = response.url.split("/")[4]
      if movie_id in self.crawled_movies:
          return
      self.crawled_movies.add(movie_id)
      # 剩余原有代码...
  

2. 修复低效的角色选择器 #

当前用tr:nth-child({i+1})全局定位角色，每次循环都要重新遍历整个表格，效率极低且易出错。

• 解决：直接在当前actor节点内提取角色：

  # 替换原有角色提取代码
  actor_role = actor.css("td.character a::text").extract_first()
  actor_role = actor_role.strip() if actor_role else None
  

3. 优化反爬策略，避免被IMDb限制 #

IMDb有严格反爬机制，单一IP频繁请求会被临时封禁（表现为请求停滞），仅设置DOWNLOAD_DELAY不够：

• 添加真实浏览器USER_AGENT：

  custom_settings = {
      'DOWNLOAD_DELAY': 1,
      'USER_AGENT': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36'
  }
  

• 启用自动限速，让Scrapy根据服务器响应调整请求速度：

  custom_settings = {
      'DOWNLOAD_DELAY': 1,
      'USER_AGENT': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36',
      'AUTOTHROTTLE_ENABLED': True,
      'AUTOTHROTTLE_START_DELAY': 1,
      'AUTOTHROTTLE_MAX_DELAY': 5
  }
  

4. 移除无用代码 #

parse_actor_bio方法中第一行response.css(".ipc-metadata-list-item__list-content ::text").extract()完全无用，直接删除以节省资源。

5. 优化数据提取逻辑 #

提取出生日期和地点的代码依赖硬编码索引，容易因页面结构变化出错：

• 改用更稳定的选择器：

  born_info = response.css('li[data-testid="nm_pd_birth"]')
  if born_info:
      born_text = born_info.css('div::text').extract()
      born_date = born_text[0].strip() if len(born_text) > 0 else ''
      born_place = born_text[1].strip() if len(born_text) > 1 else ''
  else:
      born_date = ''
      born_place = ''
  

6. 调整并发请求数 #

适当提高并发数（配合反爬设置），提升爬取效率：

custom_settings = {
    # 其他设置...
    'CONCURRENT_REQUESTS': 8,
    'CONCURRENT_REQUESTS_PER_DOMAIN': 4
}

内容的提问来源于stack exchange，提问作者Norhther