Scrapy爬虫运行过慢求助：已调优设置仍耗时27分钟

我正在为某电商网站编写Scrapy爬虫，时间有限。刚接触网页抓取，自知代码并非最优，但无法理解为何该代码运行耗时27分钟。

class ProductSpider(scrapy.Spider):
    name = "products"
    start_urls = ["there is link here"]

    def parse(self, response):
        selector1 = "ul[class='mobile-sub wsmenu-list']"
        selector2 = "li[class = 'wsshoplink-active'] div[class = 'wstitemright clearfix wstitemrightactive'] div[class = 'col-lg-12 col-md-12 clearfix'] ul li"
        categoryList = response.css(selector1)[0].css(selector2)

        for category in categoryList:
            yield scrapy.Request(
                url=category.css("li a::attr(href)").extract_first(),
                callback=self.parse_category_to_product,
            )

    def parse_category_to_product(self, response):
        product_links = response.css("#products a::attr(href)").extract()

        for eachlink in product_links:
            yield scrapy.Request(url=eachlink, callback=self.parse_main_item)

        next_page = response.css("link[rel='next']::attr(href)").extract_first()
        if next_page:
            yield scrapy.Request(url=next_page, callback=self.parse_category_to_product)

    def parse_main_item(self, response):
        item = ProductScrapperItem()

        name = response.css("h1[class = 'product-h1']::text").extract_first()
        item["name"] = name

        itemdict = dict()
        itemdict["name"] = name

        with open("test03.json", "a") as f:  # Open in append mode
            json.dump(itemdict, f, indent=4)
            f.write(",\n")

        return item

我已在settings.py中取消以下配置的注释：

CONCURRENT_REQUESTS = 32
CONCURRENT_REQUESTS_PER_DOMAIN = 32
CONCURRENT_REQUESTS_PER_IP = 32
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_START_DELAY = 5
AUTOTHROTTLE_MAX_DELAY = 60
AUTOTHROTTLE_TARGET_CONCURRENCY = 5.0
AUTOTHROTTLE_DEBUG = False

执行scrapy bench得到如下输出：

2024-04-07 01:58:07 [scrapy.core.engine] INFO: Spider opened
2024-04-07 01:58:07 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2024-04-07 01:58:07 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2024-04-07 01:58:08 [scrapy.extensions.logstats] INFO: Crawled 2 pages (at 120 pages/min), scraped 0 items (at 0 items/min)
2024-04-07 01:58:09 [scrapy.extensions.logstats] INFO: Crawled 2 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2024-04-07 01:58:10 [scrapy.extensions.logstats] INFO: Crawled 2 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2024-04-07 01:58:11 [scrapy.extensions.logstats] INFO: Crawled 2 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2024-04-07 01:58:12 [scrapy.extensions.logstats] INFO: Crawled 2 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2024-04-07 01:58:13 [scrapy.extensions.logstats] INFO: Crawled 3 pages (at 60 pages/min), scraped 0 items (at 0 items/min)
2024-04-07 01:58:14 [scrapy.extensions.logstats] INFO: Crawled 3 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2024-04-07 01:58:15 [scrapy.extensions.logstats] INFO: Crawled 4 pages (at 60 pages/min), scraped 0 items (at 0 items/min)
2024-04-07 01:58:16 [scrapy.extensions.logstats] INFO: Crawled 5 pages (at 60 pages/min), scraped 0 items (at 0 items/min)
2024-04-07 01:58:17 [scrapy.extensions.logstats] INFO: Crawled 29 pages (at 1440 pages/min), scraped 0 items (at 0 items/min)
2024-04-07 01:58:17 [scrapy.core.engine] INFO: Closing spider (closespider_timeout)
2024-04-07 01:58:18 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 18922,
 'downloader/request_count': 61,
 'downloader/request_method_count/GET': 61,
 'downloader/response_bytes': 105029,
 'downloader/response_count': 61,
 'downloader/response_status_count/200': 61,
 'elapsed_time_seconds': 10.472302,
 'finish_reason': 'closespider_timeout',
 'finish_time': datetime.datetime(2024, 4, 6, 22, 58, 18, 14594, tzinfo=datetime.timezone.utc),
 'log_count/INFO': 20,
 'request_depth_max': 3,
 'response_received_count': 61,
 'robotstxt/request_count': 1,
 'robotstxt/response_count': 1,
 'robotstxt/response_status_count/200': 1,
 'scheduler/dequeued': 60,
 'scheduler/dequeued/memory': 60,
 'scheduler/enqueued': 1201,
 'scheduler/enqueued/memory': 1201,

我尝试优化Scrapy爬虫速度但未成功，恳请各位提供帮助。

内容的提问来源于stack exchange，提问作者serkan ertas