Scrapy多URL迭代存储及CoinMarketCap数据爬取保存不全与优化问题

Hey there! Let's break down your three key issues and get your scraping project running smoothly.

1. Why Only ~2000 Records Save (Instead of 20000+) #

This is usually caused by one of a few common pitfalls—let's walk through troubleshooting steps:

• Check for Accidental Item Dropping: Look at your Item Pipeline (if you have one). If your process_item method throws a DropItem exception, or forgets to return item, valid data might be getting discarded. Double-check for any logic that filters items unintentionally.
• Pagination/Load More Logic Gaps: CoinMarketCap uses dynamic loading for its cryptocurrency lists (either via AJAX or infinite scroll). If your crawler only scrapes the initial static page, you'll miss most results. Instead, target their public API endpoints (e.g., endpoints that take start and limit parameters to paginate through results) to fetch all entries.
• Anti-Crawler Restrictions: CoinMarketCap might be throttling your requests without triggering obvious errors. Try adding a realistic USER_AGENT in settings.py, setting DOWNLOAD_DELAY = 2, or rotating user agents/proxies to avoid being blocked.
• Feed Export Buffer Issues: If you're using Scrapy's built-in feed export to save to coins.csv, it might buffer data and only write to disk once the crawler finishes. If your crawler stops prematurely (due to crashes or anti-crawl blocks), you'll lose unwritten data. Enable batch writing by adding this to settings.py:

  FEED_EXPORT_BATCH_ITEM_COUNT = 1000
  FEED_URI = "coins_%(batch_id)s.csv"
  

2. Code Optimization Tips #

Based on your file structure (hist.py, utils.py, coins.csv), here are actionable optimizations:

• Modularize Reusable Code: Move repetitive logic (like request header generation, date parsing, or API parameter formatting) into utils.py to keep hist.py clean. For example, a get_headers() function in utils.py can return consistent, realistic request headers.
• Optimize Request Handling:
  • Set CONCURRENT_REQUESTS in settings.py to a reasonable number (e.g., 10-20) to avoid overwhelming the server and getting blocked.
  • Use errback in your requests to handle failed requests gracefully and log errors for debugging:

    def start_requests(self):
        yield scrapy.Request(url=your_url, callback=self.parse, errback=self.handle_error)
    
    def handle_error(self, failure):
        self.logger.error(f"Request failed: {failure.request.url}")
    

• Efficient Data Deduplication: If you're scraping historical data, define a unique identifier for each item (e.g., coin_id + date) and add a deduplication pipeline to avoid saving duplicate entries.
• Log Strategically: Use Scrapy's built-in logging to track key events (e.g., "Fetched page X", "Saved Y items") instead of printing to console. This makes debugging way easier.

3. Iterating Over Multiple URLs & Saving Data Per Iteration #

There are two straightforward ways to handle this in Scrapy:

Option 1: Batch Saving with Custom Pipeline #

Create a pipeline that saves items in batches after each iteration/page. Add this to your project (you can put it in hist.py or a separate pipelines.py file):

class BatchSavePipeline:
    def __init__(self):
        self.batch_size = 1000  # Adjust based on your needs
        self.current_batch = []
        self.batch_number = 1

    def process_item(self, item, spider):
        self.current_batch.append(item)
        # Save batch when we hit the size limit
        if len(self.current_batch) >= self.batch_size:
            self._save_batch()
            self.current_batch = []
            self.batch_number += 1
        return item

    def _save_batch(self):
        import csv
        filename = f"coins_batch_{self.batch_number}.csv"
        with open(filename, "a", newline="", encoding="utf-8") as f:
            writer = csv.DictWriter(f, fieldnames=self.current_batch[0].keys())
            # Write header only for the first batch
            if self.batch_number == 1:
                writer.writeheader()
            writer.writerows(self.current_batch)

    def close_spider(self, spider):
        # Save any remaining items when the crawler finishes
        if self.current_batch:
            self._save_batch()

Then enable the pipeline in settings.py:

ITEM_PIPELINES = {
    'your_project_name.hist.BatchSavePipeline': 300,
}

Option 2: Generate Iterative Requests & Per-Page Saving #

In your start_requests method, loop through your target URLs (e.g., paginated API endpoints) and process each page's data. You can even save each page's results to a separate file if needed:

def start_requests(self):
    # Example: Iterate through 20 pages of results (100 items per page)
    for page in range(1, 21):
        url = f"https://api.coinmarketcap.com/data-api/v3/cryptocurrency/listing?start={page*100 - 99}&limit=100"
        yield scrapy.Request(
            url=url,
            callback=self.parse_page,
            meta={"page_number": page}  # Pass page number to save with filename
        )

def parse_page(self, response):
    data = json.loads(response.text)
    items = []
    # Extract your items from the response data here
    for coin in data['data']['cryptoCurrencyList']:
        item = YourItem()
        item['name'] = coin['name']
        item['symbol'] = coin['symbol']
        # Add other fields...
        items.append(item)
        yield item  # Send to pipeline for batch saving
    
    # Optional: Save this page's items to a separate file
    page_num = response.meta['page_number']
    with open(f"coins_page_{page_num}.csv", "w", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=items[0].keys())
        writer.writeheader()
        writer.writerows(items)

内容的提问来源于stack exchange，提问作者Dhaval Thakkar