StormCrawler集群最优拓扑配置咨询：多节点爬取问题排查

Let's break down your questions one by one, drawing on my experience with StormCrawler at scale:

1. AggregationSpout vs CollapsingSpout: Which to Choose? #

First, let's clarify the core differences between the two spouts, then address your performance issue with AggregationSpout:

Key Differences #

• CollapsingSpout: Built for large-scale crawls where Elasticsearch deep pagination becomes a performance killer. It queries the status index sorted by nextFetchDate, and resets the query's start offset once it exceeds es.status.max.start.offset (you've set this to 500). This avoids the heavy overhead of deep ES pagination, which slows down queries as you fetch results from later pages. It’s ideal for high-throughput crawls with massive URL sets.
• AggregationSpout: Relies on Elasticsearch's terms aggregation to group URLs into buckets (by hostname in your config) and fetches top URLs from each bucket. While this ensures better per-site URL distribution, aggregations are computationally expensive—especially with 40k+ sites. This adds significant load to your ES cluster and slows down crawl throughput.

Why Your AggregationSpout Performed Poorly #

Your ES config has es.status.sample: false, meaning the spout runs a full aggregation across all 40k+ hostname buckets every time it queries ES. For that scale, this is a massive operation that drags down both ES and your crawl speed. Even with sampling enabled, AggregationSpout is still less efficient than CollapsingSpout for large crawls unless you have strict per-site rate limiting needs that require per-bucket prioritization.

Recommendation: Stick with CollapsingSpout for your use case. It’s better suited for high-throughput crawls with large URL sets, and your current config (with es.status.max.start.offset: 500) is correctly set up to avoid deep pagination issues.

2. Is Your Parallelism Configuration Reasonable? #

Most parts of your topology are on the right track, but there are critical bottlenecks and over-provisioning issues to fix:

Critical Issues to Resolve #

• URLPartitionerBolt Parallelism (1): This is a major bottleneck. The URLPartitionerBolt routes URLs to fetchers by hostname to avoid overwhelming sites. A parallelism of 1 means all URLs pass through a single bolt instance, which will block the entire pipeline as your crawl scales. Fix: Set this to match your fetcher parallelism (5) or higher (e.g., 10) to distribute the routing load.
• Fetcher Thread Count (500 per bolt): 500 threads per fetcher bolt is way too high. Each thread handles an HTTP request, and having this many threads causes excessive context switching, memory overhead, and triggers anti-scraping protections on target sites (since you’re sending hundreds of requests per second from a single IP). Fix: Reduce fetcher.threads.number to 100-200 per fetcher. With 5 fetcher bolts, this gives you 500-1000 total threads—plenty for high throughput without overwhelming your nodes or target sites.
• JSoupParserBolt Parallelism (100): This is over-provisioned. Parsing is CPU-intensive, but 100 instances across 5 workers means 20 instances per worker—more than enough for most crawls. You can safely reduce this to 25-50 to free up resources for other components.
• Topology Max Spout Pending (250): This value is too low for a 5-worker cluster. It limits the number of in-flight tuples, throttling throughput. Fix: Increase this to 1000-2000 to allow more URLs to be processed in parallel without overwhelming the system.

Solid Configurations to Keep #

• Fetcher Bolt Parallelism (5): Perfect, since you have 5 static IPs—each fetcher can be bound to a unique IP to avoid anti-scraping blocks.
• Indexer/StatusUpdater Parallelism (25): Matches the throughput needs of your 3-node ES cluster, which can handle this level of bulk writes.
• Worker Heap (4096 MB): Sufficient for most components, especially if you reduce the fetcher thread count.

3. Why FETCH ERRORS Increased After Switching to 5 Nodes? #

The 20% jump in fetch errors is likely due to a combination of anti-scraping triggers and configuration mismatches. Here are the most probable causes:

1. Aggressive Request Rates from Multiple IPs:
   With 5 fetcher bolts each running 500 threads, you’re sending up to 2500 concurrent requests across 5 IPs. Many sites detect this as scraping behavior and block requests (via IP bans, rate limits, or 403/503 responses). Even if each IP sends 500 requests/sec, that’s far beyond normal human browsing patterns.

2. IP Reputation Issues:
   Your 5 static IPs might have different reputations. Some could be already flagged as crawlers (e.g., if they’re from a data center IP range), leading to higher error rates. Single-node crawls use one IP, so you only saw errors from that one IP’s reputation.

3. Bottlenecks in URL Routing:
   The URLPartitionerBolt with parallelism 1 can’t keep up with the volume of URLs from the spout. This causes URLs to pile up in the partitioner’s queue, leading to timeouts (even with topology.message.timeout.secs set to 300, fetcher requests might expire waiting for routing).

4. DNS Resolution Mismatches Across Nodes:
   If your 5 Storm nodes have different DNS configurations, some might fail to resolve certain site domains, leading to "unknown host" errors. Check that all nodes use the same reliable DNS server.

5. Elasticsearch Status Update Backlogs:
   While your StatusUpdaterBolt parallelism is 25, if ES is under load, it might not update URL statuses quickly enough. This can lead to duplicate fetch attempts for the same URL, which sites often block.

Fixes to Reduce Errors #

• Immediately reduce fetcher.threads.number to 100-200 per fetcher to lower request rates.
• Increase URLPartitionerBolt parallelism to 5 or 10 to eliminate routing bottlenecks.
• Add per-site rate limiting (using fetcher.delay.millis.per.host in your config) to slow down requests to individual sites—this will drastically reduce blocks.
• Check the error logs for specific error codes (403, 503, timeout) to pinpoint whether it’s anti-scraping, network issues, or DNS problems.

内容的提问来源于stack exchange，提问作者Zhassulan Nurushev