Cassandra 3.x多微服务共享连接疑问及BusyPoolException求助

Hey there! Let's break down your questions about Cassandra Java Driver behavior, connection pooling, and that tricky BusyPoolException you're seeing. I'll start with the basics since you're new to this, then dive into your specific concerns.

First, quick context: Protocol V4 corresponds to Cassandra Java Driver 3.x (Driver 4+ uses Protocol V5). The connection pool in Driver 3.x operates on a per-node, per-local-datacenter basis. Each Session instance (your main entry point for interacting with Cassandra) maintains its own pool of connections to every node in the local datacenter.

Why do 5 microservices appear to share a single connection? #

Let's clear up a common misconception first: Separate microservice processes can't share a single physical connection to Cassandra. What you're likely observing is one of two scenarios:

• Your config sets maxConnectionsPerHost: 1 (the default for local DC nodes in Driver 3.x). This means each microservice only opens 1 connection per Cassandra node. For 5 microservices and 6 nodes, you should see 5 connections per node (one from each microservice). If you're checking Cassandra's connection stats and only see one connection from your application IP, double-check that all microservices are running and connecting properly.
• If you accidentally reused a single Session instance across microservices (unlikely for separate processes), all traffic would funnel through one pool—but this is a bad practice and not typical for microservice architectures.

Maximum CQL requests per microservice #

Let's calculate based on your current settings:

• maxRequestPerConnection: 1024: This is the number of concurrent requests a single connection can handle (Cassandra connections are asynchronous, so one connection can process multiple requests at once).
• maxConnectionsPerHost: 1: Each microservice opens 1 connection to each of your 6 Cassandra nodes.

The theoretical maximum concurrent requests per microservice is:
6 nodes × 1 connection/node × 1024 requests/connection = 6144 concurrent requests

Note this is a hard upper limit set by your driver config. In practice, your actual throughput will be limited by Cassandra's ability to process requests (e.g., CPU, disk I/O, slow queries) before hitting this pool cap.

Why the intermittent BusyPoolException? #

This exception fires when every available request slot in your connection pool is occupied—there's no room to accept new requests. Common triggers include:

• Sudden traffic spikes: If a microservice gets a burst of requests that exceeds the 6144 concurrent limit, the pool can't keep up.
• Slow queries: Long-running CQL requests (like unfiltered full-table scans or queries missing indexes) hold onto connection slots longer than expected, clogging the pool.
• Too few connections per host: Setting maxConnectionsPerHost: 1 is very conservative, especially if your microservice has high QPS or latency-sensitive requests.
• Driver queue limits: Driver 3.x has a default queue for pending requests; if that queue fills up before slots free up, you'll get this exception.

Recommendations to resolve the issue #

• Verify Session isolation: Ensure each microservice initializes its own Session instance (sessions are thread-safe within a process, but shouldn't be shared across separate microservice processes).
• Tune connection pool settings: Increase maxConnectionsPerHost (start with 4-8, adjust based on load testing) to give the pool more breathing room. maxRequestsPerConnection: 1024 is already a high value for most use cases, so focus on connection count first.
• Optimize slow queries: Use nodetool tpstats or Cassandra's system_traces table to identify long-running queries. Fix them by adding appropriate indexes, adjusting your data model (Cassandra is optimized for specific query patterns), or limiting result sets.
• Consider upgrading the driver: If possible, move to Driver 4.x+ (supports Protocol V5). It has a more flexible connection pool implementation, better performance, and improved error handling for pool-related issues.

内容的提问来源于stack exchange，提问作者Saurabh Jindal