Spring Cloud Stream GCP Pub/Sub线程OOM问题及重试配置咨询

Hey there, let's break down your problem and walk through the solutions step by step.

根因确认 #

Your java.lang.OutOfMemoryError: unable to create new native thread is exactly what you suspected: when your app loses connectivity to GCP Pub/Sub, the default gRPC-based retry mechanism runs wild. With no limits on retries or timeouts, it keeps spinning up new threads to attempt reconnections, eventually hitting the system's maximum thread limit. The stack trace confirms this—threads are being created in the gRPC DNS resolver's thread pool as it tries to refresh connections repeatedly.

关于你问的配置有效性 #

All three of the retry configurations you mentioned are effective and will help mitigate this thread exhaustion issue:

• spring.cloud.gcp.pubsub.publisher.retry.total-timeout-seconds=5: This sets a hard cap on the total time spent retrying a failed publish operation. Once this 5-second window is up, the retry loop stops, so no more threads are created for that failed request. The default 0 means infinite retries, which is dangerous for network outages.
• spring.cloud.gcp.pubsub.publisher.retry.max-attempts=2: This limits the total number of retry attempts (including the initial call) to 2. It prevents the system from endlessly spawning threads to retry the same failed request. Again, the default 0 allows unlimited attempts, which is risky.
• spring.cloud.gcp.pubsub.publisher.retry.max-rpc-timeout-second=5: This controls the timeout for individual RPC calls to Pub/Sub. If a single publish attempt takes longer than 5 seconds, it times out instead of blocking a thread indefinitely. This stops threads from getting stuck waiting for unresponsive Pub/Sub endpoints.

额外的线程性能优化建议 #

Beyond just retry limits, here are more steps to harden your setup:

• Configure publisher thread pool size: Set spring.cloud.gcp.pubsub.publisher.executor-threads to a reasonable value (e.g., 10-20, based on your server's CPU/memory resources). This caps the maximum number of threads dedicated to Pub/Sub publishing, preventing uncontrolled thread creation. The default uses an unbounded thread pool, which is the root cause of your OOM error.
• Use asynchronous publishing: Instead of blocking with source.output().send(...), use sendAsync(...) which returns a ListenableFuture. This lets your app continue processing other tasks without tying up threads waiting for Pub/Sub responses.
• Upgrade your dependencies: You're using version 1.1.0.RELEASE, which is quite old (released in 2019). Newer versions of Spring Cloud GCP Pub/Sub (3.x+) include improved retry logic, thread pool management, and bug fixes that reduce the risk of thread exhaustion.
• Monitor thread metrics: Use NewRelic to track thread pool metrics like active threads, queue length, and rejected tasks. Set up alerts for when thread usage approaches critical levels so you can act before an OOM occurs.

内容的提问来源于stack exchange，提问作者Panupong Kongarn