RabbitMQ服务器触发资源告警后出现额外TCP连接异常

Alright, let's break down this problem you're facing with RabbitMQ 3.6.0—this is a classic issue tied to old version bugs and memory pressure interactions. Here's why it's happening and how to fix it step by step:

1. 3.6.0's Critical Connection Management Bug
   RabbitMQ 3.6.x is an end-of-life version with well-documented flaws in connection handling under memory pressure. When the server hits a memory alarm, it throttles new connections and struggles to process existing connection cleanup. Clients, configured with a 60-second heartbeat, don't get timely heartbeat responses from the overloaded server, so they assume the connection is dead and spin up new ones. The old connections, however, aren't properly closed by the server (due to resource constraints), resulting in thousands of orphaned "channel-less" connections piling up.

2. Heartbeat + Memory Pressure Feedback Loop
   Even though both server and client have heartbeats enabled, the memory-starved server can't keep up with sending heartbeat ACKs. This triggers client reconnection logic, but the server can't garbage-collect the stale connections fast enough. Each failed heartbeat leads to a new connection, creating a loop that blows up your connection count.

3. Mixed Cluster Versions
   You mentioned upgrading one node but leaving this one on 3.6.0. Cluster nodes with mismatched versions often have inconsistent connection handling logic, which can exacerbate connection leaks and cleanup failures.

• Purge Orphaned Connections Manually
  Use the RabbitMQ Management UI: navigate to the "Connections" tab, filter for connections with Channels: 0, select all, and click "Close". If the UI is slow (due to load), use the command line:

  # List all connections with channel count 0
  rabbitmqctl list_connections name channels | grep -E "\s0$" | awk '{print $1}' > stale_connections.txt
  # Close each stale connection
  while read conn; do rabbitmqctl close_connection "$conn" "Cleaning up stale connections post-memory alarm"; done < stale_connections.txt
  

• Temporarily Adjust Memory Threshold
  Ease the server's memory pressure to let it process cleanup tasks:

  # Set memory high watermark to 70% of system memory (adjust based on your server's specs)
  rabbitmqctl set_vm_memory_high_watermark 0.7
  

  This will lift the memory alarm temporarily, allowing the server to catch up on connection cleanup.

• Limit New Connections Temporarily
  Add a connection limit to prevent further growth while you fix the root issue. Edit your RabbitMQ config file (usually rabbitmq.config or advanced.config) and add:

  [{rabbit, [{tcp_listeners, [{"0.0.0.0", 5672, [{connection_limit, 10000}]}]}]}].
  

  Restart RabbitMQ to apply the change, then adjust the limit once you're stable.

• Upgrade to a Supported RabbitMQ Version
  This is non-negotiable. 3.6.x has been end-of-life since 2020, and every subsequent version (3.8+, especially 3.10+) fixes critical connection management, memory handling, and heartbeat bugs. Make sure to upgrade this node to match the already-upgraded node in your cluster—follow official upgrade steps, back up your data first, and avoid mixed versions long-term.

• Optimize Heartbeat and Reconnection Logic

  • Lower the heartbeat interval slightly (e.g., 30 seconds) to detect dead connections faster without increasing overhead.
  • Configure your clients to use exponential backoff for reconnections. Instead of immediately spamming new connections when one fails, add increasing delays between retries (e.g., 1s, 2s, 4s, 8s) to avoid overwhelming the server during pressure events.

• Auto-Clean Stale Connections
  Enable and configure connection tracking to auto-close idle, channel-less connections. In your RabbitMQ config, add:

  [{rabbit, [{connection_closed_timeout, 300}]}].
  

  This will automatically close connections that have no channels for 5 minutes (300 seconds).

• Enhance Monitoring & Alerts
  Set up alerts for:

  • Connection count exceeding 80% of your normal baseline (e.g., 4800 connections)
  • Memory usage hitting 70% of the high watermark
  • Channel count per connection (flag any connections with 0 channels that persist longer than 5 minutes)
    This lets you intervene before the issue spirals into a full-blown connection flood.

内容的提问来源于stack exchange，提问作者flam3