Great job diagnosing the core issue here—your analysis is spot-on. Let's walk through why this is happening and how to fix it properly:

First, let's confirm your key observations are correct:

• With AMQP 0-9-1 without publish confirms, your publisher treats messages as "sent" the moment they leave the application's network stack, not when RabbitMQ actually confirms receipt.
• The TCP connections between AWS Classic Load Balancer and your RabbitMQ nodes are separate from the connections between your publisher app and the LB. When RabbitMQ enters Flow Control, it stops accepting new messages from the LB—but your publisher keeps sending at full speed, causing messages to pile up in the LB's in-memory buffers.

The root cause of LB disconnects/IP switches #

AWS Classic Load Balancers are made up of multiple backend nodes (each with their own IP). When:

1. RabbitMQ's Flow Control causes the LB's outgoing buffer to fill up completely, the TCP connection between LB and RabbitMQ gets stuck in a blocked state.
2. This blocked connection hits the LB's TCP idle timeout (default 60 seconds) or exceeds buffer limits, the LB will terminate the stale connection.
3. The LB then spins up a new connection from a different LB node (hence the IP change in your RabbitMQ logs) to continue routing traffic—but all messages that were buffered in the old LB connection are permanently lost, since Classic LBs don't persist queued messages.

Your RabbitMQ logs confirm exactly this sequence:

=WARNING REPORT==== 6-Jan-2018::10:35:50 === closing AMQP connection <0.30342.375> (10.1.1.250:29564 -> 10.1.1.223:5672): client unexpectedly closed TCP connection
=INFO REPORT==== 6-Jan-2018::10:35:51 === accepting AMQP connection <0.29123.375> (10.1.1.22:1886 -> 10.1.1.223:5672)

The IPs 10.1.1.250 and 10.1.1.22 are different nodes within your Classic LB cluster—proof that the LB dropped a stuck connection and replaced it with a new one.

Is this tied to high throughput/Flow Control? #

Absolutely. Your high message rate overwhelms RabbitMQ's processing capacity, triggering Flow Control. Without a mechanism to signal back to the publisher to slow down, the LB becomes a bottleneck where messages pile up until the connection fails. This chain reaction directly causes both the Flow Control and the LB connection issues.

Here's what you can do to resolve this while keeping the benefits of AWS LB:

• Enable RabbitMQ Publish Confirms immediately
  This is the most critical fix. Modify your publisher to use AMQP publish confirms:

  1. Call channel.confirmSelect() on your AMQP channel to enable confirms.
  2. Wait for a confirmation from RabbitMQ before sending the next message (or batch of messages).
     This ensures your publisher stops sending when RabbitMQ is under load (and in Flow Control), preventing the LB from accumulating messages that will get lost on disconnect.

• Tune LB and RabbitMQ timeouts

  • Increase the Classic LB's TCP idle timeout: You can set this up to 3600 seconds via the AWS Console/CLI. This reduces the chance of the LB dropping connections during temporary Flow Control periods.
  • Sync RabbitMQ's heartbeat setting: Set RabbitMQ's heartbeat parameter (in rabbitmq.conf) to be shorter than the LB's timeout (e.g., 120 seconds if LB timeout is 300 seconds). This keeps connections alive and avoids false idle timeout triggers.

• Switch to AWS Network Load Balancer (NLB) if possible
  NLBs have better TCP performance, larger buffers, and support static IPs. They're designed for high-throughput, low-latency traffic like AMQP, and are less likely to drop connections under load compared to Classic LBs.

• Rate-limit your publisher
  Add a dynamic rate limiter to your publisher that monitors RabbitMQ's health (via the RabbitMQ Management API, checking for Flow Control status or queue depth). When RabbitMQ is under stress, throttle the message send rate to match what the cluster can handle.

• Optimize RabbitMQ to reduce Flow Control triggers

  • Ensure your RabbitMQ cluster has enough resources (CPU, memory, disk) to handle your message load.
  • Tune queue parameters (like prefetch count for consumers) to speed up message processing and reduce backlog.
  • Enable lazy queues if appropriate, to offload queue storage to disk and reduce memory pressure.

内容的提问来源于stack exchange，提问作者Marko Vranjkovic