macOS环境下confluent-kafka 1.7.0连接外部Kafka 2.6.1时报错Disconnected while requesting ApiVersion的问题咨询

Hey there, let's tackle this connection issue you're facing with confluent_kafka. I've looked through your environment details, code, and debug logs, and here's what's going on and how to fix it:

Context of Your Issue #

You're using confluent_kafka==1.7.0 on macOS to connect to an external Kafka 2.6.1 cluster over PLAINTEXT. Calling list_topics() throws the Disconnected while requesting ApiVersion error, but kafka-python and Kafka tools work fine.

Root Cause Analysis #

From the debug logs, we can see the connection to the broker does get established (Connected to ipv4#10.103.68.24:9092), but the API version negotiation step fails abruptly. This usually happens due to:

• A too-short timeout for the API version request (librdkafka's default might be too strict for your network)
• Mismatched protocol negotiation logic between librdkafka 1.7.0 and Kafka 2.6.1
• Subtle network constraints (like TCP buffer limits on macOS)

Solutions to Try #

Let's go through these fixes one by one, starting with the most likely to work:

1. Increase API Version Request Timeout & Explicitly Set Broker Version #

Librdkafka's default timeout for API version requests might be too brief if there's any network latency. Explicitly specifying your broker version also skips the auto-negotiation step that's failing:

from confluent_kafka import Consumer
c = Consumer({
    'group.id': 'test',
    "bootstrap.servers": "my-external-host:9092",
    # Extend timeout for API version request to 10 seconds
    'api.version.request.timeout.ms': 10000,
    # Tell librdkafka exactly what broker version to use
    'broker.version.fallback': '2.6.1',
    'api.version.fallback.ms': 0
})
c.list_topics()

2. Adjust TCP Socket Parameters (macOS-Specific) #

macOS has default TCP buffer sizes that might be too small for librdkafka's needs. Tweaking these can help with unstable connections:

from confluent_kafka import Consumer
c = Consumer({
    'group.id': 'test',
    "bootstrap.servers": "my-external-host:9092",
    'socket.send.buffer.bytes': 1024000,  # 1MB send buffer
    'socket.receive.buffer.bytes': 1024000,  # 1MB receive buffer
    'socket.timeout.ms': 30000  # 30-second socket timeout
})
c.list_topics()

3. Disable Automatic API Version Request (Last Resort) #

If the above fixes don't work, you can disable the API version request entirely (safe for Kafka 0.10.0+ which your 2.6.1 cluster is):

from confluent_kafka import Consumer
c = Consumer({
    'group.id': 'test',
    "bootstrap.servers": "my-external-host:9092",
    'api.version.request': False,
    'broker.version.fallback': '2.6.1'
})
c.list_topics()

Why Does kafka-python Work? #

The key difference is that kafka-python is a pure Python implementation, while confluent_kafka uses the C-based librdkafka under the hood. Their network handling and protocol negotiation logic are totally separate—kafka-python has more lenient default timeouts and handles network fluctuations differently, which is why it connects successfully.

内容的提问来源于stack exchange，提问作者Denis Sered