运行Kafka Connect分布式模式是否必须使用多台EC2/VM？

核心问题解答 #

No, you don't need multiple EC2/VM instances to run Kafka Connect in distributed mode. Distributed mode is built for scalability and high availability, but a single node can absolutely run Connect in this mode without issues. Your current failure isn't tied to the number of nodes—it's a connectivity problem between your Connect worker and the MSK cluster.

Troubleshooting Your Connection Failure #

Your error logs clearly point to a timeout when trying to reach the MSK cluster:

org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment.

Even though telnet confirms basic TCP connectivity, Kafka's client protocol has additional requirements. Here's how to resolve this:

1. Check MSK Security Configuration
   MSK clusters often enable authentication (like IAM or SASL/SCRAM) by default. If your MSK cluster uses any form of authentication, add the corresponding security settings to your connect-distributed.properties:

   • For IAM authentication:

     security.protocol=SASL_SSL
     sasl.mechanism=AWS_MSK_IAM
     sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;
     sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler
     

   • Ensure you've installed the MSK IAM auth library in your Connect plugin path if using this method.

2. Validate Basic Kafka Client Connectivity
   Test if a standard Kafka client can reach your MSK cluster from the EC2 instance. Run this command (replace with your broker addresses):

   kafka-topics.sh --list --bootstrap-servers <your-msk-brokers>:9092
   

   If this command also times out, the issue is with client-to-broker connectivity (not specific to Connect), so focus on network/security settings.

3. Verify MSK Listener Accessibility

   • If using private VPC endpoints, confirm your EC2 instance is in the same VPC and subnet as the MSK cluster.
   • If using public access, ensure MSK has public endpoints enabled and your EC2 can reach those public IPs.

4. Check Security Groups & Network ACLs

   • Confirm your EC2's security group allows outbound traffic to MSK's 9092 port (or the port your MSK cluster uses).
   • Confirm MSK's security group allows inbound traffic from your EC2's IP address or security group on the relevant port.

5. Review Connect Worker Configuration
   Double-check these critical settings in connect-distributed.properties:

   • Ensure bootstrap.servers uses the exact broker addresses from your MSK cluster (you can retrieve these from the AWS MSK console).
   • While not the root cause here, offset.storage.replication.factor=1 is not recommended for production—if your MSK cluster has 3+ nodes, set this to 3 for better durability.

Once you fix the connection issue between Connect and MSK, your distributed mode (even single-node) should start successfully, and you can then deploy your Elasticsearch sink connector.

内容的提问来源于stack exchange，提问作者Sudarshan kumar