连接基于IAM认证的MSK时Kafka客户端出现断开警告问题

Key Issues Identified #

1. The Camel S3 Source Connector wasn't applying the MSK IAM security configurations from the worker properties.
2. Using Camel component properties (camel.component.kafka.*) in the Kafka Connect connector config is invalid, causing the "cannot find component with name kafka" error.

Step-by-Step Fixes #

1. Explicitly Set Producer Security Configs in Connector Properties #

Add the following lines to your CamelAwss3SourceConnector.properties file. This ensures the connector's internal Kafka producer uses MSK IAM authentication:

producer.security.protocol=SASL_SSL
producer.sasl.mechanism=AWS_MSK_IAM
producer.sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required awsProfileName="abc";
producer.sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler

2. Verify Bootstrap Servers Configuration #

Ensure your connect-standalone.properties points to the correct MSK broker endpoints on port 9098:

bootstrap.servers=b-1.<cluster-id>.kafka.<region>.amazonaws.com:9098,b-2.<cluster-id>.kafka.<region>.amazonaws.com:9098,b-3.<cluster-id>.kafka.<region>.amazonaws.com:9098

3. Confirm AWS IAM Permissions #

Double-check that the credentials in the "abc" profile (or the EC2 instance's attached IAM role) have these required permissions:

• kafka:DescribeCluster on the MSK cluster
• kafka:Connect on the MSK cluster
• kafka:WriteData on the target topic

4. Restart the Kafka Connect Worker #

Stop the running connect-standalone process and restart it with your updated configurations:

./bin/connect-standalone.sh config/connect-standalone.properties config/CamelAwss3SourceConnector.properties

Why This Works #

The Kafka console producer works because it directly uses the security configs from connect-standalone.properties. The Camel S3 Source Connector requires explicit producer-level settings in its own config file to ensure the underlying Kafka producer adopts the IAM auth mechanism. Camel component properties are intended for standalone Camel applications, not Kafka Connect connectors—using them here leads to configuration errors.

内容的提问来源于stack exchange，提问作者Dhananjay Shinde