本地Spring Boot连接Event Hub Kafka报错：authBytes序列化null

问题场景 #

本地机器向启用Kafka协议的标准层Azure Event Hub发送消息时，触发non-nullable field authBytes was serialized as null错误。已参考相关Stack Overflow帖子及微软官方文档配置，问题仍未解决，怀疑依赖配置存在问题，寻求解决思路。

当前配置信息 #

application.properties #

spring.cloud.function.definition=consume;supply
spring.cloud.stream.bindings.consume-in-0.destination=shub
spring.cloud.stream.bindings.consume-in-0.group=$Default
spring.cloud.stream.bindings.supply-out-0.destination=shub
spring.cloud.stream.kafka.binder.brokers=stsgdjhsacy.servicebus.windows.net:9093

Maven依赖配置 #

<properties>
    <java.version>17</java.version>
    <spring-cloud-azure.version>5.12.0</spring-cloud-azure.version>
    <spring-cloud.version>2023.0.1</spring-cloud.version>
</properties>
<dependencies>
    <dependency>
        <groupId>com.azure.spring</groupId>
        <artifactId>spring-cloud-azure-starter</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-web</artifactId>
    </dependency>

    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-starter-stream-kafka</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-test</artifactId>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-stream-test-binder</artifactId>
        <scope>test</scope>
    </dependency>
</dependencies>

错误堆栈信息 #

2024-05-29T18:41:51.075+01:00  INFO 21320 --- [hubproducer]
[onPool-worker-2] c.azure.identity.ChainedTokenCredential  : Azure
Identity => Attempted credential AzureDeveloperCliCredential returns a
token 2024-05-29T18:41:51.109+01:00  INFO 21320 --- [hubproducer] [<br />
main] .o.i.e.ExpiringCredentialRefreshingLogin : Successfully logged
in.

2024-05-29T18:41:52.411+01:00  WARN 21320 --- [hubproducer] [| adminclient-1] o.apache.kafka.common.network.Selector   : [AdminClient clientId=adminclient-1] Unexpected error from sto....cy.servicebus.windows.net/266.066.142.1954 (channelId=-1); closing connection

java.lang.RuntimeException: non-nullable field authBytes was
serialized as null    at
org.apache.kafka.common.message.SaslAuthenticateResponseData.read(SaslAuthenticateResponseData.java:137)
~[kafka-clients-3.6.2.jar:na]     at
org.apache.kafka.common.message.SaslAuthenticateResponseData.(SaslAuthenticateResponseData.java:86)
~[kafka-clients-3.6.2.jar:na]     at
org.apache.kafka.common.requests.SaslAuthenticateResponse.parse(SaslAuthenticateResponse.java:81)
~[kafka-clients-3.6.2.jar:na]     at
org.apache.kafka.common.requests.AbstractResponse.parseResponse(AbstractResponse.java:187)
~[kafka-clients-3.6.2.jar:na]

解决思路 #

• 替换专用的Azure Event Hub Kafka绑定器：移除通用的spring-cloud-starter-stream-kafka依赖，改用Azure官方提供的spring-cloud-azure-starter-stream-eventhubs-kafka，该绑定器针对Azure Event Hub的Kafka兼容特性做了适配，能避免通用Kafka绑定器与Azure认证逻辑的冲突。
• 完善Kafka认证配置：在application.properties中补充以下认证相关配置，确保OAuth2认证流程与Azure Event Hub兼容：

  spring.cloud.stream.kafka.binder.configuration.sasl.mechanism=OAUTHBEARER
  spring.cloud.stream.kafka.binder.configuration.security.protocol=SASL_SSL
  spring.cloud.stream.kafka.binder.configuration.sasl.jaas.config=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required;
  spring.cloud.stream.kafka.binder.configuration.sasl.login.callback.handler.class=com.azure.spring.cloud.autoconfigure.eventhub.kafka.AzureEventHubKafkaOAuthBearerLoginCallbackHandler
  

• 排查依赖版本冲突：检查项目中kafka-clients的版本是否统一，若存在多版本依赖，可通过Maven的dependency:tree命令排查，并在相关依赖中排除冲突版本，确保使用与Azure Event Hub兼容的Kafka客户端版本。
• 验证Azure身份认证权限：虽然日志显示成功获取token，但需确认当前使用的Azure凭据（AzureDeveloperCliCredential）拥有Event Hub的发送/接收权限，避免因权限不足导致的认证响应异常。

内容的提问来源于stack exchange，提问作者Kris Swat