使用Apache Pulsar Lakehouse Sink Connector对接Minio触发IllegalArgumentException

我在部署并运行Apache Pulsar的Lakehouse Sink Connector时，触发了java.lang.IllegalArgumentException，环境配置与操作流程如下：

• 基于docker-compose.yaml部署Pulsar、Pulsar Manager与Minio服务；
• 下载pulsar-io-lakehouse的NAR包并复制到Pulsar容器的connectors目录；
• 在Minio中创建名为lakehouse的存储桶；
• 参考官方文档编写JSON格式的sink配置文件sink-connector-config.json，指定Minio的tablePath；
• 在容器内执行pulsar-admin sink localrun命令启动连接器，出现如下错误：

2022-09-06T16:53:08,396+0000 [main] INFO  org.apache.pulsar.functions.utils.io.ConnectorUtils - Found connector ConnectorDefinition(name=lakehouse, description=Lakehouse connectors, sourceClass=org.apache.pulsar.ecosystem.io.lakehouse.SourceConnector, sinkClass=org.apache.pulsar.ecosystem.io.lakehouse.SinkConnector, sourceConfigClass=org.apache.pulsar.ecosystem.io.lakehouse.SourceConnectorConfig, sinkConfigClass=org.apache.pulsar.ecosystem.io.lakehouse.SinkConnectorConfig) from /pulsar/connectors/pulsar-io-lakehouse-2.9.3.7-cloud.nar
2022-09-06T16:53:44,562+0000 [main] ERROR org.apache.pulsar.functions.LocalRunner - Encountered error starting localrunner
java.lang.IllegalArgumentException: Could not validate sink config: Cannot construct instance of `org.apache.pulsar.ecosystem.io.lakehouse.SinkConnectorConfig` (no Creators, like default constructor, exist): abstract types either need to be mapped 
to concrete types, have custom deserializer, or contain additional type information
 at [Source: UNKNOWN; byte offset: #UNKNOWN]
        at org.apache.pulsar.functions.utils.SinkConfigUtils.validateSinkConfig(SinkConfigUtils.java:594) ~[org.apache.pulsar-pulsar-functions-utils-2.9.3.jar:2.9.3]
        at org.apache.pulsar.functions.utils.SinkConfigUtils.validateAndExtractDetails(SinkConfigUtils.java:441) ~[org.apache.pulsar-pulsar-functions-utils-2.9.3.jar:2.9.3]
        at org.apache.pulsar.functions.LocalRunner.start(LocalRunner.java:439) ~[org.apache.pulsar-pulsar-functions-local-runner-original-2.9.3.jar:2.9.3]
        at org.apache.pulsar.functions.LocalRunner.main(LocalRunner.java:198) [org.apache.pulsar-pulsar-functions-local-runner-original-2.9.3.jar:2.9.3]
root@pulsar:/pulsar#

经排查，问题根源是通过CLI发送消息时未定义Schema，导致连接器无法正确解析配置与消息结构。

通过以下步骤解决问题，最终成功将数据写入Delta Lake：

• 为目标主题创建Schema；
• 将连接器配置文件改为YAML格式编写；
• 编写Python脚本，携带Schema发送消息；
• 编写bash脚本自动化完成连接器的部署与测试流程。

内容的提问来源于stack exchange，提问作者sejuba