Kafka S3 Connect报错‘Value is not Struct type’求助
Confluent S3 Sink Connector报错:Value is not Struct type(FieldPartitioner使用问题)
最近我在使用Confluent Open Source v4.1部署S3 Sink Connector时遇到了棘手问题——之前运行正常的配置,现在抛出了分区编码错误,完全无法处理消息。
我的配置与操作步骤
1. 加载S3 Sink Connector的命令
我用confluent load命令加载了Connector,配置参数如下:
confluent load s3-sink { "name": "s3-sink", "config": { "connector.class": "io.confluent.connect.s3.S3SinkConnector", "tasks.max": "1", "topics": "s3_topic", "s3.region": "us-east-1", "s3.bucket.name": "some_bucket", "s3.part.size": "5242880", "flush.size": "1", "storage.class": "io.confluent.connect.s3.storage.S3Storage", "format.class": "io.confluent.connect.s3.format.json.JsonFormat", "schema.generator.class": "io.confluent.connect.storage.hive.schema.DefaultSchemaGenerator", "partitioner.class": "io.confluent.connect.storage.partitioner.FieldPartitioner", "schema.compatibility": "NONE", "partition.field.name": "f1", "key.converter": "org.apache.kafka.connect.json.JsonConverter", "value.converter": "org.apache.kafka.connect.json.JsonConverter", "key.converter.schemas.enable": "false", "value.converter.schemas.enable": "false", "name": "s3-sink" }, "tasks": [ { "connector": "s3-sink", "task": 0 } ], "type": null }
2. 发送测试消息
通过kafka-console-producer发送了一条JSON格式的消息:
{"f1":"partition","data":"some data"}
遇到的错误日志
结果在Connect的日志里看到了这样的错误:
[2018-05-16 16:32:05,150] ERROR Value is not Struct type. (io.confluent.connect.storage.partitioner.FieldPartitioner:67) [2018-05-16 16:32:05,150] ERROR WorkerSinkTask{id=s3-sink-0} Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted. (org.apache.kafka.connect.runtime.WorkerSinkTask:515) io.confluent.connect.storage.errors.PartitionException: Error encoding partition.
这个配置在之前的版本里是完全正常的,升级到v4.1之后就出问题了。有没有大佬能帮忙分析一下问题出在哪,怎么解决?
内容的提问来源于stack exchange,提问作者Vova l
相关产品推荐
相关产品推荐

