构建含数组类型值映射的Avro POJO记录报错，求解决方法

Avro Schema定义 #

{
  "type": "record",
  "name": "ASillySchema",
  "namespace": "com.my.company.redacted",
  "fields": [
    {
      "name": "my_overly_complex_map",
      "type": [
        "null",
        {
          "type": "map",
          "values": [
            "null",
            "string",
            "boolean",
            "int",
            "long",
            "float",
            "double",
            {
              "type": "array",
              "items": [
                "null",
                "string",
                "boolean",
                "int",
                "long",
                "float",
                "double"
              ]
            }
          ]
        }
      ],
      "default": null
    }
  ]
}

尝试的Java代码 #

Map<CharSequence, Object> myOverlyComplexMap = new HashMap<>();
List<Object> mixedTypeObjectArray = Arrays.asList(
                null,
                "a string",
                true,
                35,
                20L,
                201f,
                202d
);
myOverlyComplexMap.put("arrayKey", mixedTypeObjectArray);
ASillySchema testRecord = ASillySchema.newBuilder()
                             .setMyOverlyComplexMap(myOverlyComplexMap)
                             .build();       

// send testRecord to topic using Java kafka producer...

报错信息 #

Caused by: org.apache.avro.UnresolvedUnionException: Not in union ["null","string","boolean","int","long","float","double"]: [null, a string, true, 35, 20, 201.0, 202.0]
    at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:896)
    at org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:272)
    at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:143)
    at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:83)
    at org.apache.avro.generic.GenericDatumWriter.writeMap(GenericDatumWriter.java:305)
    at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:140)
...

疑问 #

如何正确构建这个要发送的Java对象？

错误原因 #

你用普通Java List来对应Avro Schema中的array类型，但Avro处理Union类型时，无法将普通List识别为Union定义里的array选项，因此抛出类型不匹配异常。必须使用Avro提供的GenericData.Array实例并关联对应array Schema，才能让Avro正确识别这是Union中的array类型。

正确代码示例 #

// 获取ASillySchema的元数据Schema
Schema recordSchema = ASillySchema.getClassSchema();
// 取出my_overly_complex_map字段的Union类型Schema
Schema mapUnionSchema = recordSchema.getField("my_overly_complex_map").schema();
// 获取Union中的map类型（索引1，因为Union第一个元素是null）
Schema mapSchema = mapUnionSchema.getTypes().get(1);
// 获取map值的Union类型，再取出其中的array类型（索引7，对应Union最后一个元素）
Schema arraySchema = mapSchema.getValueType().getTypes().get(7);

// 创建Avro标准的Array实例，传入对应的array Schema
GenericData.Array<Object> mixedTypeArray = new GenericData.Array<>(arraySchema);
// 向数组添加各类元素
mixedTypeArray.add(null);
mixedTypeArray.add("a string");
mixedTypeArray.add(true);
mixedTypeArray.add(35);
mixedTypeArray.add(20L);
mixedTypeArray.add(201f);
mixedTypeArray.add(202d);

// 构建map并放入array
Map<CharSequence, Object> myOverlyComplexMap = new HashMap<>();
myOverlyComplexMap.put("arrayKey", mixedTypeArray);

// 构建测试用的ASillySchema实例
ASillySchema testRecord = ASillySchema.newBuilder()
                             .setMyOverlyComplexMap(myOverlyComplexMap)
                             .build();

// 发送testRecord到Kafka Topic...

说明 #

1. Schema解析：通过生成的ASillySchema类获取整体记录Schema，逐层解析出map值Union中的array类型Schema。
2. GenericData.Array作用：这是Avro专门用于表示数组的容器类，必须传入对应array Schema，才能让序列化器正确识别类型并匹配Union选项。
3. 元素添加：直接用add()方法向GenericData.Array中添加各类元素即可，Avro会根据array的items Union定义自动处理每个元素的类型匹配。

内容的提问来源于stack exchange，提问作者qlangman