Protostuff转Protobuf字节Map兼容问题及Delegate实现咨询

问题背景 #

Protostuff的ProtobufIOUtil工具可将Java对象转为Protobuf字节，但处理Map类型时，生成的字节结构与原生Protobuf存在差异，导致原生Protobuf解析出异常的Map数据。

代码示例 #

Java POJO类 #

@Getter
@Setter
public class User {
    @Tag(1)
    private long uid;

    @Tag(2)
    private String name;

    @Tag(3)
    private Map<Long, Integer> map = new HashMap<>(); // Map类型字段
}

原生Protobuf定义 #

syntax="proto3";

message User {
  int64 uid = 1;
  string name = 2;
  map<int64, int32> map = 3;
}

测试代码 #

public class Test {
    public static void main(String[] args) throws InvalidProtocolBufferException {
        User user = new User();
        user.setUid(1);
        user.setName("s");
        Map<Long, Integer> map = new HashMap<Long, Integer>();
        map.put(1L, 1);
        user.setMap(map);
        Schema<User> schema = RuntimeSchema.getSchema(User.class);
        byte[] bytes = ProtobufIOUtil.toByteArray(user, schema, LinkedBuffer.allocate());
        System.out.println(Arrays.toString(bytes));

        UserOuterClass.User pbUser = UserOuterClass.User.newBuilder()
                .setUid(1)
                .setName("s")
                .putMap(1, 1)
                .build();
        System.out.println(Arrays.toString(pbUser.toByteArray()));
        UserOuterClass.User newPbUser = UserOuterClass.User.parseFrom(bytes);
        System.out.println(newPbUser.toString());
    }
}

控制台输出 #

[8, 1, 18, 1, 115, 26, 6, 10, 4, 8, 1, 16, 1]
[8, 1, 18, 1, 115, 26, 4, 8, 1, 16, 1]
uid: 1
name: "s"
map {
  key: 0
  value: 0
}

依赖配置 #

<dependency>
    <groupId>io.protostuff</groupId>
    <artifactId>protostuff-core</artifactId>
    <version>1.7.4</version>
</dependency>
<dependency>
    <groupId>io.protostuff</groupId>
    <artifactId>protostuff-runtime</artifactId>
    <version>1.7.4</version>
</dependency>
<dependency>
    <groupId>io.protostuff</groupId>
    <artifactId>protostuff-runtime-registry</artifactId>
    <version>1.7.4</version>
</dependency>

<dependency>
    <groupId>com.google.protobuf</groupId>
    <artifactId>protobuf-java</artifactId>
    <version>3.21.8</version>
</dependency>

差异分析 #

原生Protobuf对Map的编码结构为fieldTag, eleLen, eleData, fieldTag, eleLen, eleData，而Protostuff转换后的字节会额外增加一层fieldTag, fieldLen包裹整个Map内容，导致原生Protobuf无法正确识别Map键值对，解析出异常数据。

解决方案 #

1. 是否有配置可避免额外的fieldTag和fieldLen？ #

Protostuff默认的RuntimeSchema将Map当作嵌套消息处理，没有直接配置可以关闭这层额外包裹。要实现与原生Protobuf一致的Map序列化，需要自定义Delegate覆盖默认逻辑。

2. 自定义Map Delegate的实现方案 #

以下是针对Map<Long, Integer>类型的Delegate实现，可根据实际键值类型调整泛型和编码逻辑：

import io.protostuff.Input;
import io.protostuff.Output;
import io.protostuff.Pipe;
import io.protostuff.Schema;
import io.protostuff.WireFormat;
import io.protostuff.runtime.Delegate;

import java.io.IOException;
import java.util.HashMap;
import java.util.Map;

public class LongIntegerMapDelegate implements Delegate<Map<Long, Integer>> {

    @Override
    public WireFormat.FieldType getFieldType() {
        return WireFormat.FieldType.MESSAGE;
    }

    @Override
    public Map<Long, Integer> readFrom(Input input) throws IOException {
        Map<Long, Integer> map = new HashMap<>();
        while (input.readFieldNumber() != 0) {
            int fieldNumber = input.getFieldNumber();
            if (fieldNumber == 1) {
                input.readMessage(new EntrySchema(), map);
            } else {
                input.skipField();
            }
        }
        return map;
    }

    @Override
    public void writeTo(Output output, int number, Map<Long, Integer> value, boolean repeated) throws IOException {
        if (value == null || value.isEmpty()) {
            return;
        }
        for (Map.Entry<Long, Integer> entry : value.entrySet()) {
            output.writeMessage(number, new EntrySchema(), entry, false);
        }
    }

    @Override
    public void transfer(Pipe pipe, Input input, Output output, int number, boolean repeated) throws IOException {
        Map<Long, Integer> map = readFrom(input);
        writeTo(output, number, map, repeated);
    }

    private static class EntrySchema implements Schema<Map.Entry<Long, Integer>> {
        @Override
        public Map.Entry<Long, Integer> newMessage() {
            return new HashMap.SimpleEntry<>(0L, 0);
        }

        @Override
        public void mergeFrom(Input input, Map.Entry<Long, Integer> message) throws IOException {
            while (input.readFieldNumber() != 0) {
                int fieldNumber = input.getFieldNumber();
                if (fieldNumber == 1) {
                    message.setKey(input.readInt64());
                } else if (fieldNumber == 2) {
                    message.setValue(input.readInt32());
                } else {
                    input.skipField();
                }
            }
        }

        @Override
        public void writeTo(Output output, Map.Entry<Long, Integer> message) throws IOException {
            if (message.getKey() != null) {
                output.writeInt64(1, message.getKey(), false);
            }
            if (message.getValue() != null) {
                output.writeInt32(2, message.getValue(), false);
            }
        }

        @Override
        public String getFieldName(int number) {
            return number == 1 ? "key" : "value";
        }

        @Override
        public int getFieldNumber(String name) {
            return "key".equals(name) ? 1 : "value".equals(name) ? 2 : 0;
        }

        @Override
        public Class<Map.Entry<Long, Integer>> typeClass() {
            return (Class<Map.Entry<Long, Integer>>) (Class<?>) Map.Entry.class;
        }

        @Override
        public boolean isInitialized(Map.Entry<Long, Integer> message) {
            return message.getKey() != null && message.getValue() != null;
        }

        @Override
        public String messageName() {
            return "Entry";
        }

        @Override
        public String messageFullName() {
            return "Entry";
        }

        @Override
        public void validate(Map.Entry<Long, Integer> message) throws IOException {}
    }
}

注册Delegate到RuntimeSchema：

RuntimeSchema.registerDelegate(Map.class, new LongIntegerMapDelegate());

配置后，Protostuff序列化的Map数据将与原生Protobuf结构一致，原生Protobuf可解析出正确的键值对。

内容的提问来源于stack exchange，提问作者Criwran