如何用Python Protobuf库动态加载.desc/proto文件并反射字段？

问题原因 #

报错UnicodeDecodeError是因为.desc文件中google.protobuf.FileDescriptorProto.name字段包含非UTF-8编码的字节（0xf8），protobuf默认要求字符串字段使用UTF-8编码，导致解析时失败。

解决方案 #

方案1：修复.desc文件生成过程 #

优先检查生成.desc文件的脚本或命令，确保所有proto文件中的字符串（如包名、文件名）使用UTF-8编码，重新生成符合规范的.desc文件。

方案2：兼容非UTF-8编码的.desc文件 #

如果无法重新生成文件，可以手动解析并修复编码错误后再导入：

from google.protobuf import descriptor_pb2
from google.protobuf import symbol_database, reflection
from google.protobuf.internal import _descriptor_pool

def gen1(desc_file, db=None):
    if not db:
        db = symbol_database.Default()

    # 读取并反序列化FileDescriptorSet
    fds = descriptor_pb2.FileDescriptorSet()
    with open(desc_file, 'rb') as fh:
        fds.ParseFromString(fh.read())
    
    # 修复每个文件描述符的name字段编码错误
    for prot in fds.file:
        # 尝试用Latin-1兼容解码，替换无法识别的字符
        try:
            prot.name = prot.name.encode('latin-1').decode('utf-8', errors='replace')
        except (UnicodeEncodeError, UnicodeDecodeError):
            # 极端情况直接替换为占位符
            prot.name = f"fixed_{hash(prot.name)}"
    
    # 导入处理后的描述符到池
    pool = _descriptor_pool.DescriptorPool()
    generated = {}
    for prot in fds.file:
        fd = pool.Add(prot)
        for name in fd.message_types_by_name:
            mdesc = fd.message_types_by_name[name]
            Klass = reflection.MakeClass(mdesc)
            generated[(fd.package, name)] = db.RegisterMessage(Klass)

    return generated

依赖树构建实现 #

基于动态解析的描述符，递归遍历消息字段和嵌套类型，构建父-子结构的依赖树：

def build_message_dependency_tree(message_desc):
    """递归构建单个消息的依赖树"""
    tree_node = {
        "node_type": "message",
        "name": message_desc.name,
        "fields": [],
        "nested_messages": []
    }

    # 遍历字段，记录字段类型信息
    for field in message_desc.fields:
        field_info = {
            "name": field.name,
            "field_type": field.type_name.split(".")[-1] if field.type == field.TYPE_MESSAGE else field.GetTypeName()
        }
        tree_node["fields"].append(field_info)

    # 递归处理嵌套消息
    for nested_msg in message_desc.nested_types:
        tree_node["nested_messages"].append(build_message_dependency_tree(nested_msg))

    return tree_node

# 使用示例：在gen1生成描述符后构建全量依赖树
def build_full_dependency_tree(generated_map):
    full_tree = []
    for (package, name), klass in generated_map.items():
        msg_desc = klass.DESCRIPTOR
        tree_node = build_message_dependency_tree(msg_desc)
        tree_node["package"] = package
        full_tree.append(tree_node)
    return full_tree

使用流程 #

1. 遍历指定目录下所有.desc文件，逐个调用gen1并将结果合并到同一个字典中
2. 调用build_full_dependency_tree传入合并后的字典，生成完整的依赖树

内容的提问来源于stack exchange，提问作者0x SLC