单线程任务流触发Protobuf序列化并发修改核心转储问题及规避咨询

在单线程任务流运行场景中，触发了核心转储（core dump），关联错误为：Protobuf: Protocol message was modified concurrently during serialization。

完整上下文日志 #

Program terminated with signal 6, Aborted.
#0  0x00007f871bb011f7 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install glibc-2.17-196.el7.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-51.el7_9.x86_64 libcom_err-1.42.9-19.el7.x86_64 libgcc-4.8.5-44.el7.x86_64 libgomp-4.8.5-44.el7.x86_64 libselinux-2.5-15.el7.x86_64 libstdc++-4.8.5-44.el7.x86_64 openssl-libs-1.0.2k-22.el7_9.x86_64 pcre-8.32-17.el7.x86_64 zlib-1.2.7-19.el7_9.x86_64
(gdb) bt
#0  0x00007f871bb011f7 in raise () from /lib64/libc.so.6
#1  0x00007f871bb028e8 in abort () from /lib64/libc.so.6
#2  0x00007f871c5e8935 in google::DumpStackTraceAndExit () at src/utilities.cc:159
#3  0x00007f871c5def9d in google::LogMessage::Fail () at src/logging.cc:1685
#4  0x00007f871c5e1c97 in google::LogMessage::SendToLog (this=0x7f820942b3a0) at src/logging.cc:1642
#5  0x00007f871c5deb33 in google::LogMessage::Flush (this=this@entry=0x7f820942b3a0) at src/logging.cc:1509
#6  0x00007f871c5e052e in google::LogMessageFatal::~LogMessageFatal (this=0x7f820942b3a0, __in_chrg=<optimized out>) at src/logging.cc:2256
#7  0x000000000145f8b3 in brpc::BaiduStreamingLogHandler (level=<optimized out>, filename=0x1a62e20 "google/protobuf/message_lite.cc", line=75,
    message="CHECK failed: (byte_size_before_serialization) == (byte_size_after_serialization): fs.interface.FsResponse was modified concurrently during serialization."
) at src/brpc/global.cpp:308
#8  0x00000000015ed651 in google::protobuf::internal::LogMessage::Finish (this=0x7f820942b440) at google/protobuf/stubs/common.cc:254
#9  0x00000000015edad8 in google::protobuf::internal::LogFinisher::operator= (this=this@entry=0x7f820942b42f, other=...) at google/protobuf/stubs/common.cc:267
#10 0x00000000015ff509 in google::protobuf::(anonymous namespace)::ByteSizeConsistencyError (byte_size_before_serialization=byte_size_before_serialization@entry=1706832,
    byte_size_after_serialization=<optimized out>, bytes_produced_by_serialization=bytes_produced_by_serialization@entry=1706862, message=...) at google/protobuf/message_lite.cc:75
#11 0x00000000015fff1f in google::protobuf::MessageLite::SerializePartialToArray (this=0x7f8387d1c048, data=0x7f7f81bbbe00, size=size@entry=1706832) at google/protobuf/message_lite.cc:488

规避方案 #

• 排查隐式并发修改：单线程场景不代表绝对无并发，检查是否存在信号处理函数、异步回调（如brpc框架的异步逻辑）在序列化过程中修改了同一个FsResponse对象。信号处理函数是常见的隐式并发来源，需确保其不会操作正在序列化的Protobuf实例。
• 序列化前生成对象副本：在调用Serialize系列方法前，调用message->Clone()生成对象的深拷贝，对副本执行序列化操作，彻底避免原对象被意外修改的风险。
• 检查延迟修改逻辑：排查代码中是否存在lambda表达式、延迟执行钩子（如defer）等逻辑，这些代码可能在序列化过程中触发字段修改，导致字节大小校验失败。
• 谨慎禁用并发检查（不推荐）：若确认无真实并发但触发误报，可通过编译选项PROTOBUF_DISABLE_CONSISTENCY_CHECKS关闭Protobuf的字节大小一致性检查，但此操作会掩盖真实的并发问题，仅作为临时应急方案。
• 检查brpc对象复用机制：日志中出现brpc相关逻辑，需确认框架是否存在请求对象复用（如连接池、对象池复用FsResponse），导致看似单线程的任务流中，对象被不同请求实例修改。

内容的提问来源于stack exchange，提问作者Leo