使用FlatBufferBuilder序列突破2GB限制仍触发断言错误的问题

问题描述 #

需要向FlatBuffer写入超过2GB的数据，尝试通过拆分多个FlatBufferBuilder实例绕过单Buffer的2GB限制。结构体ValidationReportT包含约350万条数据的vector，序列化后会超出限制。

相关结构体定义：

// flatbuffer struct contains a vector that can grow ~3.5 million in size(which consequently exceeds 2GB)
struct ValidationReportT : public flatbuffers::NativeTable
{
  std::vector<flatbuffers::unique_ptr<ValidationDefectT>> defects{};
}

拆分逻辑代码：

void createReport(std::vector<flatbuffers::FlatBufferBuilder>& fbbDefectsVec)
{ 

    ValidationReportT report;          // report has all defects already populated
    ValidationReportT partialReport;   // Partial report captures blocks of max_elems, writes to flatbufferBuilder class, then the partialReport defects vector is cleared

    if (report.defects.size())
    {
        int size       = report.defects.size();
        auto elem_size  = sizeof(*report.defects[0]);
        float32_t max_elems  = FLATBUFFERS_MAX_BUFFER_SIZE / (float32_t)elem_size;
        auto fbbsNeeded = size < max_elems ? 1 : (int)ceil(size / max_elems);

        fbbDefectsVec.resize(fbbsNeeded);
        int offset = 0;
        int idx    = 0;

        size_t start = offset;
        size_t end = offset + max_elems;     //Loop over blocks of max elems

        while (size > max_elems)
        {
            for (size_t i = start; i < end; i++)
            {
                partialReport.defects.push_back(std::move(report.defects[i]));
            }
            offset += max_elems;
            size -= max_elems;

            fbbDefectsVec[idx].Finish(
                validation::ValidationReport::Pack(fbbDefectsVec[idx], &partialReport));
            idx++;
            start = offset;
            end = offset + max_elems;
            partialReport.defects.clear();
        }

        if(size > 0)
        {
            //Copy the remaining defects
            //Set the loop bounds
            if(max_elems >= report.defects.size()) //All defects can be fit into a flatbuffer vector
            {
                start = 0;
                end = report.defects.size();
            }
            else
            {
                start = end;
                end = report.defects.size();
            }
            for(size_t i = start; i < end; i++)
            {
                partialReport.defects.push_back(std::move(report.defects[i]));
            }
            fbbDefectsVec[idx].Finish(
                validation::ValidationReport::Pack(fbbDefectsVec[idx], &partialReport));
        }
    }
}

执行后触发断言错误：

Assertion `size() < FLATBUFFERS_MAX_BUFFER_SIZE' failed.
Aborted

错误发生在Finish调用行：

fbbDefectsVec[idx].Finish(
                    validation::ValidationReport::Pack(fbbDefectsVec[idx], &partialReport));

错误原因 #

1. 序列化大小估算错误
   代码中用sizeof(*report.defects[0])计算单个元素的大小，这是ValidationDefectT原生C++对象的内存大小（仅包含指针、基础类型等），但FlatBuffer序列化后的实际大小远大于这个值——序列化会把对象的所有嵌套字段、字符串、数组等完整写入，实际占用的字节数和原生对象大小没有直接关系。按这个值计算出的max_elems会远大于实际能容纳的元素数，导致单个拆分后的Buffer序列化后仍超过2GB限制，触发断言。

2. 原容器元素失效问题
   使用std::move(report.defects[i])后，report.defects[i]会变成空的unique_ptr，后续循环中再次访问report.defects[i]（比如剩余元素处理逻辑）会导致未定义行为，可能间接引发序列化异常。

解决方案 #

1. 正确估算序列化后的元素大小 #

不要依赖原生对象的sizeof，而是通过序列化少量元素来估算单个元素的平均序列化大小：

// 先序列化一个元素，计算实际大小
flatbuffers::FlatBufferBuilder test_fbb;
auto test_defect = validation::ValidationDefect::Pack(test_fbb, report.defects[0].get());
test_fbb.Finish(test_defect);
size_t avg_elem_serialized_size = test_fbb.GetSize();

// 计算最大容纳元素数（预留10%的安全空间，避免边界溢出）
const size_t safety_margin = 10; // 10%
size_t max_elems = (FLATBUFFERS_MAX_BUFFER_SIZE * (100 - safety_margin) / 100) / avg_elem_serialized_size;
// 确保至少为1
if (max_elems == 0) max_elems = 1;

2. 调整拆分逻辑，避免访问已移动的元素 #

不要直接移动原容器的元素，改用迭代器遍历并转移所有权，避免后续访问失效元素：

void createReport(std::vector<flatbuffers::FlatBufferBuilder>& fbbDefectsVec)
{ 
    ValidationReportT report; // 假设已填充数据
    if (report.defects.empty()) return;

    // 估算平均序列化大小
    flatbuffers::FlatBufferBuilder test_fbb;
    auto test_defect = validation::ValidationDefect::Pack(test_fbb, report.defects[0].get());
    test_fbb.Finish(test_defect);
    size_t avg_size = test_fbb.GetSize();
    const size_t safety_margin = 10;
    size_t max_elems = (FLATBUFFERS_MAX_BUFFER_SIZE * (100 - safety_margin) / 100) / avg_size;
    if (max_elems == 0) max_elems = 1;

    size_t total = report.defects.size();
    size_t fbbs_needed = (total + max_elems - 1) / max_elems; // 向上取整
    fbbDefectsVec.resize(fbbs_needed);

    size_t idx = 0;
    size_t current = 0;

    while (current < total) {
        ValidationReportT partial;
        size_t end = std::min(current + max_elems, total);
        // 转移当前块的元素所有权
        for (size_t i = current; i < end; ++i) {
            partial.defects.push_back(std::move(report.defects[i]));
        }
        // 序列化并完成
        fbbDefectsVec[idx].Finish(validation::ValidationReport::Pack(fbbDefectsVec[idx], &partial));
        idx++;
        current = end;
    }
}

3. 保守拆分策略（备选） #

如果估算大小有困难，可以直接设置一个保守的元素数量（比如每50万条一个Buffer），确保单个Buffer序列化后远小于2GB：

size_t max_elems = 500000; // 根据实际序列化后的大小调整

4. 实时监控Buffer大小 #

在序列化过程中，通过FlatBufferBuilder::GetSize()实时监控当前Buffer的大小，接近2GB时停止添加元素：

ValidationReportT partial;
flatbuffers::FlatBufferBuilder& fbb = fbbDefectsVec[idx];
for (size_t i = current; i < total; ++i) {
    auto defect = validation::ValidationDefect::Pack(fbb, report.defects[i].get());
    partial.defects.push_back(flatbuffers::unique_ptr<ValidationDefectT>(validation::ValidationDefect::UnPack(defect)));
    // 检查当前Buffer大小，接近限制则停止
    if (fbb.GetSize() > FLATBUFFERS_MAX_BUFFER_SIZE * 0.9) {
        current = i + 1;
        break;
    }
    current = i + 1;
}
fbb.Finish(validation::ValidationReport::Pack(fbb, &partial));

内容的提问来源于stack exchange，提问作者Abhishek Kusnoor