Apache Arrow C++中ListArray元素设为Required写Parquet崩溃求助

背景说明 #

基于Apache Arrow行转列固定Schema的示例开发，需要将component_cost字段的List元素从Optional改为Required，修改Schema后执行WriteTable时程序直接崩溃。

当前代码实现 #

struct data_row {
  int64_t id;
  int64_t components;
  std::vector<double> component_cost;
};

void VectorToColumnarTable(const std::vector<struct data_row>& rows) {
  // Create filepath
  std::filesystem::path file_path = "C:/my_example/example.parquet";
  std::shared_ptr<arrow::io::FileOutputStream> outfile {};
  PARQUET_ASSIGN_OR_THROW(outfile, arrow::io::FileOutputStream::Open(file_path.string()));

  arrow::MemoryPool* pool = arrow::default_memory_pool();
  arrow::Int64Builder id_builder(pool);
  arrow::Int64Builder components_builder(pool);
  arrow::ListBuilder component_cost_builder(pool, std::make_shared<arrow::DoubleBuilder>(pool));
  arrow::DoubleBuilder* component_item_cost_builder =
      (static_cast<arrow::DoubleBuilder*>(component_cost_builder.value_builder()));

  for (const data_row& row : rows) {
    PARQUET_THROW_NOT_OK(id_builder.Append(row.id));
    PARQUET_THROW_NOT_OK(components_builder.Append(row.components));

    PARQUET_THROW_NOT_OK(component_cost_builder.Append());
    PARQUET_THROW_NOT_OK(
        component_item_cost_builder->AppendValues(row.component_cost.data(), row.component_cost.size()));
  }
  std::shared_ptr<arrow::Array> id_array;
  PARQUET_THROW_NOT_OK((id_builder.Finish(&id_array)));
  std::shared_ptr<arrow::Array> components_array;
  PARQUET_THROW_NOT_OK((components_builder.Finish(&components_array)));
  std::shared_ptr<arrow::Array> component_cost_array;
  PARQUET_THROW_NOT_OK((component_cost_builder.Finish(&component_cost_array)));

  // My schema vector -----&gt; different to the website one
  std::vector<std::shared_ptr<arrow::Field>> schema_vector = {
      arrow::field("id", arrow::int64()),
      arrow::field("components", arrow::int64()),
      arrow::field("component_cost", arrow::list(arrow::field("item", arrow::float64(), false)))};

  std::shared_ptr<arrow::Table> table = arrow::Table::Make(arrow::schema(schema_vector), {id_array, components_array, component_cost_array});
  PARQUET_THROW_NOT_OK(parquet::arrow::WriteTable(*table, arrow::default_memory_pool(), outfile, 100));

}

目标Schema #

期望生成的Parquet Schema中，component_cost的List元素为Required：

"fields": { "PrimitiveType": { "field_info": { "name": "item", "repetition": "Required", "id": null }, "logical_type": null, "converted_type": null, "physical_type": "Double" } }

解决建议 #

1. 修复类型不匹配核心问题 #

代码中存在Schema类型与Builder类型不匹配的错误：

• 使用arrow::DoubleBuilder（对应Arrow的double类型，即64位浮点数）构建List元素
• 但Schema中item字段写的是arrow::float64()（对应32位浮点数）

这种类型不匹配会直接导致Parquet写入时崩溃，修改Schema的item类型为arrow::double()即可：

std::vector<std::shared_ptr<arrow::Field>> schema_vector = {
    arrow::field("id", arrow::int64()),
    arrow::field("components", arrow::int64()),
    arrow::field("component_cost", arrow::list(arrow::field("item", arrow::double(), false)))};

2. 确保符合Required约束（可选） #

List元素设为Required意味着List中的每个元素都不能为null，但空List本身是允许的（除非你同时把component_cost字段本身设为Required）。如果你的业务场景要求每个List至少包含一个元素，需要在循环中添加校验：

for (const data_row& row : rows) {
    PARQUET_THROW_NOT_OK(id_builder.Append(row.id));
    PARQUET_THROW_NOT_OK(components_builder.Append(row.components));

    if (row.component_cost.empty()) {
        // 根据业务需求处理，比如抛出错误或写入默认值
        PARQUET_THROW(parquet::ParquetException("component_cost cannot be empty"));
    } else {
        PARQUET_THROW_NOT_OK(component_cost_builder.Append());
        PARQUET_THROW_NOT_OK(
            component_item_cost_builder->AppendValues(row.component_cost.data(), row.component_cost.size()));
    }
}

3. 添加前置校验提前排查问题 #

在执行WriteTable之前，添加Table合法性校验，能提前发现Schema与数据不匹配的问题，避免直接崩溃：

std::shared_ptr<arrow::Table> table = arrow::Table::Make(arrow::schema(schema_vector), {id_array, components_array, component_cost_array});

// 添加Table校验
auto validate_status = table->Validate();
if (!validate_status.ok()) {
    std::cerr << "Table validation failed: " << validate_status.message() << std::endl;
    return;
}

PARQUET_THROW_NOT_OK(parquet::arrow::WriteTable(*table, arrow::default_memory_pool(), outfile, 100));

内容的提问来源于stack exchange，提问作者ads_mc_1234