RecordBatch 具备哪些 StructArray 无法实现的能力？

Great question! Let's break down the key capabilities that make RecordBatch distinct from StructArray—even though they seem to hold similar information, their design makes them suited for entirely different use cases:

1. Column-level operations are far more efficient #

RecordBatch is a collection of independent, equal-length arrays (each representing a column). This means you can directly access, modify, or compute on individual columns without touching the rest of the data. For example:

# Access the 'x' column directly from a RecordBatch
x_column = batch['x']
# Compute statistics on 'z' without processing other fields
z_mean = batch['z'].mean()

With a StructArray, you'd have to extract the field first (using struct_array.field('x')) which involves extra offset calculations under the hood—especially slow for large datasets. The columnar layout of RecordBatch is optimized for vectorized operations, which is a huge win for performance.

2. Better alignment with Arrow's ecosystem tools #

Most of Arrow's core tools (like Parquet/IORC readers/writers, Arrow Flight for data transfer, Datasets for querying large data lakes) are built around RecordBatch as the fundamental data unit. For example:

• When writing to Parquet, a RecordBatch maps directly to Parquet's columnar storage format, preserving types and schema seamlessly.
• Using Arrow Dataset to filter a large dataset? You can push down predicates to specific columns in a RecordBatch, avoiding loading unnecessary data.

A StructArray would be treated as a single nested column in these tools, which limits your ability to work with individual fields efficiently.

3. Flexible schema evolution #

Modifying the schema of a RecordBatch is straightforward—you can add, remove, or reorder columns without rebuilding the entire dataset. For example:

# Add a new column to a RecordBatch
new_col = pa.array([10, 20], type=pa.int64())
new_schema = batch.schema.append(pa.field('w', pa.int64()))
new_batch = pa.RecordBatch.from_arrays(batch.columns + [new_col], schema=new_schema)

With a StructArray, changing the schema (like adding a new field to the struct) requires reconstructing every element in the array, which is computationally expensive for large data.

4. Natural fit for tabular data workflows #

As your example shows, RecordBatch converts directly to a pd.DataFrame, which aligns perfectly with common data analysis workflows (cleaning, filtering, aggregating tabular data). Each column is a first-class citizen, making it easy to use with pandas or Arrow's own compute functions.

A StructArray, on the other hand, is better suited for scenarios where each element is a self-contained nested object (like a list of complex records), but it's clunky for tabular operations—you end up with a pandas Series of dictionaries, which lacks the performance and convenience of a DataFrame.

5. Memory layout optimizations #

RecordBatch stores each column as a separate Arrow array, which allows for better memory utilization and cache efficiency. For example, numeric columns are stored as contiguous blocks of memory, which makes vectorized operations (like sum, mean) much faster.

StructArray stores data in a nested format, where each struct element's fields are interleaved. This means accessing a single field requires jumping through memory offsets, which is less cache-friendly and slower for large-scale computations.

内容的提问来源于stack exchange，提问作者user554319