如何将arrow-rs数组导出至Java？实践遇段错误求助

问题背景 #

Arrow Java的C Data Interface采用Java分配内存后由C++填充数据的模式，但arrow-rs的工作逻辑相反：先在Rust中填充数据，再将指针传递给其他语言，因此无法直接复用Java与C++通信的代码。

Rust实现细节 #

由于JNI无法直接识别指针，我通过pointer as i64将FFI_ArrowArray和FFI_ArrowSchema的地址以64位整数形式返回：

pub fn array_example() -> Int32Array
{
    let array = Int32Array::from(vec![Some(1), None, Some(3)]);
    assert_eq!(array.len(), 3);
    assert_eq!(array.value(0), 1);
    assert_eq!(array.is_null(1), true);

    let collected: Vec<_> = array.iter().collect();
    assert_eq!(collected, vec![Some(1), None, Some(3)]);
    assert_eq!(array.values(), [1, 0, 3]);
    array
}

pub fn export_array_example() -> [i64; 2] {
    // Export it
    let array = array_example();
    let data = array.data();
    let out_array = FFI_ArrowArray::new(&data);
    let out_schema = FFI_ArrowSchema::try_from(data.data_type()).unwrap();

    // Use ManuallyDrop to avoid Box:Drop recursing
    let schema = Box::new(ManuallyDrop::new(out_schema));
    let array = Box::new(ManuallyDrop::new(out_array));

    let schema_ptr = &**schema as *const _;
    let array_ptr = &**array as *const _;
    let schema_addr = schema_ptr as i64;
    let array_addr = array_ptr as i64;

    let schema_addr2 = schema_addr as *const FFI_ArrowSchema;
    let array_addr2 = array_addr as *const FFI_ArrowArray;
    let array = unsafe {
        ArrowArray::new(std::ptr::read(array_addr2), std::ptr::read(schema_addr2))
    };

    let array = Int32Array::from(ArrayData::try_from(array).unwrap());
    println!("pointer schema as long: {}", schema_addr);
    println!("pointer array as long: {}", array_addr);
    println!("recovered arr: {:?}",array);

    [schema_addr, array_addr]
}

Java实现细节 #

Java端使用ArrowSchema.wrap和ArrowArray.wrap方法接收指针地址：

public class Converter {

    static {
        System.loadLibrary("lance_jni");
    }

    public static void getInt32ArrayExample() {
        BufferAllocator allocator = new RootAllocator();
        ConverterJni.getInt32Arr();
        long[] arr = ConverterJni.getInt32Arr();
        System.out.println(arr[0]);
        System.out.println(arr[1]);
        ArrowSchema arrowSchema = ArrowSchema.wrap(arr[0]);
        ArrowArray array = ArrowArray.wrap(arr[1]);
        var vec = Data.importVector(allocator, array, arrowSchema, null);
    }
}

错误信息 #

运行后触发JVM段错误：

[info] #
[info] # A fatal error has been detected by the Java Runtime Environment:
[info] #
[info] #  SIGSEGV (0xb) at pc=0x00000001127708e7, pid=47483, tid=5891
[info] #
[info] # JRE version: OpenJDK Runtime Environment Homebrew (11.0.12) (build 11.0.12+0)
[info] # Java VM: OpenJDK 64-Bit Server VM Homebrew (11.0.12+0, mixed mode, tiered, compressed oops, g1 gc, bsd-amd64)
[info] # Problematic frame:
[info] # J 414 c2 java.nio.DirectByteBuffer.get()B java.base@11.0.12 (28 bytes) @ 0x00000001127708e7 [0x00000001127708a0+0x0000000000000047]

问题分析与修复方案 #

核心问题 #

Rust侧返回的指针指向的内存已被释放：

• 在export_array_example中，out_array和out_schema被包裹在Box中，函数返回时Box会被销毁，对应的堆内存被Rust自动回收，导致Java侧拿到悬空指针。
• ManuallyDrop仅抑制FFI_ArrowArray/FFI_ArrowSchema内部的Drop逻辑，但无法阻止Box本身的内存回收。

修复步骤 #

1. Rust侧：保留内存控制权

使用Box::leak将内存脱离Rust的自动管理，确保Java侧使用期间内存不被释放：

pub fn export_array_example() -> [i64; 2] {
    let array = array_example();
    let data = array.data();
    let out_array = FFI_ArrowArray::new(&data);
    let out_schema = FFI_ArrowSchema::try_from(data.data_type()).unwrap();

    // 用Box::leak让内存脱离Rust自动管理
    let schema_ptr = Box::leak(Box::new(out_schema)) as *mut FFI_ArrowSchema;
    let array_ptr = Box::leak(Box::new(out_array)) as *mut FFI_ArrowArray;

    let schema_addr = schema_ptr as i64;
    let array_addr = array_ptr as i64;

    // 验证指针有效性（可选）
    let array = unsafe {
        ArrowArray::new(std::ptr::read(array_ptr), std::ptr::read(schema_ptr))
    };
    let array = Int32Array::from(ArrayData::try_from(array).unwrap());
    println!("recovered arr: {:?}", array);

    [schema_addr, array_addr]
}

2. 新增Rust释放逻辑

通过JNI暴露释放函数，让Java侧使用完内存后主动回收：

#[no_mangle]
pub extern "C" fn Java_com_example_Converter_releaseArrowResources(
    _env: JNIEnv,
    _class: JClass,
    schema_addr: i64,
    array_addr: i64,
) {
    unsafe {
        let schema_ptr = schema_addr as *mut FFI_ArrowSchema;
        let array_ptr = array_addr as *mut FFI_ArrowArray;
        // 先执行内部资源清理
        std::ptr::drop_in_place(schema_ptr);
        std::ptr::drop_in_place(array_ptr);
        // 释放堆内存
        Box::from_raw(schema_ptr);
        Box::from_raw(array_ptr);
    }
}

3. Java侧：主动释放内存

在导入数据完成后，调用Rust的释放函数避免内存泄漏：

public class Converter {

    static {
        System.loadLibrary("lance_jni");
    }

    // 新增JNI释放方法
    public static native void releaseArrowResources(long schemaAddr, long arrayAddr);

    public static void getInt32ArrayExample() {
        BufferAllocator allocator = new RootAllocator();
        long[] arr = ConverterJni.getInt32Arr();
        System.out.println(arr[0]);
        System.out.println(arr[1]);
        
        ArrowSchema arrowSchema = ArrowSchema.wrap(arr[0]);
        ArrowArray array = ArrowArray.wrap(arr[1]);
        
        try {
            var vec = Data.importVector(allocator, array, arrowSchema, null);
            // 处理导入后的vector
        } finally {
            // 释放Rust侧内存
            releaseArrowResources(arr[0], arr[1]);
            // 关闭Java侧Arrow对象
            arrowSchema.close();
            array.close();
        }
    }
}

内容的提问来源于stack exchange，提问作者Renkai