为何`File::read_to_end`会随缓冲区容量增大而变慢？

截至2023年4月23日，此问题的修复已合并到rust-lang/rust:master。很快你就可以无忧使用File::read_to_end了。

我正在处理一个特殊需求：读取数十万份从几字节到几百MB不等的文件。由于操作核心是枚举文件并从磁盘传输数据，我选择复用Vec缓冲区来避免部分内存管理开销。

这时我遇到了意外情况：file.read_to_end(&mut buffer)?的速度会随缓冲区容量增大而逐渐变慢。先读取一个300MB文件再读取一千个1KB文件，比反过来操作慢得多（只要不截断缓冲区）。

令人困惑的是，如果用Take包装文件或使用read_exact()，就不会出现速度下降的情况。

有人知道这是怎么回事吗？是否每次调用时都会（重新）初始化整个缓冲区？这是Windows特有的问题吗？处理此类问题时，推荐哪些Windows性能分析工具？

以下是一个简单的复现代码，可展示这些方法间巨大的性能差异（在我的机器上相差50倍以上，忽略磁盘速度影响）：

use std::io::Read;
use std::fs::File;

// with a smaller buffer, there's basically no difference between the methods...
// const BUFFER_SIZE: usize = 2 * 1024;

// ...but the larger the Vec, the bigger the discrepancy.
// for simplicity's sake, let's assume this is a hard upper limit.
const BUFFER_SIZE: usize = 300 * 1024 * 1024;


fn naive() {
    let mut buffer = Vec::with_capacity(BUFFER_SIZE);

    for _ in 0..100 {
        let mut file = File::open("some_1kb_file.txt").expect("opening file");

        let metadata = file.metadata().expect("reading metadata");
        let len = metadata.len();
        assert!(len <= BUFFER_SIZE as u64);

        buffer.clear();
        file.read_to_end(&mut buffer).expect("reading file");

        // do "stuff" with buffer
        let check = buffer.iter().fold(0usize, |acc, x| acc.wrapping_add(*x as usize));

        println!("length: {len}, check: {check}");
    }
}

fn take() {
    let mut buffer = Vec::with_capacity(BUFFER_SIZE);

    for _ in 0..100 {
        let file = File::open("some_1kb_file.txt").expect("opening file");

        let metadata = file.metadata().expect("reading metadata");
        let len = metadata.len();
        assert!(len <= BUFFER_SIZE as u64);

        buffer.clear();
        file.take(len).read_to_end(&mut buffer).expect("reading file");

        // this also behaves like the straight `read_to_end` with a significant slowdown:
        // file.take(BUFFER_SIZE as u64).read_to_end(&mut buffer).expect("reading file");

        // do "stuff" with buffer
        let check = buffer.iter().fold(0usize, |acc, x| acc.wrapping_add(*x as usize));

        println!("length: {len}, check: {check}");
    }
}

fn exact() {
    let mut buffer = vec![0u8; BUFFER_SIZE];

    for _ in 0..100 {
        let mut file = File::open("some_1kb_file.txt").expect("opening file");

        let metadata = file.metadata().expect("reading metadata");
        let len = metadata.len() as usize;
        assert!(len <= BUFFER_SIZE);

        // SAFETY: initialized by `vec!` and within capacity by `assert!`
        unsafe { buffer.set_len(len); }
        file.read_exact(&mut buffer[0..len]).expect("reading file");

        // do "stuff" with buffer
        let check = buffer.iter().fold(0usize, |acc, x| acc.wrapping_add(*x as usize));

        println!("length: {len}, check: {check}");
    }
}

fn main() {
    let args: Vec<String> = std::env::args().collect();

    if args.len() < 2 {
        println!("usage: {} <method>", args[0]);
        return;
    }

    match args[1].as_str() {
        "naive" => naive(),
        "take" => take(),
        "exact" => exact(),
        _ => println!("Unknown method: {}", args[1]),
    }
}

我尝试了--release模式、LTO甚至+crt-static等多种组合，但均未带来明显差异。

内容的提问来源于stack exchange，提问作者ducktherapy