使用tokio-tar异步读取tar.gz条目时，读取字节数>0但缓冲区为空

我正在编写基于Tokio的异步程序，支持透明读取普通文件、gzip压缩文件和gzip压缩tar归档文件。前两种文件读取正常，但用tokio-tar读取tar归档文件时遇到问题：读取的字节数大于0，但缓冲区始终为空。

复现代码 #

#![feature(let_chains)]
use std::path::Path;

use async_compression::tokio::bufread::GzipDecoder;
use futures::StreamExt;
use tokio::{
    fs::File,
    io::{AsyncReadExt, BufReader},
};
use tokio_tar::{Archive, Entry};

#[derive(Debug)]
pub enum Handle {
    File(BufReader<File>),
    Gz(GzipDecoder<BufReader<File>>),
    TarGz(Entry<Archive<GzipDecoder<BufReader<File>>>>),
}

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // prepare the reader
    let f = File::open("./archive.tar.gz").await?;
    let gz = GzipDecoder::new(BufReader::new(f));
    let mut archive = Archive::new(gz);

    // the path of the file to read within the archive
    let path_wanted = Path::new("tar/archive.txt");
    let mut entries = archive.entries()?;
    let mut handle: Option<Handle> = None;

    // iterate through the files in the archive until the desired file is found
    while let Some(entry_result) = entries.next().await {
        if let Ok(entry) = entry_result
            && let Ok(path) = entry.path()
            && path == path_wanted
        {
            handle = Some(Handle::TarGz(entry));
        }
    }

    // use the assigned variable to read the archive entry's contents
    if let Some(mut tgz) = handle {
        match tgz {
            Handle::TarGz(ref mut tgz) => {
                let mut vec_buf = vec![0; 10];
                loop {
                    match tgz.take(10).read_to_end(&mut vec_buf).await {
                        // here the actual problem becomes visible
                        // while reading the entry, n is greater than 0 but nothing is written to the buffer
                        Ok(n) if n > 0 => {
                            println!("n: {} - {:?}", n, &vec_buf[..n]);
                        }
                        _ => break,
                    }
                }
            }
            _ => panic!("case not covered"),
        }
    } else {
        println!("path_wanted not found in archive");
    }
    Ok(())
}

归档文件内容 #

127.0.0.1   test.com
127.0.0.1   test.de

程序输出 #

tarvester on  master [?] is 📦 v0.1.0 via 🦀 v1.66.0-nightly
❯ cargo run
   Compiling tarvester v0.1.0 (/home/bob/code/gardion/tarvester)
    Finished dev [unoptimized + debuginfo] target(s) in 0.54s
     Running `target/debug/tarvester`
n: 10 - [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
n: 10 - [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
n: 10 - [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
n: 7 - [0, 0, 0, 0, 0, 0, 0]

修改版代码（输出混乱） #

#![feature(let_chains)]
use std::path::Path;

use async_compression::tokio::bufread::GzipDecoder;
use futures::StreamExt;
use tokio::{
    fs::File,
    io::{AsyncReadExt, BufReader},
};
use tokio_tar::{Archive, Entry};

#[derive(Debug)]
pub enum Handle {
    File(BufReader<File>),
    Gz(GzipDecoder<BufReader<File>>),
    TarGz(Entry<Archive<GzipDecoder<BufReader<File>>>>),
}

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let f = File::open("../harvester/archive.tar.gz").await?;
    let gz = GzipDecoder::new(BufReader::new(f));
    let mut archive = Archive::new(gz);

    let path_wanted = Path::new("tar/archive.txt");
    let mut entries = archive.entries()?;
    let mut handle: Option<Handle> = None;
    while let Some(entry_result) = entries.next().await {
        if let Ok(entry) = entry_result
            && let Ok(path) = entry.path()
            && path == path_wanted
        {
            handle = Some(Handle::TarGz(entry));
            // !! the handle assigned to the variable is used right away without leaving the block
            match handle.as_mut().unwrap() {
                Handle::TarGz(tgz) => {
                    let mut vec_buf = vec![0; 10];
                    loop {
                        match tgz.take(10).read_to_end(&mut vec_buf).await {
                            // some messed up output shows up here
                            Ok(n) if n > 0 => {
                                println!("{}", String::from_utf8_lossy(&vec_buf[..]));
                            },
                            _ => break
                        }
                    }
                },
                _ => panic!("case not covered"),
            }
        }
    }
    Ok(())
}

修改版输出 #

127.0.0.1
127.0.0.1       test.com
1
127.0.0.1       test.com
127.0.0.1       t
127.0.0.1       test.com
127.0.0.1       test.de

问题原因与修复 #

1. 第一个版本的核心问题：Archive被提前销毁 #

Entry类型持有Archive的内部引用，当你找到目标条目后继续遍历所有entries直到流结束，archive变量会被自动销毁。后续读取已失效的Entry时，实际操作的是已释放的资源，导致缓冲区全为0。

修复方法：找到目标条目后立即跳出循环，保留Archive的生命周期：

handle = Some(Handle::TarGz(entry));
break; // 关键：停止遍历，避免Archive被销毁

2. 读取方式误用：take()与read_to_end()的错误搭配 #

• tgz.take(10)每次调用都会创建新的Take包装器，消耗Entry的读取状态，导致读取异常。
• read_to_end(&mut vec_buf)会将数据追加到Vec末尾，而你复用固定长度缓冲区的方式不符合该方法的使用逻辑。

正确的读取方式是直接使用read(&mut vec_buf)读取指定长度数据：

let mut vec_buf = vec![0; 10];
loop {
    match tgz.read(&mut vec_buf).await {
        Ok(n) if n > 0 => {
            println!("n: {} - {:?}", n, &vec_buf[..n]);
            println!("content: {}", String::from_utf8_lossy(&vec_buf[..n]));
        }
        Ok(0) => break, // 读取完成
        Err(e) => {
            eprintln!("read error: {}", e);
            break;
        }
    }
}

3. 修改版输出混乱的原因 #

修改版中每次打印整个vec_buf，而read_to_end会不断向缓冲区追加数据，导致输出包含之前读取的旧内容。正确做法是只打印读取到的n个字节，即&vec_buf[..n]。

内容的提问来源于stack exchange，提问作者bob3000