如何用reqwest实现异步请求？Rust重写Python异步代码未达预期

我想用Rust的tokio和reqwest crate重写下面的Python异步请求代码：

#!/usr/bin/env python3

import asyncio
import httpx
import time

async def async_req(url, client):
    headers = {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36 Edg/111.0.1661.54",
        "Cookie": ""
    }
    response = await client.get(url, headers=headers, timeout=180)
    print(response.status_code)
    print(response.text)

async def main():
    urls = ["http://localhost:5000/hello", "http://localhost:5000/world"]
    start = time.time()
    async with httpx.AsyncClient() as client:
        tasks = [async_req(url, client) for url in urls]
        await asyncio.gather(*tasks)
    print(time.time()-start)

if __name__ == "__main__":
    asyncio.run(main())

我的Rust实现代码如下：

async fn async_req(
    url: &str,
    client: reqwest::Client,
) -> Result<reqwest::Response, Box<dyn std::error::Error>> {
    let response = client
        .get(url)
        .timeout(std::time::Duration::from_secs(180))
        .send()
        .await?;

    Ok(response)
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let urls = vec![
        "http://localhost:5000/hello",
        "http://localhost:5000/world",
    ];

    let mut handles = vec![];

    let client = reqwest::Client::new();
    for url in urls {
        let handle = {
            let client = client.clone();
            async move { async_req(url, client).await }
        };
        handles.push(handle);
    }

    let mut responses = vec![];
    for handle in handles {
        responses.push(handle.await?);
    }

    for response in responses {
        println!("{}", response.text().await?);
    }

    Ok(())
}

但程序并没有像Python版本那样发起异步请求，目前还没找到可行的解决办法。我的Cargo.toml配置如下，使用的Rust编译器版本是1.72.1（d5c2e9c34, 2023-09-13）：

[package]
name = "reqwest_async_demo"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
reqwest = { version = "^0.11" }
tokio = { version = "^1", features = [ "full" ] }

解决方案 #

问题核心是逐个await任务句柄会导致任务串行执行，而非并行。Python的asyncio.gather会同时启动所有任务并等待全部完成，对应Rust可使用futures::future::join_all或tokio::join!（任务数量固定时）实现。另外，你的Rust代码缺失了Python版本中的请求头，需要补充。

修改后的完整代码

首先在Cargo.toml中添加futures依赖（使用join_all时需要）：

[dependencies]
reqwest = { version = "^0.11" }
tokio = { version = "^1", features = [ "full" ] }
futures = "0.3"

然后修改Rust代码：

use futures::future::join_all;

async fn async_req(
    url: &str,
    client: reqwest::Client,
) -> Result<(), Box<dyn std::error::Error>> {
    // 构造请求头，对齐Python逻辑
    let headers = reqwest::header::HeaderMap::from_iter([
        (
            reqwest::header::USER_AGENT,
            "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36 Edg/111.0.1661.54".parse()?,
        ),
        (reqwest::header::COOKIE, "".parse()?),
    ]);

    let response = client
        .get(url)
        .headers(headers)
        .timeout(std::time::Duration::from_secs(180))
        .send()
        .await?;

    // 直接打印响应，和Python逻辑一致
    println!("{}", response.status());
    println!("{}", response.text().await?);
    Ok(())
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let urls = vec![
        "http://localhost:5000/hello",
        "http://localhost:5000/world",
    ];

    let client = reqwest::Client::new();
    // 批量创建异步任务
    let tasks = urls.into_iter().map(|url| {
        let client = client.clone();
        async move { async_req(url, client).await }
    }).collect::<Vec<_>>();

    // 并行等待所有任务完成，等价于Python的asyncio.gather
    join_all(tasks).await;

    Ok(())
}

关键修改点

1. 使用join_all替代逐个await任务，确保所有任务并行执行
2. 补充Python代码中的请求头（User-Agent和Cookie）
3. 调整async_req逻辑，直接在内部打印响应内容，对齐Python行为
4. 简化任务创建与收集的代码逻辑

如果任务数量固定（如本例仅2个），也可使用tokio::join!实现更简洁的并行：

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let client = reqwest::Client::new();
    
    let client1 = client.clone();
    let client2 = client.clone();
    
    // 并行执行两个任务
    tokio::join!(
        async_req("http://localhost:5000/hello", client1),
        async_req("http://localhost:5000/world", client2)
    );

    Ok(())
}

修改后程序会像Python版本一样发起异步请求，同时处理多个URL。

内容的提问来源于stack exchange，提问作者baby195lxl