运行数小时后CPU占满、工作线程挂起，疑与reqwest/tokio相关

程序架构 #

• 通过tokio::spawn创建多个工作线程
• 所有线程处理相似数据，线程函数参数通过snd_a.clone()传递mpsc通道（期望无需使用Arc<Mutex<...>>）
• 接收线程持有Receiver句柄rcv_a，对工作线程传递的数据发起网络调用
• 另有1-2个独立线程操作相同共享数据，通过Mutex同步

问题现象 #

• 程序存在全量执行完成和部分执行的情况，15-20次挂起案例中触发时机不固定，多数与try_poll函数相关
• 挂起时所有工作线程、接收线程及其他调用线程均挂起，仅完全独立线程不受影响；CPU占用率锁定100%，UI操作正常
• 至少2次挂起发生在两个线程并行调用try_poll时，卡在函数第一行；至少10次挂起最后输出为函数第一行打印语句，无错误或超时提示

核心代码（try_poll函数） #

async fn try_poll(link:String, req_client: &Client, params: &[(&str, &str)]) -> Result<String, MyError> {
    println!("{:?}",line!());
    let res = match timeout(Duration::from_secs(10),req_client.get(link).query(params).send()).await{
        Ok(x)=> match x{
            Ok(x)=> match timeout(Duration::from_secs(10), x.text()).await{
                Ok(x)=> match x{
                    Ok(x)=> x,
                    Err(x)=>{
                        println!("--[tp 0]{:?}",x);
                        return Err(MyError::ReqwestFail)
                    }
                },
                Err(x)=>{
                    println!("--[tp 1]{:?}",x);
                    return Err(MyError::ReqwestFail)
                }
            },
            Err(x)=>{
                println!("--[tp 2]{:?}",x);
                return Err(MyError::ReqwestFail)
            }
        },
        Err(x)=> {
            println!("--[tp 3]{:?}",x);
            return Err(MyError::ReqwestFail)
        }
    };
    println!("{:?}",line!());
    Ok(res)
}

怀疑方向 #

• 怀疑是reqwest的bug，或是其与std而非tokio协作导致
• 猜测某个API无限挂起，但无法理解为何仅处理数据的工作线程也会挂起

补充调试信息（挂起后执行性能计数器和strace的输出） #

性能计数器输出 #

Performance counter stats for process id '232825':

 472 004,37 msec task-clock                #    3,698 CPUs utilized          
 116 359      context-switches          #    0,247 K/sec                   
 1 058      cpu-migrations            #    0,002 K/sec                   
 917      page-faults               #    0,002 K/sec                   
 1 995 523 580 533      cycles                    #    4,228 GHz                     
 1 425 266 118 862      stalled-cycles-frontend   #   71,42% frontend cycles idle   
 987 982 567 869      stalled-cycles-backend    #   49,51% backend cycles idle    
 1 285 536 694 249      instructions              #    0,64  insn per cycle         
                                                  #    1,11  stalled cycles per insn
 240 695 202 055      branches                  #  509,943 M/sec                   
 5 222 709 470      branch-misses             #    2,17% of all branches        

 127,641810678 seconds time elapsed

strace输出 #

strace: Process 232825 attached with 16 threads
[pid 239382] futex(0x7f7990027968, FUTEX_WAIT_BITSET_PRIVATE, 4294967295, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid 239377] restart_syscall(<... resuming interrupted read ...> <unfinished ...>
[pid 239376] futex(0x5585fc023c98, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 239375] futex(0x5585fc0233e0, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 239374] futex(0x5585fc0233e0, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 239373] futex(0x5585fc0233e0, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 239372] futex(0x5585fc0233e0, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 239371] futex(0x5585fbfb6b0c, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 239370] futex(0x5585fbfb69ac, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 239369] futex(0x5585fbfb684c, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 239368] futex(0x5585fbfb66ec, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 232825] recvmsg(10, {msg_namelen=0}, 0) = -1 EAGAIN (Resource temporarily unavailable)
[pid 232825] recvmsg(10, {msg_namelen=0}, 0) = -1 EAGAIN (Resource temporarily unavailable)
[pid 232825] recvmsg(10, {msg_namelen=0}, 0) = -1 EAGAIN (Resource temporarily unavailable)
[pid 232825] recvmsg(10, {msg_namelen=0}, 0) = -1 EAGAIN (Resource temporarily unavailable)
[pid 232825] recvmsg(10, {msg_namelen=0}, 0) = -1 EAGAIN (Resource temporarily unavailable)
[pid 232825] recvmsg(10, {msg_namelen=0}, 0) = -1 EAGAIN (Resource temporarily unavailable)
[pid 232825] epoll_wait(11, 

求助需求 #

如何获取所有执行操作的详细信息，准确定位导致CPU占满的操作？

内容的提问来源于stack exchange，提问作者tommy1213