NXP ARM Linux平台std::thread::join()偶发挂死问题求助

问题背景 #

我编写了一个创建多线程并在其终止时调用std::thread::join()的程序，该程序在AWS/X86/Ubuntu平台运行正常，但在NXP/ARM 5.10发行版上频繁出现join挂死（偶尔正常）。挂死时线程已消失，但主线程CPU占用率达100%（正常情况下主线程无CPU消耗）。

编译命令 #

g++ --std=c++11 -O3

程序代码 #

// Generate load and then be idle, do it until killed with different loading
// patterns.  Also, have several threads.

#include <sys/prctl.h>
#include <unistd.h>
#include <pthread.h>
#include <thread>
#include <atomic>
#include <cstring>
#include <cstdio>
#include <signal.h>

#ifdef __cplusplus
extern "C" {
#endif

char *module_tag = (char *)"loader";

#ifdef __cplusplus
}
#endif

static const int threadCount = 8;

static std::atomic<bool> finished { false };

static void sigfcn(int signum)
{
    struct sigaction sig;
    sig.sa_handler = sigfcn;
    sigaction(SIGINT, &sig, NULL);
    const char *msg = "Loader: Received Control-C\n";
    ssize_t res = write(2, msg, strlen(msg));
    finished = true;
}

void func(int threadNum)
{
    char buf[128];
    snprintf(buf, sizeof(buf), "thread%d", threadNum);
    prctl(PR_SET_NAME, buf);
    while (!finished) {
        for (volatile int i = 0; i < 100000; i++) {
            if (finished) {
                fprintf(stderr, "Thread %d finished (2)\n", threadNum);
                return;
            }
            if (threadNum & 2) {
                std::this_thread::yield();
            }
        }
        if (threadNum & 1) {
            usleep(threadNum * 100);
        }
    }
    fprintf(stderr, "Thread %d finished (1)\n", threadNum);
}

int main(int ac, char **av)
{
    struct sigaction sig;
    sig.sa_handler = sigfcn;
    sigaction(SIGINT, &sig, NULL);

    std::thread threads[threadCount];
    for (int i = 0; i < threadCount; i++) {
        threads[i] = std::thread(func, i);
    }
    fprintf(stderr, "Waiting until finished:\n");

    for (int i = 0; i < threadCount; i++) {
        fprintf(stderr, "Index %d, waiting\n", i);
        if (threads[i].joinable()) {
            fprintf(stderr, "Index %d, joining\n", i);
            threads[i].join();
            fprintf(stderr, "Index %d, joined\n", i);
        }
        fprintf(stderr, "Index %d, done waiting\n", i);
    }
    fprintf(stderr, "done waiting, killing procstat\n");

    return 0;
}

运行成功时输出 #

Waiting until finished:
Index 0, waiting
Index 0, joining
^CLoader: Received Control-C
Thread 7 finished (2)
Thread 1 finished (2)
Thread 4 finished (2)
Thread 2 finished (2)
Thread 6 finished (2)
Thread 0 finished (2)
Thread 3 finished (2)
Index 0, joined
Index 0, done waiting
Index 1, waiting
Index 1, joining
Index 1, joined
Index 1, done waiting
Index 2, waiting
Index 2, joining
Index 2, joined
Index 2, done waiting
Index 3, waiting
Index 3, joining
Index 3, joined
Index 3, done waiting
Index 4, waiting
Index 4, joining
Index 4, joined
Index 4, done waiting
Index 5, waiting
Index 5, joining
Thread 5 finished (1)
Index 5, joined
Index 5, done waiting
Index 6, waiting
Index 6, joining
Index 6, joined
Index 6, done waiting
Index 7, waiting
Index 7, joining
Index 7, joined
Index 7, done waiting
done waiting, killing procstat

运行失败时输出 #

Waiting until finished:
Index 0, waiting
Index 0, joining
^CLoader: Received Control-C
Thread 4 finished (2)
Thread 3 finished (2)
Thread 0 finished (2)
Thread 6 finished (2)
Thread 5 finished (1)
Thread 2 finished (2)
Thread 1 finished (2)
Thread 7 finished (2)

已做排查 #

• 添加大量printf日志，但目标ARM平台无GDB且无法远程调试；
• 通过/proc/<pid>/task/确认线程已终止；
• 使用perf top查看发现glibc中的__libc_csu_init、call_weak_fn等函数占用大量CPU。

请问有人能指出问题所在或提供后续排查方向吗？

问题分析与排查方向 #

1. 信号处理的潜在问题 #

信号处理函数sigfcn中，每次收到SIGINT都会重新调用sigaction设置自己，但未初始化信号集（sa_mask），可能导致信号在处理过程中被重复触发，干扰线程终止逻辑。建议设置信号处理时初始化sa_mask，屏蔽当前信号避免递归：

static void sigfcn(int signum)
{
    struct sigaction sig;
    sig.sa_handler = sigfcn;
    sigemptyset(&sig.sa_mask);
    sigaddset(&sig.sa_mask, SIGINT); // 处理时屏蔽SIGINT
    sigaction(SIGINT, &sig, NULL);
    const char *msg = "Loader: Received Control-C\n";
    ssize_t res = write(2, msg, strlen(msg));
    finished = true;
}

同时主线程的信号初始化也需要补全sa_mask和sa_flags，确保信号处理稳定性。

2. std::thread底层pthread实现问题 #

ARM平台的glibc或pthread库可能存在特定版本bug，尤其是在O3优化下。可以尝试：

• 降低编译优化等级（如-O2或-O0），排除优化导致的指令重排问题；
• 替换std::thread为原生pthread API（pthread_create创建线程、pthread_join等待），验证是否是C++标准库封装的问题；
• 检查目标平台glibc版本，查看是否有已知的pthread join相关bug（如线程销毁时的资源竞态）。

3. 原子变量的内存序问题 #

ARM是弱内存模型平台，即使std::atomic<bool>默认是memory_order_seq_cst，极端情况仍可能出现可见性延迟。可显式指定内存屏障强化可见性：

// 设置时用release，确保写操作对其他线程可见
finished.store(true, std::memory_order_release);
// 读取时用acquire，确保读操作能获取到最新值
while (!finished.load(std::memory_order_acquire)) { ... }

4. 全局对象的构造/析构问题 #

perf显示__libc_csu_init、call_weak_fn占用CPU，这两个函数和全局对象的构造/析构相关。可能线程终止时，全局对象的析构出现死循环或竞态。可以：

• 移除module_tag全局变量，排查是否是全局变量初始化/析构的问题；
• 编译时添加-fno-weak选项，禁用弱符号处理，看是否缓解问题；
• 检查是否有其他全局/静态对象包含线程相关的析构逻辑。

5. 内核层面的线程调度问题 #

NXP/ARM 5.10内核可能存在线程调度或销毁的bug，比如线程退出后内核未正确通知用户态pthread库。可以：

• 升级内核到最新5.10稳定版本，查看是否修复相关bug；
• 查看内核日志（dmesg），检查是否有线程销毁相关的错误信息；
• 调整线程调度策略（用pthread_setschedparam设置为SCHED_FIFO），观察问题出现频率是否变化。

内容的提问来源于stack exchange，提问作者Dave Gotwisner