使用NVIDIA静态PTX编译库链接时的pthreads未定义引用错误

问题背景 #

我在GitHub Actions的Ubuntu 20虚拟机上，通过CMake生成链接命令使用NVIDIA PTX编译库，链接命令如下：

usr/bin/c++ -O3 -DNDEBUG \
    CMakeFiles/vectorAdd_ptx.dir/modified_cuda_samples/vectorAdd_ptx/vectorAdd_ptx.cpp.o \
    -o bin/vectorAdd_ptx  \
    -Wl,-rpath,/usr/local/cuda-11.7/lib64:/usr/local/cuda-11.7/lib64/stubs \
    /usr/local/cuda-11.7/lib64/libcudart.so \
    -lpthread \
    /usr/local/cuda-11.7/lib64/libnvrtc.so \
    /usr/local/cuda-11.7/lib64/stubs/libcuda.so \
    /usr/local/cuda-11.7/lib64/libnvptxcompiler_static.a 

执行后出现大量pthread、sem相关的未定义引用错误：

usr/bin/ld: /usr/local/cuda-11.7/lib64/libnvptxcompiler_static.a(ptxtmp245.o): in function `__ptx989':
stdThreads.cpp:(.text+0x3c): undefined reference to `pthread_key_create'
/usr/bin/ld: stdThreads.cpp:(.text+0x44): undefined reference to `pthread_mutexattr_init'
/usr/bin/ld: stdThreads.cpp:(.text+0x51): undefined reference to `pthread_mutexattr_settype'
/usr/bin/ld: stdThreads.cpp:(.text+0x68): undefined reference to `pthread_mutexattr_destroy'
/usr/bin/ld: /usr/local/cuda-11.7/lib64/libnvptxcompiler_static.a(ptxtmp245.o): in function `__ptx2472':
stdThreads.cpp:(.text+0xfe): undefined reference to `pthread_setspecific'
/usr/bin/ld: /usr/local/cuda-11.7/lib64/libnvptxcompiler_static.a(ptxtmp245.o): in function `__ptx18096':
stdThreads.cpp:(.text+0x14e): undefined reference to `pthread_join'
/usr/bin/ld: /usr/local/cuda-11.7/lib64/libnvptxcompiler_static.a(ptxtmp245.o): in function `__ptx18072':
stdThreads.cpp:(.text+0x169): undefined reference to `sem_init'
/usr/bin/ld: /usr/local/cuda-11.7/lib64/libnvptxcompiler_static.a(ptxtmp245.o): in function `__ptx18073':
stdThreads.cpp:(.text+0x1ce): undefined reference to `sem_wait'
/usr/bin/ld: /usr/local/cuda-11.7/lib64/libnvptxcompiler_static.a(ptxtmp245.o): in function `__ptx18075':
stdThreads.cpp:(.text+0x1fe): undefined reference to `sem_trywait'
/usr/bin/ld: /usr/local/cuda-11.7/lib64/libnvptxcompiler_static.a(ptxtmp245.o): in function `__ptx18035':
stdThreads.cpp:(.text+0x262): undefined reference to `pthread_mutexattr_init'
/usr/bin/ld: stdThreads.cpp:(.text+0x26f): undefined reference to `pthread_mutexattr_settype'
/usr/bin/ld: stdThreads.cpp:(.text+0x284): undefined reference to `pthread_mutexattr_destroy'
/usr/bin/ld: /usr/local/cuda-11.7/lib64/libnvptxcompiler_static.a(ptxtmp245.o): in function `__ptx2045':
stdThreads.cpp:(.text+0x384): undefined reference to `sem_destroy'
/usr/bin/ld: /usr/local/cuda-11.7/lib64/libnvptxcompiler_static.a(ptxtmp245.o): in function `__ptx2133':
stdThreads.cpp:(.text+0x425): undefined reference to `pthread_key_delete'
/usr/bin/ld: /usr/local/cuda-11.7/lib64/libnvptxcompiler_static.a(ptxtmp245.o): in function `__ptx18094':
stdThreads.cpp:(.text+0x4cc): undefined reference to `pthread_getspecific'
/usr/bin/ld: /usr/local/cuda-11.7/lib64/libnvptxcompiler_static.a(ptxtmp245.o): in function `__ptx18102':
stdThreads.cpp:(.text+0x528): undefined reference to `sem_init'
/usr/bin/ld: stdThreads.cpp:(.text+0x56e): undefined reference to `sem_wait'
/usr/bin/ld: stdThreads.cpp:(.text+0x591): undefined reference to `sem_destroy'
/usr/bin/ld: stdThreads.cpp:(.text+0x5a8): undefined reference to `pthread_key_delete'
/usr/bin/ld: /usr/local/cuda-11.7/lib64/libnvptxcompiler_static.a(ptxtmp245.o): in function `__ptx2076':
stdThreads.cpp:(.text+0x632): undefined reference to `sem_init'
/usr/bin/ld: /usr/local/cuda-11.7/lib64/libnvptxcompiler_static.a(ptxtmp245.o): in function `__ptx18099':
stdThreads.cpp:(.text+0x6b9): undefined reference to `pthread_getspecific'
/usr/bin/ld: stdThreads.cpp:(.text+0x6e2): undefined reference to `pthread_setspecific'
/usr/bin/ld: /usr/local/cuda-11.7/lib64/libnvptxcompiler_static.a(ptxtmp245.o): in function `__ptx18101':
stdThreads.cpp:(.text+0x7ce): undefined reference to `sem_wait'
/usr/bin/ld: /usr/local/cuda-11.7/lib64/libnvptxcompiler_static.a(ptxtmp245.o): in function `__ptx14389':
stdThreads.cpp:(.text+0x836): undefined reference to `pthread_attr_setstacksize'
/usr/bin/ld: stdThreads.cpp:(.text+0x862): undefined reference to `pthread_create'
/usr/bin/ld: /usr/local/cuda-11.7/lib64/libnvptxcompiler_static.a(ptxtmp245.o): in function `__ptx17995':
stdThreads.cpp:(.text+0x939): undefined reference to `pthread_getspecific'
/usr/bin/ld: stdThreads.cpp:(.text+0x962): undefined reference to `pthread_setspecific'
/usr/bin/ld: /usr/local/cuda-11.7/lib64/libnvptxcompiler_static.a(ptxtmp245.o): in function `__ptx17967':
stdThreads.cpp:(.text+0xb7e): undefined reference to `sem_wait'
/usr/bin/ld: /usr/local/cuda-11.7/lib64/libnvptxcompiler_static.a(ptxtmp245.o): in function `__ptx17962':
stdThreads.cpp:(.text+0xbbf): undefined reference to `sem_post'
/usr/bin/ld: /usr/local/cuda-11.7/lib64/libnvptxcompiler_static.a(ptxtmp245.o): in function `__ptx18098':
stdThreads.cpp:(.text+0x128): undefined reference to `sem_post'
/usr/bin/ld: /usr/local/cuda-11.7/lib64/libnvptxcompiler_static.a(ptxtmp245.o): in function `__ptx18097':
stdThreads.cpp:(.text+0x138): undefined reference to `pthread_kill'
/usr/bin/ld: /usr/local/cuda-11.7/lib64/libnvptxcompiler_static.a(ptxtmp245.o): in function `__ptx18074':
stdThreads.cpp:(.text+0x181): undefined reference to `sem_destroy'
/usr/bin/ld: /usr/local/cuda-11.7/lib64/libnvptxcompiler_static.a(ptxtmp245.o): in function `__ptx18076':
stdThreads.cpp:(.text+0x221): undefined reference to `sem_post'
/usr/bin/ld: /usr/local/cuda-11.7/lib64/libnvptxcompiler_static.a(ptxtmp245.o): in function `__ptx2045':
stdThreads.cpp:(.text+0x39d): undefined reference to `sem_post'
/usr/bin/ld: /usr/local/cuda-11.7/lib64/libnvptxcompiler_static.a(ptxtmp245.o): in function `__ptx18094':
stdThreads.cpp:(.text+0x4ea): undefined reference to `pthread_setspecific'
collect2: error: ld returned 1 exit status

我的疑问：

• 明明已通过-lpthread链接pthreads库，为何仍报错？
• stdThreads.cpp来自标准库还是NVIDIA PTX编译库？
• 是否因libnvptxcompiler_static.a是静态库，需调整pthreads的链接方式？

另外，我本地Devuan Daedalus系统上的链接命令可成功完成链接：

/usr/bin/c++ -g \
    CMakeFiles/vectorAdd_ptx.dir/modified_cuda_samples/vectorAdd_ptx/vectorAdd_ptx.cpp.o \
    -o bin/vectorAdd_ptx  \
    /usr/local/cuda/lib64/libcudart.so \
    /usr/local/cuda/lib64/libnvrtc.so /usr/lib/x86_64-linux-gnu/libcuda.so \
    /usr/local/cuda/lib64/libnvptxcompiler_static.a 

问题解析与解决方案 #

1. 加了-lpthread仍报错的原因 #

链接器的库顺序是核心问题：它会严格按照命令行中库的顺序解析符号。静态库libnvptxcompiler_static.a依赖pthread和sem相关符号，但你把-lpthread放在了这个静态库前面——链接器处理-lpthread时，还没遇到需要这些符号的静态库，不会保留这些符号的引用；等到处理静态库时，再回头找-lpthread已经无法回溯。

另外，sem相关的符号还需要链接-lrt（实时库），当前命令里缺少这个库，也是报错的原因之一。

2. stdThreads.cpp的来源 #

从错误信息里的路径libnvptxcompiler_static.a(ptxtmp245.o)可以明确，stdThreads.cpp是NVIDIA PTX编译库内部的源文件，不是系统标准库代码。它是NVIDIA实现线程相关逻辑的模块，依赖pthread和sem接口。

3. 静态库的链接调整方案 #

针对静态库的依赖问题，需要做两个关键调整：

• 把-lpthread和-lrt放在所有依赖它们的静态库之后；
• 遵循“依赖者在前，被依赖者在后”的链接顺序原则。

调整后的链接命令如下：

usr/bin/c++ -O3 -DNDEBUG \
    CMakeFiles/vectorAdd_ptx.dir/modified_cuda_samples/vectorAdd_ptx/vectorAdd_ptx.cpp.o \
    -o bin/vectorAdd_ptx  \
    -Wl,-rpath,/usr/local/cuda-11.7/lib64:/usr/local/cuda-11.7/lib64/stubs \
    /usr/local/cuda-11.7/lib64/libcudart.so \
    /usr/local/cuda-11.7/lib64/libnvrtc.so \
    /usr/local/cuda-11.7/lib64/stubs/libcuda.so \
    /usr/local/cuda-11.7/lib64/libnvptxcompiler_static.a \
    -lpthread \
    -lrt

本地成功的原因 #

你本地Devuan系统无需加-lpthread就能成功，大概率是因为本地编译器/链接器默认自动链接了pthread和rt库（部分发行版默认开启线程支持），或者本地的libnvptxcompiler_static.a依赖关系与Ubuntu 20上的版本不同。

内容的提问来源于stack exchange，提问作者einpoklum