grouped_gemm安装失败：PyTorch与CUDA版本不匹配疑云

问题背景 #

部署多模态模型ARIA时，需安装grouped_gemm加速框架提升推理速度，但执行pip install grouped_gemm==0.1.6时安装失败，报错显示检测到的CUDA版本（11.8）与编译PyTorch的CUDA版本（12.4）不匹配，可本地已反复确认未安装基于CUDA12.4编译的PyTorch版本。

错误日志 #

Collecting grouped_gemm==0.1.6
Using cached grouped_gemm-0.1.6.tar.gz (978 kB)
Installing build dependencies ... done
Getting requirements to build wheel ... done
Installing backend dependencies ... done
Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: grouped_gemm
Building wheel for grouped_gemm (pyproject.toml) ... error
error: subprocess-exited-with-error

× Building wheel for grouped_gemm (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [67 lines of output]
/tmp/pip-build-env-uuagnhy5/overlay/lib/python3.10/site-packages/torch/_subclasses/functional_tensor.py:275: UserWarning: Failed to initialize NumPy: No module named 'numpy' (Triggered internally at /pytorch/torch/csrc/utils/tensor_numpy.cpp:81.)
cpu = _conversion_method_template(device=torch.device("cpu"))
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-cpython-310
creating build/lib.linux-x86_64-cpython-310/grouped_gemm
copying grouped_gemm/ops_test.py -> build/lib.linux-x86_64-cpython-310/grouped_gemm
copying grouped_gemm/__init__.py -> build/lib.linux-x86_64-cpython-310/grouped_gemm
copying grouped_gemm/ops.py -> build/lib.linux-x86_64-cpython-310/grouped_gemm
copying grouped_gemm/backend.py -> build/lib.linux-x86_64-cpython-310/grouped_gemm
running build_ext
Traceback (most recent call last):
File "/root/anaconda3/envs/unsloth_env3/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
main()
File "/root/anaconda3/envs/unsloth_env3/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
File "/root/anaconda3/envs/unsloth_env3/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 251, in build_wheel
return _build_backend().build_wheel(wheel_directory, config_settings,
File "/tmp/pip-build-env-uuagnhy5/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 410, in build_wheel
return self._build_with_temp_dir(
File "/tmp/pip-build-env-uuagnhy5/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 395, in _build_with_temp_dir
self.run_setup()
File "/tmp/pip-build-env-uuagnhy5/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 311, in run_setup
exec(code, locals())
File "<string>", line 58, in <module>
File "/tmp/pip-build-env-uuagnhy5/overlay/lib/python3.10/site-packages/setuptools/__init__.py", line 104, in setup
return distutils.core.setup(**attrs)
File "/tmp/pip-build-env-uuagnhy5/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 184, in setup
return run_commands(dist)
File "/tmp/pip-build-env-uuagnhy5/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 200, in run_commands
dist.run_commands()
File "/tmp/pip-build-env-uuagnhy5/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
self.run_command(cmd)
File "/tmp/pip-build-env-uuagnhy5/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 967, in run_command
super().run_command(command)
File "/tmp/pip-build-env-uuagnhy5/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
cmd_obj.run()
File "/tmp/pip-build-env-uuagnhy5/normal/lib/python3.10/site-packages/wheel/_bdist_wheel.py", line 387, in run
self.run_command("build")
File "/tmp/pip-build-env-uuagnhy5/overlay/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 316, in run_command
self.distribution.run_command(command)
File "/tmp/pip-build-env-uuagnhy5/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 967, in run_command
super().run_command(command)
File "/tmp/pip-build-env-uuagnhy5/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
cmd_obj.run()
File "/tmp/pip-build-env-uuagnhy5/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/build.py", line 132, in run
self.run_command(cmd_name)
File "/tmp/pip-build-env-uuagnhy5/overlay/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 316, in run_command
self.distribution.run_command(command)
File "/tmp/pip-build-env-uuagnhy5/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 967, in run_command
super().run_command(command)
File "/tmp/pip-build-env-uuagnhy5/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
cmd_obj.run()
File "/tmp/pip-build-env-uuagnhy5/overlay/lib/python3.10/site-packages/setuptools/command/build_ext.py", line 91, in run
_build_ext.run(self)
File "/tmp/pip-build-env-uuagnhy5/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/build_ext.py", line 359, in run
self.build_extensions()
File "/tmp/pip-build-env-uuagnhy5/overlay/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 552, in build_extensions
_check_cuda_version(compiler_name, compiler_version)
File "/tmp/pip-build-env-uuagnhy5/overlay/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 447, in _check_cuda_version
raise RuntimeError(CUDA_MISMATCH_MESSAGE.format(cuda_str_version, torch.version.cuda))
RuntimeError:
The detected CUDA version (11.8) mismatches the version that was used to compile
PyTorch (12.4). Please make sure to use the same CUDA versions.

      [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for grouped_gemm
Failed to build grouped_gemm
ERROR: Could not build wheels for grouped_gemm, which is required to install pyproject.toml-based projects

本地环境配置 #

• 本地通过以下命令安装了对应CUDA11.8的PyTorch：

  pip install torch==2.1.1+cu118 torchvision==0.16.1+cu118 torchaudio==2.1.1 --index-url https://download.pytorch.org/whl/cu118
  

• 多环境CUDA版本情况：
  • Windows本地：nvcc --version显示CUDA12.1
  • Linux（WSL）及Conda环境：nvcc --version显示CUDA11.8
  • Conda环境内确认PyTorch版本为2.1.1+cu118，CUDA版本11.8且可用
  • nvidia-smi显示系统驱动仅支持CUDA12.3

已做排查 #

• 检查环境路径：CUDA_HOME、PATH、PYTHONPATH、LD_LIBRARY_PATH
• 尝试指定路径安装，均无效：
  • CUDA_HOME=/usr/local/cuda-11.8 pip install grouped_gemm==0.1.6
  • /root/anaconda3/envs/unsloth_env3/bin/python -m pip install grouped_gemm==0.1.6

核心疑问 #

1. grouped_gemm在临时构建目录（/tmp/pip-build-env-uuagnhy5/overlay/lib/python3.10/site-packages/torch/）安装时，是否自行下载并使用了基于CUDA12.4编译的PyTorch，绕过了本地已安装的版本？
2. 是否意味着grouped_gemm必须在CUDA12.4环境中才能安装？

内容的提问来源于stack exchange，提问作者Eirudy L