Ubuntu20.04下TensorFlow GPU运行报错：libdevice未找到，求解决

按照TensorFlow官方pip安装教程完成安装，执行步骤6验证命令输出正常，但运行自定义模型时出现如下错误：

2023-01-06 18:39:14.692537: W tensorflow/compiler/xla/service/gpu/nvptx_helper.cc:56] Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice. This may result in compilation or runtime failures, if the program we try to run uses routines from libdevice.
Searched for CUDA in the following directories:
  ./cuda_sdk_lib
  /usr/local/cuda-11.2
  /usr/local/cuda
  .
You can choose the search directory by setting xla_gpu_cuda_data_dir in HloModule's DebugOptions.  For most apps, setting the environment variable XLA_FLAGS=--xla_gpu_cuda_data_dir=/path/to/cuda will work.
2023-01-06 18:39:14.693094: W tensorflow/compiler/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:326] libdevice is required by this HLO module but was not found at ./libdevice.10.bc
2023-01-06 18:39:14.693196: I tensorflow/compiler/jit/xla_compilation_cache.cc:477] Compiled cluster using XLA!  This line is logged at most once for the lifetime of the process.
2023-01-06 18:39:14.693275: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at xla_ops.cc:446 : INTERNAL: libdevice not found at ./libdevice.10.bc
2023-01-06 18:39:14.704458: W tensorflow/compiler/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:326] libdevice is required by this HLO module but was not found at ./libdevice.10.bc
2023-01-06 18:39:14.704603: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at xla_ops.cc:446 : INTERNAL: libdevice not found at ./libdevice.10.bc
Traceback (most recent call last):
  File "/home/jerry/Woodburn/Woodburn_Model/model/main/Model_Main.py", line 42, in <module>
    main(sys.argv[1:])
  File "/home/jerry/Woodburn/Woodburn_Model/model/main/Model_Main.py", line 27, in main
    model.train()
  File "/home/jerry/Woodburn/Woodburn_Model/model/main/Model_V5.py", line 99, in train
    history = self.model.fit(x, y, batch_size = batchSize, epochs = epochs)
  File "/home/jerry/miniconda3/envs/tensorflow_gpu/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 70, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/home/jerry/miniconda3/envs/tensorflow_gpu/lib/python3.9/site-packages/tensorflow/python/eager/execute.py", line 52, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.InternalError: Graph execution error:

Detected at node 'StatefulPartitionedCall_10' defined at (most recent call last):
    File "/home/jerry/Woodburn/Woodburn_Model/model/main/Model_Main.py", line 42, in <module>
      main(sys.argv[1:])
    File "/home/jerry/Woodburn/Woodburn_Model/model/main/Model_Main.py", line 27, in main
      model.train()
    File "/home/jerry/Woodburn/Woodburn_Model/model/main/Model_V5.py", line 99, in train
      history = self.model.fit(x, y, batch_size = batchSize, epochs = epochs)
    File "/home/jerry/miniconda3/envs/tensorflow_gpu/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 65, in error_handler
      return fn(*args, **kwargs)
    File "/home/jerry/miniconda3/envs/tensorflow_gpu/lib/python3.9/site-packages/keras/engine/training.py", line 1650, in fit
      tmp_logs = self.train_function(iterator)
    File "/home/jerry/miniconda3/envs/tensorflow_gpu/lib/python3.9/site-packages/keras/engine/training.py", line 1249, in train_function
      return step_function(self, iterator)
    File "/home/jerry/miniconda3/envs/tensorflow_gpu/lib/python3.9/site-packages/keras/engine/training.py", line 1233, in step_function
      outputs = model.distribute_strategy.run(run_step, args=(data,))
    File "/home/jerry/miniconda3/envs/tensorflow_gpu/lib/python3.9/site-packages/keras/engine/training.py", line 1222, in run_step
      outputs = model.train_step(data)
    File "/home/jerry/miniconda3/envs/tensorflow_gpu/lib/python3.9/site-packages/keras/engine/training.py", line 1027, in train_step
      self.optimizer.minimize(loss, self.trainable_variables, tape=tape)
    File "/home/jerry/miniconda3/envs/tensorflow_gpu/lib/python3.9/site-packages/keras/optimizers/optimizer_experimental/optimizer.py", line 527, in minimize
      self.apply_gradients(grads_and_vars)
    File "/home/jerry/miniconda3/envs/tensorflow_gpu/lib/python3.9/site-packages/keras/optimizers/optimizer_experimental/optimizer.py", line 1140, in apply_gradients
      return super().apply_gradients(grads_and_vars, name=name)
    File "/home/jerry/miniconda3/envs/tensorflow_gpu/lib/python3.9/site-packages/keras/optimizers/optimizer_experimental/optimizer.py", line 634, in apply_gradients
      iteration = self._internal_apply_gradients(grads_and_vars)
    File "/home/jerry/miniconda3/envs/tensorflow_gpu/lib/python3.9/site-packages/keras/optimizers/optimizer_experimental/optimizer.py", line 1166, in _internal_apply_gradients
      return tf.__internal__.distribute.interim.maybe_merge_call(
    File "/home/jerry/miniconda3/envs/tensorflow_gpu/lib/python3.9/site-packages/keras/optimizers/optimizer_experimental/optimizer.py", line 1216, in _distributed_apply_gradients_fn
      distribution.extended.update(
    File "/home/jerry/miniconda3/envs/tensorflow_gpu/lib/python3.9/site-packages/keras/optimizers/optimizer_experimental/optimizer.py", line 1211, in apply_grad_to_update_var
      return self._update_step_xla(grad, var, id(self._var_key(var)))
Node: 'StatefulPartitionedCall_10'
libdevice not found at ./libdevice.10.bc
         [[{{node StatefulPartitionedCall_10}}]] [Op:__inference_train_function_8591]

核心错误为XLA无法找到CUDA的libdevice文件，运行环境：Ubuntu 20.04、Python 3.9。

1. 定位CUDA与libdevice的实际路径 #

首先确认你的CUDA安装位置：

• 系统级安装通常在/usr/local/cuda或/usr/local/cuda-11.x，进入该目录后检查nvvm/libdevice/下是否存在libdevice.10.bc文件。
• Conda虚拟环境中的CUDA路径类似~/miniconda3/envs/tensorflow_gpu/lib/python3.9/site-packages/nvidia/cuda_nvvm/libdevice/，同样确认该目录下的libdevice文件。

2. 临时设置环境变量运行模型 #

在启动模型前，通过环境变量指定CUDA根目录：

# 替换为你的CUDA实际路径
export XLA_FLAGS=--xla_gpu_cuda_data_dir=/usr/local/cuda-11.2
# 运行模型
python Model_Main.py

也可直接在一行命令中完成：

XLA_FLAGS=--xla_gpu_cuda_data_dir=/usr/local/cuda-11.2 python Model_Main.py

3. 永久设置环境变量（可选） #

若不想每次手动设置，可将环境变量添加到shell配置文件：

• Bash用户：编辑~/.bashrc，添加以下内容：

export XLA_FLAGS=--xla_gpu_cuda_data_dir=/usr/local/cuda-11.2

执行source ~/.bashrc使设置立即生效。

• Zsh用户：编辑~/.zshrc，添加上述内容后执行source ~/.zshrc。

4. 验证设置有效性 #

运行模型前，执行以下命令确认环境变量已正确设置：

echo $XLA_FLAGS

输出应显示你指定的CUDA路径。

额外检查 #

确保TensorFlow版本与CUDA版本兼容，比如TensorFlow 2.8+对应CUDA 11.2，版本不匹配也可能导致类似问题。

内容的提问来源于stack exchange，提问作者David3186