升级NVIDIA驱动至384.111后出现CUDA/cuDNN错误求助

Hey there, let's dig into this issue you're hitting after updating your NVIDIA driver. I've seen similar problems pop up with version mismatches between drivers, CUDA, cuDNN, and TensorFlow, so let's break it down step by step.

核心错误原因 #

The CUDNN_STATUS_INTERNAL_ERROR and subsequent crash usually boil down to one of these key issues:

1. 驱动-CUDA-cuDNN-TensorFlow版本兼容性冲突
   TensorFlow 1.4.1 was built to work with specific versions of CUDA and NVIDIA drivers. While CUDA 8.0 does support drivers newer than 384.90, the 384.111 update introduced subtle changes that break compatibility with cuDNN 7.0 and TensorFlow 1.4.1. Official TensorFlow 1.4.1 docs recommend CUDA 8.0 paired with cuDNN 6.0 (not 7.0) and NVIDIA drivers in the 375.x to 384.98 range—384.111 pushes beyond that stable compatibility window.

2. GPU显存资源不足
   Sometimes even if versions are compatible, your desktop environment (like Cinnamon in Mint) or background apps might be hogging GPU memory, leaving insufficient space for TensorFlow to initialize the cuDNN handle.

3. cuDNN文件损坏/权限问题
   Updating the driver can occasionally overwrite or alter permissions on cuDNN library files, even if you didn't touch them directly.

解决方法（按优先级排序） #

1. 回滚NVIDIA驱动到稳定兼容版本 #

This is the most likely fix for your setup. Roll back to the 384.90 driver you were using before, or the slightly newer 384.98 (both are validated to work with CUDA 8.0):

• Using Update Manager (GUI):

  1. Open Update Manager > Go to "View" > Select "Linux Kernels" > Switch to the "Additional Drivers" tab
  2. Find the "NVIDIA driver metapackage from nvidia-384 (version 384.90)" option
  3. Select it, click "Apply Changes", then restart your system

• Using Command Line:
  First purge the current driver:

  sudo apt-get purge nvidia-*
  

  Then install the specific 384.90 version:

  sudo apt-get install nvidia-384=384.90-0ubuntu0.16.04.1
  

  Lock the version to prevent accidental updates later (optional but recommended):

  sudo apt-mark hold nvidia-384
  

  Finally, restart your system.

2. 配置TensorFlow显存自适应增长 #

If rolling back the driver doesn't work (or you want to keep the new driver), try limiting how much GPU memory TensorFlow uses to avoid conflicts with other processes:

Add this code at the very start of your TensorFlow script:

import tensorflow as tf

# Enable dynamic GPU memory allocation
config = tf.ConfigProto()
config.gpu_options.allow_growth = True
# Optional: Limit memory usage to 70% of GPU capacity
# config.gpu_options.per_process_gpu_memory_fraction = 0.7

# Initialize your session with this config
with tf.Session(config=config) as sess:
    # Your model code here

3. 重新安装/验证cuDNN 7.0 #

Double-check that your cuDNN files are intact and have the correct permissions:

1. Download the cuDNN 7.0 package for CUDA 8.0 (you'll need an NVIDIA developer account)
2. Extract the archive, then copy the files to your CUDA directory:

   sudo cp include/cudnn.h /usr/local/cuda/include
   sudo cp lib64/libcudnn* /usr/local/cuda/lib64
   

3. Set proper permissions:

   sudo chmod a+r /usr/local/cuda/include/cudnn.h /usr/local/cuda/lib64/libcudnn*
   

4. Update the system library cache:

   sudo ldconfig
   

内容的提问来源于stack exchange，提问作者lorenzo