TensorFlow Multi_Gpu_Model用4块GPU时致Ubuntu 18.04系统崩溃排查求助

Hey there, let’s tackle this problem head-on. Since your code runs fine on Windows 10 but causes Ubuntu to crash and reboot, the issue is almost certainly rooted in system/environment configuration rather than your code—but we’ll cover both angles to be thorough.

First: Rule Out Hardware & Low-Level System Issues #

Reboots under GPU load usually point to hardware or driver instability, especially with 4 RTX 2080 Ti cards. Here’s what to check:

• Power Supply Capacity: Each RTX 2080 Ti draws ~250-300W at full load. For 4 cards plus your CPU, motherboard, and peripherals, you need a 1500W+ 80+ Platinum/Gold certified PSU—and every GPU must have its own dedicated power cables (no splitting). Ubuntu’s power management is often stricter than Windows, so marginal power can trigger hard resets.
• NVIDIA Driver Compatibility: Ubuntu 18.04’s default drivers are outdated for RTX 2080 Ti. Uninstall your current driver and install a stable, compatible version (470.x works great for 20-series cards and most TensorFlow 2.x versions):

  sudo apt purge nvidia*
  sudo apt install nvidia-driver-470
  sudo reboot
  

  After rebooting, run nvidia-smi to confirm all 4 GPUs are detected and the driver is active.
• GPU Temperature & Cooling: Overheating can trigger hardware protection reboots. Check temps with:

  nvidia-smi -q -d TEMPERATURE
  

  If temps exceed 85°C under load, clean dust from your GPUs/case, improve airflow, or adjust fan curves via nvidia-settings.
• Kernel & BIOS Tweaks: Ubuntu 18.04’s default 4.15 kernel may have bugs with multi-GPU setups. Upgrade to the 5.4 LTS kernel (more stable for modern hardware):

  sudo apt install linux-generic-hwe-18.04
  sudo reboot
  

  Also, enter your BIOS and reset settings to default—disable any experimental PCIe power management or SR-IOV options that might interfere with multi-GPU operation.

Next: Validate TensorFlow Environment Configuration #

Mismatched libraries are a common culprit in cross-platform TensorFlow issues:

• CUDA/CuDNN & TensorFlow Version Match: Ensure your Ubuntu TensorFlow, CUDA, and CuDNN versions align exactly with TensorFlow’s official compatibility matrix. For example:
  • TensorFlow 2.4 → CUDA 11.0 + CuDNN 8.0
  • TensorFlow 2.6 → CUDA 11.2 + CuDNN 8.1
    Avoid installing CUDA via deb packages (they often conflict with system libraries); use NVIDIA’s runfile installer and choose to skip driver installation (since you already installed the 470.x driver).
• Optimize GPU Memory Management: Add these environment variables to your shell profile (~/.bashrc or ~/.zshrc) to prevent TensorFlow from hogging all GPU memory at once:

  export CUDA_VISIBLE_DEVICES=0,1,2,3
  export TF_FORCE_GPU_ALLOW_GROWTH=true
  

  TF_FORCE_GPU_ALLOW_GROWTH=true lets TensorFlow allocate memory on-demand, reducing the chance of system-wide memory exhaustion.
• Limit CPU Resources During Compilation: Compiling the model on CPU can overload your system if left unchecked. Add these lines before creating your model to cap CPU threads:

  import tensorflow as tf
  tf.config.threading.set_intra_op_parallelism_threads(8)
  tf.config.threading.set_inter_op_parallelism_threads(8)
  

  Adjust the thread count based on your CPU’s core count (e.g., use half your total cores).

Finally: Check Code for Ubuntu-Specific Compatibility #

While your code works on Windows, there’s one key tweak for Ubuntu/TensorFlow 2.x:

• Replace Deprecated multi_gpu_model: The multi_gpu_model function is deprecated in TensorFlow 2.x and can cause stability issues on Linux. Switch to TensorFlow’s official MirroredStrategy, which is designed for multi-GPU training and more reliable:

  # Replace your existing model creation code with this
  strategy = tf.distribute.MirroredStrategy()
  with strategy.scope():
      model = create_image_model()
      # Add your model.compile() call here too
  

  This handles device placement and parallelization natively, avoiding the quirks of the old multi_gpu_model function.
• Pre-Enable GPU Memory Growth: Add this code before creating your model to ensure TensorFlow doesn’t reserve all GPU memory upfront:

  gpus = tf.config.list_physical_devices('GPU')
  if gpus:
      for gpu in gpus:
          tf.config.experimental.set_memory_growth(gpu, True)
  

Quick Recap #

Start with hardware/driver checks (power, temps, kernel) since those are the most likely causes of reboots. Then validate your CUDA/TensorFlow environment, and finally update your code to use TF2.x’s recommended multi-GPU strategy.

内容的提问来源于stack exchange，提问作者Chetak Kandaswamy