如何在单GPU（GTX 1050ti）上高效运行多Keras程序并占满GPU？

Hey there! Let's tackle how to get your GTX 1050 Ti running at full capacity for your Python/Keras projects—whether that's making a single model use more GPU power or efficiently running multiple models in parallel.

First: Fix Low GPU Utilization in a Single Program #

If your single model is only using 10% of the GPU, the issue is almost always that the GPU is waiting on something else (like data) or isn't given enough work to do. Here are the top fixes:

• Increase your batch size
  Small batch sizes mean the GPU has very little computation to do between data transfers. Start doubling your batch_size (e.g., from 32 to 64, then 128) until you hit a GPU memory error (OOM). Even if you have to back off a little, a larger batch will keep the GPU busy longer.

• Fix data processing bottlenecks
  More often than not, the CPU is too slow at preparing data for the GPU. Try these tweaks:

  • Use tf.data.Dataset instead of older Keras generators—it's optimized for pipeline parallelism and can offload preprocessing to the GPU where possible.
  • If sticking with ImageDataGenerator, set workers=4 (or match your CPU core count) and use_multiprocessing=True to parallelize data loading.
  • Preprocess your data upfront (save augmented/normalized data to disk) so the CPU doesn't have to do it on the fly during training.

• Check if your model is too lightweight
  A simple model with few layers/parameters won't tax your GPU. If your project allows, try increasing model complexity (e.g., add more convolutional filters, deeper layers) to give the GPU more computation to handle.

• Cut down on CPU-bound overhead
  Frequent model saves, verbose logging, or custom callbacks that run heavy CPU code can starve the GPU. Try saving checkpoints every 5-10 epochs instead of every epoch, and reduce the verbosity of your training loop (set verbose=1 instead of 2).

Second: Efficiently Running Multiple Programs on the Same GPU #

If you still need to run multiple models in parallel, your earlier issue (each epoch doubling in time) is likely due to GPU memory contention and unoptimized resource allocation. Here's how to fix it:

• Enable dynamic GPU memory allocation
  By default, TensorFlow/Keras grabs all available GPU memory on startup, which means multiple programs fight for the same pool. Add this code at the start of each script to let TensorFlow allocate memory only as needed:

  import tensorflow as tf
  gpus = tf.config.list_physical_devices('GPU')
  if gpus:
      try:
          for gpu in gpus:
              tf.config.experimental.set_memory_growth(gpu, True)
      except RuntimeError as e:
          print(e)
  

  Alternatively, you can set a fixed memory fraction per program (e.g., 40% each for two programs) with tf.config.experimental.set_virtual_device_configuration().

• Avoid CPU bottlenecks when running multiple programs
  If two programs are both fighting for CPU resources to preprocess data, the GPU will still sit idle. Assign separate CPU cores to each program (e.g., use os.sched_setaffinity on Linux) or preprocess all data upfront so training doesn't rely on real-time CPU work.

• Train multiple models in a single script (better than separate programs)
  Instead of running two separate Python scripts, build both models in one script and use a custom training loop to parallelize their forward/backward passes. This lets the GPU batch computations more efficiently. For example, you can feed both models' inputs at once and compute losses/gradients in parallel, reducing context-switching overhead between separate processes.

• Adjust process priorities
  Use nvidia-smi to check GPU process IDs, then use your OS's process manager (e.g., renice on Linux, Task Manager on Windows) to prioritize critical models. This ensures important training jobs get more GPU resources when needed.

Final Recommendation #

Start with optimizing a single model first—this will give you the most consistent speedup and avoid the overhead of managing multiple processes. Only move to parallel programs if you absolutely need to train multiple models at once, and use dynamic memory allocation to prevent resource contention.

内容的提问来源于stack exchange，提问作者Srinivas Rampelli