为何GPU与CPU运行我的TensorFlow梯度下降脚本耗时相同？

Great question! Let’s break down the key reasons why your K80 GPU and CPU are performing similarly with this specific setup and workload:

• Your workload is too small to leverage GPU parallelism
  GPUs shine with large, highly parallelizable tasks—but your model is extremely tiny: only 5 variables and 100 sigmoid terms. The overhead of transferring data between CPU and GPU (via PCIe) and initializing GPU resources can easily outweigh any speed gains from parallel computation. For such a minimal workload, a modern CPU (especially a multi-core instance on GCP) can handle the calculations just as quickly, since the GPU’s thousands of cores aren’t being utilized effectively.

• GPU initialization overhead is eating into potential gains
  When TensorFlow runs on GPU for the first time, it has to set up the GPU context, load CUDA kernels, and allocate device memory. For a short job like 100 iterations, this initialization time might make up a big chunk of your total 6-7 seconds. The CPU doesn’t have this startup overhead, so even if the GPU is faster during actual computation, the setup costs bring the total runtime in line with the CPU.

• Adam Optimizer’s overhead dominates for small parameter counts
  Adam maintains per-parameter state (first and second moment estimates) which adds computational overhead. With only 5 parameters, this overhead is relatively large compared to the simple sigmoid sum calculations. Both CPU and GPU handle this small amount of state management efficiently, so there’s no noticeable gap in total runtime.

• The K80 GPU is overkill for this tiny workload
  The K80 is designed for heavy parallel tasks like large batch training or complex neural networks. For a workload this small, even a mid-range CPU can keep up because the task doesn’t require the massive parallel processing power the K80 provides.

What to try next to see GPU speedups #

To observe a clear GPU advantage, scale up your workload:

• Increase the number of variables (e.g., to 1000+)
• Boost the number of sigmoid terms (e.g., to 10,000+)
• Run far more iterations (e.g., 10,000+ instead of 100)

As the computation becomes large enough to amortize the GPU’s initialization and data transfer overhead, you’ll start to see the GPU pull ahead significantly.

内容的提问来源于stack exchange，提问作者Rodolphe LAMPE