Jetson Xavier平台OpenCV DNN结合ONNX模型GPU推理性能异常排查求助

Based on your setup details and the symptoms you're seeing—low GPU utilization despite enabling CUDA backend, slow inference even in MAXN power mode—I’ve broken down the most likely issues and actionable fixes below:

1. Optimize the YOLOv5 ONNX Model for OpenCV DNN #

OpenCV’s CUDA backend works best with simplified, fixed-input-size ONNX models. Your current model might have dynamic dimensions or unoptimized layers that prevent full GPU utilization.

• Re-export your YOLOv5m model with these flags to simplify the structure and lock in your target input resolution:

  python export.py --weights yolov5m.pt --include onnx --simplify --img-size 1280 720
  

• This removes redundant operations and ensures the model expects a fixed 1280×720 input, which helps the CUDA backend pre-allocate memory and optimize kernels upfront.

2. Switch to FP16 Inference for Jetson Hardware #

Jetson Xavier has native support for FP16 arithmetic, which is faster and more memory-efficient than FP32. Your current code uses DNN_TARGET_CUDA (default FP32)—switching to FP16 can drastically boost GPU utilization and speed:

net = cv2.dnn.readNetFromONNX("yolov5m.onnx")
net.setPreferableBackend(cv2.dnn.DNN_BACKEND_CUDA)
# Use FP16 target for Jetson's optimized hardware
net.setPreferableTarget(cv2.dnn.DNN_TARGET_CUDA_FP16)

3. Minimize CPU-GPU Data Transfer Overhead #

Low GPU utilization often stems from bottlenecks in moving data between CPU and GPU. If you’re using regular cv2.Mat objects, every inference requires copying data to the GPU, leaving the GPU idle while waiting for data.

• Use cv2.cuda.GpuMat to keep image data on the GPU throughout preprocessing and inference:

  # Upload input image to GPU once
  img_gpu = cv2.cuda.GpuMat()
  img_gpu.upload(your_input_image)
  
  # Resize on GPU (avoids CPU-GPU copy)
  img_resized_gpu = cv2.cuda.resize(img_gpu, (1280, 720))
  
  # Create blob directly from GPU mat (reduces transfer)
  blob_gpu = cv2.dnn.blobFromGpuMat(
      img_resized_gpu, 
      1/255.0, 
      (1280, 720), 
      swapRB=True, 
      crop=False
  )
  
  net.setInput(blob_gpu)
  outputs = net.forward()
  

• This keeps the GPU busy with inference instead of waiting for data transfers.

4. Compile OpenCV with TensorRT Support (Critical for Jetson) #

Your current OpenCV build doesn’t include TensorRT integration, which is the most impactful optimization for ONNX models on Jetson. TensorRT generates hardware-specific optimized engines that outperform OpenCV’s native CUDA backend by a wide margin.

• Re-compile OpenCV with TensorRT enabled:

  cd your_opencv_build_directory
  cmake -D CMAKE_BUILD_TYPE=RELEASE \
        -D CMAKE_INSTALL_PREFIX=/usr/local \
        -D WITH_CUDA=ON \
        -D CUDA_ARCH_BIN="7.2;8.7" \
        -D WITH_CUDNN=ON \
        -D WITH_TENSORRT=ON \
        -D OPENCV_DNN_CUDA=ON \
        -D OPENCV_EXTRA_MODULES_PATH=../opencv_contrib-4.6.0/modules \
        -D OPENCV_ENABLE_NONFREE=OFF \
        ..
  make -j$(nproc)
  sudo make install
  

• Then update your code to use the TensorRT backend:

  net = cv2.dnn.readNetFromONNX("yolov5m.onnx")
  net.setPreferableBackend(cv2.dnn.DNN_BACKEND_TENSORRT)
  net.setPreferableTarget(cv2.dnn.DNN_TARGET_CUDA_FP16)
  

• TensorRT will automatically optimize the ONNX model for Jetson Xavier’s GPU, drastically increasing GPU utilization and reducing inference time.

5. Force Jetson to Run at Maximum Clocks #

Even in MAXN mode, Jetson might not always run at full clock speeds. Use the jetson_clocks tool to lock hardware at maximum performance:

sudo jetson_clocks

• You can verify clock speeds and GPU utilization using jtop (install with sudo pip install jetson-stats) to confirm the GPU is running at full capacity.

6. Verify CUDA Layer Support in OpenCV #

Some layers in your YOLOv5 model might not be supported by OpenCV’s CUDA backend, falling back to CPU execution and causing GPU idling. Check which layers are using CUDA:

layers = net.getLayerNames()
for idx in range(net.getLayerCount()):
    layer = net.getLayer(idx)
    print(f"Layer {idx}: {layer.name} | Backend: {layer.preferableBackend} | Target: {layer.preferableTarget}")

• If you see layers using DNN_BACKEND_DEFAULT (CPU) instead of DNN_BACKEND_CUDA, consider upgrading to a newer OpenCV version (4.8.x or later) which has better YOLOv5 layer support for CUDA.

内容的提问来源于stack exchange，提问作者Arman Paknia