Hey there! I’ve tackled similar video processing setups before, so let’s walk through the best approaches to get your four video streams running smoothly with OpenCV and CUDA 9.0 on Ubuntu 16.04 + GTX 1080.

The key here is to leverage both CPU multi-threading (for managing independent streams) and CUDA’s parallel computing (for accelerating per-frame processing) to avoid bottlenecks.

1. First, Ensure OpenCV is Compiled with CUDA Support #

Ubuntu 16.04’s default OpenCV packages don’t include CUDA acceleration, so you’ll need to build OpenCV from source with these critical flags enabled:

• WITH_CUDA=ON
• CUDA_ARCH_BIN=6.1 (GTX 1080 uses the Pascal architecture, sm_61 is the correct compute capability for CUDA 9.0)
• WITH_FFMPEG=ON (for faster, more reliable video decoding—essential for stream handling)

Once compiled, use cv::cuda::GpuMat instead of regular cv::Mat to store frames on the GPU, and call CUDA-optimized OpenCV functions (like cv::cuda::cvtColor, cv::cuda::resize) to offload processing from the CPU.

2. Use CPU Threads to Manage Individual Video Streams #

Each video stream should run in its own dedicated CPU thread. This keeps stream reading/decoding (a CPU-bound task) isolated, so one slow stream doesn’t block others. Here’s how to structure this:

• Create 4 std::thread instances (one per stream)
• Each thread handles its own cv::VideoCapture instance (never share capture objects across threads—they aren’t thread-safe)
• Inside each thread, bind all GPU operations to a unique cv::cuda::Stream (more on this next)

Example Thread Function Snippet #

void processStream(int streamId, const std::string& source) {
    cv::VideoCapture cap(source, cv::CAP_FFMPEG);
    if (!cap.isOpened()) {
        std::cerr << "Failed to open stream " << streamId << std::endl;
        return;
    }

    // Create a dedicated CUDA stream for this thread
    cv::cuda::Stream cudaStream;
    cv::Mat frame;
    cv::cuda::GpuMat d_frame, d_processed;

    while (cap.read(frame)) {
        // Asynchronously upload frame to GPU (bound to our stream)
        d_frame.upload(frame, cudaStream);

        // Run GPU-accelerated processing steps asynchronously
        cv::cuda::cvtColor(d_frame, d_processed, cv::COLOR_BGR2GRAY, 0, cudaStream);
        cv::cuda::resize(d_processed, d_processed, cv::Size(640, 480), 0, 0, cv::INTER_LINEAR, cudaStream);

        // If you need to bring data back to CPU (e.g., for display), do it asynchronously
        cv::Mat processed;
        d_processed.download(processed, cudaStream);

        // Wait for all GPU operations in this stream to finish before using the CPU frame
        cudaStream.waitForCompletion();

        // Display or save the processed frame
        cv::imshow("Stream " + std::to_string(streamId), processed);
        cv::waitKey(1); // Critical to keep the imshow window responsive
    }

    cap.release();
    cv::destroyWindow("Stream " + std::to_string(streamId));
}

3. Optimize GPU Utilization with CUDA Streams #

CUDA streams let you run multiple GPU tasks asynchronously. By assigning a unique stream to each video thread, the GTX 1080 (which supports concurrent kernel execution) can:

• Upload a frame from Stream 2 while processing a frame from Stream 1
• Resize a frame from Stream 3 while downloading a frame from Stream 4

This drastically boosts GPU utilization compared to using the default synchronous stream. Just make sure all GPU operations for a stream are explicitly bound to its dedicated cv::cuda::Stream instance.

4. Tweak Video Capture for Better Performance #

• For RTSP or network streams, reduce buffer latency with cap.set(cv::CAP_PROP_BUFFERSIZE, 1); (adjust based on your network stability)
• Stick with cv::CAP_FFMPEG as the capture backend (we enabled this during OpenCV compilation) for faster decoding
• Avoid unnecessary CPU-side processing—offload as much as possible to CUDA functions

5. Monitor and Tune Resources #

• Use nvidia-smi in the terminal to track GPU utilization and memory usage. Aim for 70-90% GPU utilization (if it’s too low, your processing tasks are too light; if it’s 100%, you may need to optimize steps or reduce resolution)
• Keep an eye on GPU memory: GTX 1080 has 8GB of VRAM, which is more than enough for four 1080p streams, but if you’re adding extra tasks (like object detection), adjust frame sizes or batch processing accordingly

6. Avoid Common Pitfalls #

• Never share cv::cuda::GpuMat or cv::cuda::Stream instances across threads—each thread needs its own to avoid context conflicts
• Always call cudaStream.waitForCompletion() before using CPU-side data that depends on GPU processing
• Ensure your Ubuntu 16.04 system has a compatible NVIDIA driver for CUDA 9.0 (minimum version 384.81)

Overall, the sweet spot is one CPU thread per video stream, each paired with its own CUDA stream for asynchronous GPU processing. This setup balances CPU and GPU workloads perfectly, and your GTX 1080 should handle four streams without breaking a sweat.

内容的提问来源于stack exchange，提问作者santosh adhikari