PyTorch模型处理海康威视IP摄像头流时FPS骤降、延迟过高问题排查

问题概述 #

使用OpenCV处理海康威视IP摄像头RTSP流时，初始状态FPS约20-25、延迟2-3秒，但运行一段时间后FPS快速降至2-3，延迟持续增加最终视频冻结。已尝试GPU加速、多线程、降低分辨率等优化，效果不佳。

用户提供代码 #

VideoLoader类 #

class VideoLoader:
    @staticmethod
    def load_video(video_path):
        cap = cv2.VideoCapture(video_path)
        cap.set(cv2.CAP_PROP_FOURCC, cv2.VideoWriter.fourcc('M', 'J', 'P', 'G'))
        cap.set(cv2.CAP_PROP_FPS, 25)
        cap.set(cv2.CAP_PROP_FRAME_WIDTH, 640)
        cap.set(cv2.CAP_PROP_FRAME_HEIGHT, 480)  
        assert cap.isOpened(), "Video dosyası okunurken hata oluştu"
        return cap

线程处理代码 #

def frame_reader(cap, frame_queue):
    while cap.isOpened():
        success, frame = cap.read()
        if not success:
            break
        if not frame_queue.full():  # Check if the queue has space
            frame_queue.put(frame)
    cap.release()
    frame_queue.put(None)  # End-of-stream signal

def video_processor(frame_queue, output_queue, model, class_names, played_sounds):
    threshold = 0.5  # Tespit eşik değeri
    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
    model.to(device)

    while True:
        frame = frame_queue.get()
        if frame is None:  # End-of-stream signal
            break
        start_time = time.time()
        img = cv2.resize(frame, (640, 480))
        img_rgb = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
        img_tensor = torch.from_numpy(img_rgb).permute(2, 0, 1).float().unsqueeze(0).to(device)
        img_tensor /= 255.0  # Normalize

def video_detection(video_source):
    cap = VideoLoader.load_video(video_source)
    model = YOLO("YOLO-Weights/ppe.pt")
    class_names = ['safety-glasses', 'gloves', 'orange-vest', 'yellow-vest',]
    played_sounds = set()

    frame_queue = Queue(maxsize=5)
    output_queue = Queue(maxsize=5)

    reader_thread = threading.Thread(target=frame_reader, args=(cap, frame_queue), daemon=True)
    processor_thread = threading.Thread(target=video_processor, args=(frame_queue,output_queue, model,class_names, played_sounds), daemon=True)

    reader_thread.start()
    processor_thread.start()

问题诊断 #

1. 摄像头参数设置无效：OpenCV的cap.set()对海康RTSP流基本不生效，RTSP流的编码、分辨率、FPS由摄像头端决定，强制设置会导致解码兼容性问题，长期运行引发缓冲区溢出。
2. 帧读写速率不匹配：frame_reader在队列满时仅跳过入队，但持续调用cap.read()，导致OpenCV内部缓冲区异常；且video_processor代码不完整，推理耗时未优化，处理速度远低于读取速度，帧积压导致延迟飙升。
3. 内存泄漏风险：未释放推理后的Tensor资源，GPU缓存未及时清理，长期运行内存占用持续升高，拖慢程序速度。
4. 线程管理缺失：主线程启动子线程后直接退出，虽设为守护线程，但长期运行可能导致线程资源未正常释放。

解决方案 #

1. 修正摄像头流参数设置 #

海康RTSP流参数需在URL中指定，而非cap.set()，同时使用FFMPEG后端提升稳定性：

class VideoLoader:
    @staticmethod
    def load_video(video_path):
        # 示例海康RTSP URL（替换为实际账号、IP、参数）
        # rtsp://username:password@camera_ip:554/mjpeg/ch1/main/av_stream?resolution=640x480&fps=25
        cap = cv2.VideoCapture(video_path, cv2.CAP_FFMPEG)
        cap.set(cv2.CAP_PROP_BUFFERSIZE, 1)  # 减小缓冲区降低延迟
        assert cap.isOpened(), "无法打开视频流"
        return cap

2. 解决帧读写速率不匹配 #

修改frame_reader，队列满时丢弃旧帧，确保始终保留最新帧：

def frame_reader(cap, frame_queue):
    while cap.isOpened():
        success, frame = cap.read()
        if not success:
            break
        if frame_queue.full():
            frame_queue.get()  # 丢弃最旧帧
        frame_queue.put(frame)
    cap.release()
    frame_queue.put(None)

完善video_processor的推理逻辑，利用YOLO自带前处理优化效率：

def video_processor(frame_queue, output_queue, model, class_names, played_sounds):
    threshold = 0.5
    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
    model.to(device).eval()  # 评估模式减少内存占用

    while True:
        frame = frame_queue.get()
        if frame is None:
            break
        
        # YOLO自带前处理，直接传入BGR帧即可
        results = model(frame, conf=threshold, device=device)
        
        # 后处理示例（按需调整）
        for result in results:
            for box in result.boxes:
                cls = int(box.cls[0].item())
                class_name = class_names[cls]
                # 后续业务逻辑...
        
        # 清理资源避免内存泄漏
        del results
        torch.cuda.empty_cache()  # GPU场景强制清理缓存

3. 完善线程管理 #

主线程等待子线程结束，避免资源异常释放：

def video_detection(video_source):
    cap = VideoLoader.load_video(video_source)
    model = YOLO("YOLO-Weights/ppe.pt")
    class_names = ['safety-glasses', 'gloves', 'orange-vest', 'yellow-vest']
    played_sounds = set()

    frame_queue = Queue(maxsize=2)  # 减小队列大小降低延迟
    output_queue = Queue(maxsize=2)

    reader_thread = threading.Thread(target=frame_reader, args=(cap, frame_queue), daemon=True)
    processor_thread = threading.Thread(target=video_processor, args=(frame_queue, output_queue, model, class_names, played_sounds), daemon=True)

    reader_thread.start()
    processor_thread.start()

    # 等待线程正常结束
    reader_thread.join()
    processor_thread.join()

4. 额外优化建议 #

• 使用YOLOv8原生stream方法：内部已优化流处理逻辑，无需手动线程管理

  def video_detection(video_source):
      model = YOLO("YOLO-Weights/ppe.pt")
      class_names = ['safety-glasses', 'gloves', 'orange-vest', 'yellow-vest']
      played_sounds = set()
  
      for result in model.stream(video_source, stream_buffer=True, conf=0.5, device='cuda', imgsz=640):
          frame = result.orig_img
          # 处理结果逻辑...
  

• 降低推理复杂度：使用更小的YOLO模型（如yolov8n.pt）、减小推理分辨率（如416x416）
• 检查网络带宽：确保摄像头到服务器的网络稳定，避免丢包引发的帧读取异常

内容的提问来源于stack exchange，提问作者Cemoo