MoveNet Pose Estimation渲染关键点不准确问题排查求助

问题描述 #

我尝试在视频上运行MoveNet Pose Estimation模型，但渲染出的关键点存在严重偏差。推测问题并非源于模型预测本身，而是关键点的计算与绘制环节，但无法定位具体原因。

原实现代码 #

import tensorflow as tf
import numpy as np
from matplotlib import pyplot as plt
import cv2

interpreter = tf.lite.Interpreter(model_path='lite-model_movenet_singlepose_lightning_3.tflite')
interpreter.allocate_tensors()

def draw_keypoints(frame, keypoints, confidence_threshold):
    y, x, c = frame.shape

    shaped = np.squeeze(np.multiply(keypoints, [y,x,1]))
    
    for kp in shaped:
        ky, kx, kp_conf = kp
        if kp_conf > confidence_threshold:
            cv2.circle(frame, (int(kx), int(ky)), 4, (0,255,0), -1) 

cap = cv2.VideoCapture("pushup-stock-compressed.mp4")
while cap.isOpened():
    ret, frame = cap.read()
    
    # Reshape image
    img = frame.copy()
    img = tf.image.resize_with_pad(np.expand_dims(img, axis=0), 192,192)
    input_image = tf.cast(img, dtype=tf.float32)
    
    # Setup input and output 
    input_details = interpreter.get_input_details()
    output_details = interpreter.get_output_details()
    
    # Make predictions 
    interpreter.set_tensor(input_details[0]['index'], np.array(input_image))
    interpreter.invoke()
    keypoints_with_scores = interpreter.get_tensor(output_details[0]['index'])
    
    # Rendering 
    draw_keypoints(frame, keypoints_with_scores, 0.4)
    
    cv2.imshow('MoveNet Lightning', frame)
    
    if cv2.waitKey(10) & 0xFF==ord('q'):
        break
        
cap.release()
cv2.destroyAllWindows()

效果截图 #

问题原因与修复 #

问题核心是MoveNet输出的坐标顺序与代码中的缩放对应关系不匹配：

• MoveNet输出的关键点格式为[x坐标（宽度占比）, y坐标（高度占比）, 置信度]
• 原代码中错误地用[y,x,1]进行缩放，把x和y的映射维度搞反，导致坐标完全错位。

修正后的关键代码 #

def draw_keypoints(frame, keypoints, confidence_threshold):
    y, x, c = frame.shape
    # 修正：缩放数组改为[x,y,1]，匹配MoveNet的x、y输出顺序
    shaped = np.squeeze(np.multiply(keypoints, [x,y,1]))
    
    for kp in shaped:
        kx, ky, kp_conf = kp  # 同步修正变量顺序，对应x、y坐标
        if kp_conf > confidence_threshold:
            cv2.circle(frame, (int(kx), int(ky)), 4, (0,255,0), -1) 

额外性能优化 #

将输入输出细节的获取移到循环外部，避免重复调用浪费资源：

# 移到循环外执行一次即可
input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()

cap = cv2.VideoCapture("pushup-stock-compressed.mp4")
while cap.isOpened():
    # ... 循环内原有代码 ...

内容的提问来源于stack exchange，提问作者Jonas