训练基于MediaPipe的手语识别模型时遇Numpy序列赋值错误

问题背景 #

基于MediaPipe提取手部关键点训练RandomForestClassifier时，反复触发上述错误，尝试过降级numpy、修改数组声明方式等操作均无效，核心表现为样本特征维度不一致（ragged nested sequences）。

训练代码 #

import pickle

import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score


data_dict = pickle.load(open('./data.pickle', 'rb'))

data = np.asarray(data_dict['data'])
labels = np.asarray(data_dict['labels'])

model = RandomForestClassifier()

model.fit(x_train, y_train)

y_predict = model.predict(x_test)

score = accuracy_score(y_predict, y_test)

print('{}% of samples were classified correctly !'.format(score * 100))

f = open('model.p', 'wb')
pickle.dump({'model': model}, f)
f.close()

报错信息 #

/home/PycharmProjects/SignLanguage/train_classifier.py:13: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.
  data = np.asarray(data_dict['data'])
TypeError: float() argument must be a string or a real number, not 'list'

The above exception was the direct cause of the following exception:


Traceback (most recent call last):
  File "/home/PycharmProjects/SignLanguage/train_classifier.py", line 21, in <module>
    model.fit(x_train, y_train)
  File "/home/PycharmProjects/SignLanguage/venv/lib/python3.10/site-packages/sklearn/base.py", line 1151, in wrapper
    return fit_method(estimator, *args, **kwargs)
  File "/home/PycharmProjects/SignLanguage/venv/lib/python3.10/site-packages/sklearn/ensemble/_forest.py", line 348, in fit
    X, y = self._validate_data(
  File "/home/PycharmProjects/SignLanguage/venv/lib/python3.10/site-packages/sklearn/base.py", line 621, in _validate_data
    X, y = check_X_y(X, y, **check_params)
  File "/home/PycharmProjects/SignLanguage/venv/lib/python3.10/site-packages/sklearn/utils/validation.py", line 1147, in check_X_y
    X = check_array(
  File "/home/PycharmProjects/SignLanguage/venv/lib/python3.10/site-packages/sklearn/utils/validation.py", line 917, in check_array
    array = _asarray_with_order(array, order=order, dtype=dtype, xp=xp)
  File "/home/PycharmProjects/SignLanguage/venv/lib/python3.10/site-packages/sklearn/utils/_array_api.py", line 380, in _asarray_with_order
    array = numpy.asarray(array, order=order, dtype=dtype)
ValueError: setting an array element with a sequence.

Process finished with exit code 1

数据生成代码 #

import mediapipe as mp
import cv2
import os
import pickle
import matplotlib
matplotlib.use('TkAgg')
import matplotlib.pyplot as plt

mp_hands = mp.solutions.hands
mp_drawing = mp.solutions.drawing_utils
mp_drawing_styles = mp.solutions.drawing_styles

hands = mp_hands.Hands(static_image_mode=True, min_detection_confidence=0.3)

DATA_DIR = "./data"

data = []
labels = []

for dir_ in os.listdir(DATA_DIR):
    for img_path in os.listdir(os.path.join(DATA_DIR, dir_)):
        data_aux = []

        img = cv2.imread(os.path.join(DATA_DIR, dir_, img_path))
        img_rgb = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)

        results = hands.process(img_rgb)
        if results.multi_hand_landmarks:
            for hand_landmarks in results.multi_hand_landmarks:
                for i in range(len(hand_landmarks.landmark)):
                    x = hand_landmarks.landmark[i].x
                    y = hand_landmarks.landmark[i].y
                    data_aux.append(x)
                    data_aux.append(y)

            data.append(data_aux)
            labels.append(dir_)

f = open('data.pickle', 'wb')
pickle.dump({'data': data, 'labels': labels}, f)
f.close()

报错根源 #

• 数据集中样本特征长度不一致：MediaPipe处理部分图片时检测到1只手（特征长度42），部分检测到2只手（特征长度84）
• 这种不规则的列表转numpy数组会生成ragged数组，sklearn模型无法处理此类输入

解决方案 #

1. 统一特征维度（推荐） #

修改数据生成代码，强制只检测单只手，并固定特征长度，确保所有样本维度一致：

import mediapipe as mp
import cv2
import os
import pickle

mp_hands = mp.solutions.hands

# 限制只检测单只手，避免多手导致特征长度差异
hands = mp_hands.Hands(static_image_mode=True, min_detection_confidence=0.3, max_num_hands=1)

DATA_DIR = "./data"

data = []
labels = []
# 单只手固定特征长度：21个关键点 × 2（x,y坐标）=42
FIXED_FEATURE_LENGTH = 42

for dir_ in os.listdir(DATA_DIR):
    for img_path in os.listdir(os.path.join(DATA_DIR, dir_)):
        data_aux = []
        img = cv2.imread(os.path.join(DATA_DIR, dir_, img_path))
        img_rgb = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)

        results = hands.process(img_rgb)
        if results.multi_hand_landmarks:
            # 只取第一只手的关键点
            hand_landmarks = results.multi_hand_landmarks[0]
            for i in range(len(hand_landmarks.landmark)):
                x = hand_landmarks.landmark[i].x
                y = hand_landmarks.landmark[i].y
                data_aux.append(x)
                data_aux.append(y)
            
            # 确保特征长度统一，不足补0，超出截断
            if len(data_aux) < FIXED_FEATURE_LENGTH:
                data_aux += [0.0] * (FIXED_FEATURE_LENGTH - len(data_aux))
            elif len(data_aux) > FIXED_FEATURE_LENGTH:
                data_aux = data_aux[:FIXED_FEATURE_LENGTH]
            
            data.append(data_aux)
            labels.append(dir_)

# 重新保存处理后的数据集
with open('data.pickle', 'wb') as f:
    pickle.dump({'data': data, 'labels': labels}, f)

2. 过滤无效样本 #

如果不需要统一特征，直接过滤掉特征长度不符合预期的样本（比如只保留单只手的样本）：
在数据生成代码的data.append(data_aux)前添加判断：

# 只保留单只手的样本
if len(data_aux) == 42:
    data.append(data_aux)
    labels.append(dir_)

3. 修正训练代码（补充缺失步骤） #

原训练代码遗漏了train_test_split划分数据集的步骤，补充完整：

import pickle
import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

data_dict = pickle.load(open('./data.pickle', 'rb'))

data = np.asarray(data_dict['data'])
labels = np.asarray(data_dict['labels'])

# 新增：划分训练集和测试集
x_train, x_test, y_train, y_test = train_test_split(data, labels, test_size=0.2, random_state=42)

model = RandomForestClassifier()
model.fit(x_train, y_train)

y_predict = model.predict(x_test)
score = accuracy_score(y_predict, y_test)

print('{}% of samples were classified correctly !'.format(score * 100))

with open('model.p', 'wb') as f:
    pickle.dump({'model': model}, f)

验证步骤 #

1. 运行修改后的数据生成代码，重新生成data.pickle
2. 运行修正后的训练代码，即可解决维度不一致问题，避免报错

内容的提问来源于stack exchange，提问作者wanted