融合数值、类别与图像特征构建机器学习模型时遇ValueError求助

尝试将数值、类别和图像特征合并为机器学习模型的单一特征集时出现ValueError，已完成特征提取与预处理流程，但问题未解决。操作流程如下：

• 加载并预处理数值和类别特征
• 使用预训练CNN模型提取并预处理图像特征
• 将三类特征合并为单一数据集

代码与错误信息 #

# Features and target variable
X = data[["ID",'Thinckness', 'Weight', 'Surface', 'Color', 'Transparence']]
y = data['Material']

# Image loading function
def load_image(image_id, base_path='Camera2/front'):
    # Replace with the path to your images directory
    image_path = f"{base_path}/{image_id}.jpeg"
    try:
        with Image.open(image_path) as img:
            img = img.resize((128, 128))  # Resize image
            return np.array(img)
    except FileNotFoundError:
        # Return NaN or a placeholder image (e.g., all zeros)
        print(f"Image file not found: {image_path}")
        return np.full((128, 128, 3), np.nan)  # Return a placeholder image with NaN values
# Load images
images = np.array([load_image(image_id) for image_id in data['ID']])

from tensorflow.keras.applications import VGG16
from tensorflow.keras.applications.vgg16 import preprocess_input
from tensorflow.keras.models import Model
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer


# Load a pre-trained CNN model for feature extraction
base_model = VGG16(weights='imagenet', include_top=False, input_shape=(128, 128, 3))
model = Model(inputs=base_model.input, outputs=base_model.output)


def extract_features_from_images(images):
    features = []
    for img in images:
        if np.isnan(img).any():  # Check if the image contains NaN values
            features.append(np.zeros((4 * 4 * 512,)))  # Return a zero-filled vector as placeholder
        else:
            img = preprocess_input(img)
            img = np.expand_dims(img, axis=0)
            feature = model.predict(img)
            features.append(feature.flatten())
    return np.array(features)

# Extract image features
image_features = extract_features_from_images(images)


if image_features.ndim == 3:
    # Flatten the image features to 2D: [n_samples, height * width * channels]
    image_features = image_features.reshape(image_features.shape[0], -1)

# Define numerical and categorical features
numeric_features = ['Thinckness', 'Weight', 'Surface']
categorical_features = ['Color', 'Transparence']

# Preprocessing for numerical data
numeric_transformer = Pipeline(steps=[
    ('imputer', SimpleImputer(strategy='mean')),  # Handle missing values
    ('scaler', StandardScaler())  # Normalize numerical data
])

# Preprocessing for categorical data
categorical_transformer = Pipeline(steps=[
    ('imputer', SimpleImputer(strategy='most_frequent')),  # Handle missing values
    ('onehot', OneHotEncoder(handle_unknown='ignore'))  # One-hot encode categorical data
])

# Combine preprocessing steps
preprocessor = ColumnTransformer(
    transformers=[
        ('num', numeric_transformer, numeric_features),
        ('cat', categorical_transformer, categorical_features)
    ])

numeric_categorical_features = preprocessor.fit_transform(data[numeric_features + categorical_features])


# Combine numerical, categorical, and image features
combined_features = np.hstack([
    numeric_categorical_features,
    image_features
])

特征形状 #

Numeric/Categorical Features Shape: (1099, 19)
Image Features Shape: (1099, 8192)

错误信息 #

ValueError                                Traceback (most recent call last)

Cell In[103], line 2
      1 # Combine numerical, categorical, and image features
----> 2 combined_features = np.hstack([
      3     numeric_categorical_features,
      4     image_features
      5 ])


ValueError: all the input arrays must have same number of dimensions, but the array at index 0 has 1 dimension(s) and the array at index 1 has 2 dimension(s)

解决方案 #

问题根源 #

preprocessor.fit_transform()返回的是scipy稀疏矩阵，而非numpy数组。np.hstack无法正确识别稀疏矩阵的维度，导致误判为1维，与2维的image_features数组无法合并。

修改步骤 #

将稀疏的数值类别特征转换为numpy数组，使用.toarray()方法即可，修改后的代码片段如下：

# 预处理数值和类别特征并转换为numpy数组
numeric_categorical_features = preprocessor.fit_transform(data[numeric_features + categorical_features]).toarray()

# 合并三类特征
combined_features = np.hstack([
    numeric_categorical_features,
    image_features
])

备选方案（大数据场景） #

若数据集特征维度极大，转换为numpy数组会占用过多内存，可使用scipy.sparse.hstack合并稀疏矩阵与数组（需将数组转为稀疏矩阵）：

from scipy.sparse import hstack, csr_matrix

# 将图像特征转为稀疏矩阵
image_features_sparse = csr_matrix(image_features)

# 合并稀疏矩阵
combined_features = hstack([numeric_categorical_features, image_features_sparse])

内容的提问来源于stack exchange，提问作者Agura