TensorFlow中tf.data.Dataset.from_tensor_slices维度不兼容问题求助

问题现象 #

使用TensorFlow（Keras）的DeeplabV3Plus架构进行语义分割时，切换自定义数据集后，调用tf.data.Dataset.from_tensor_slices加载验证集时触发以下错误：

ValueError: Dimensions 37 and 50 are not compatible

训练集数据生成正常，但验证集生成失败。

问题原因 #

tf.data.Dataset.from_tensor_slices要求传入的多个张量（或列表）的第一维度必须完全一致。报错中的37和50分别是val_images和val_masks的长度，说明验证集的图像与掩码数量不匹配。

具体看代码逻辑：

• 代码中设定NUM_TRAIN_IMAGES=250、NUM_VAL_IMAGES=50，期望验证集取250到300索引的文件
• 若DATA_DIR/out或DATA_DIR/Seman文件夹中的文件总数不足300，或者图像与掩码文件数量本身就不一致，就会导致val_images和val_masks的长度不同

解决方案 #

1. 检查数据集文件数量 #

先打印列表长度确认问题：

print("验证集图像数量:", len(val_images))
print("验证集掩码数量:", len(val_masks))

手动检查DATA_DIR下的out和Seman文件夹，确保每个图像都有对应的掩码文件，且总数一致。

2. 修正验证集切片逻辑 #

避免硬编码切片范围导致越界，先获取实际文件总数再计算切片结束索引：

# 获取图像和掩码的完整列表
all_images = sorted(glob(os.path.join(DATA_DIR, "out/*")))
all_masks = sorted(glob(os.path.join(DATA_DIR, "Seman/*")))

# 确保图像和掩码总数一致
assert len(all_images) == len(all_masks), "图像和掩码总数量不匹配"

# 划分训练集和验证集，避免超出实际文件数
total_available = len(all_images)
train_end_idx = min(NUM_TRAIN_IMAGES, total_available)
val_end_idx = min(NUM_TRAIN_IMAGES + NUM_VAL_IMAGES, total_available)

train_images = all_images[:train_end_idx]
train_masks = all_masks[:train_end_idx]
val_images = all_images[train_end_idx:val_end_idx]
val_masks = all_masks[train_end_idx:val_end_idx]

3. 增加数据生成前的校验 #

在data_generator函数开头添加断言，提前发现数量不匹配问题：

def data_generator(image_list, mask_list):
    assert len(image_list) == len(mask_list), f"图像列表长度({len(image_list)})与掩码列表长度({len(mask_list)})不匹配"
    dataset = tf.data.Dataset.from_tensor_slices((image_list, mask_list))
    dataset = dataset.map(load_data, num_parallel_calls=tf.data.experimental.AUTOTUNE)
    dataset = dataset.batch(BATCH_SIZE, drop_remainder=True)
    return dataset

修正后的完整代码 #

import os
import cv2
import numpy as np
from glob import glob
from scipy.io import loadmat
import matplotlib.pyplot as plt

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers


IMAGE_SIZE = 512
BATCH_SIZE = 4
NUM_CLASSES = 20
DATA_DIR = r'C:/Users/Joshi/Desktop/CARLA_0.9.13/WindowsNoEditor/PythonAPI/examples/out'
NUM_TRAIN_IMAGES = 250
NUM_VAL_IMAGES = 50


# 获取完整的图像和掩码列表
all_images = sorted(glob(os.path.join(DATA_DIR, "out/*")))
all_masks = sorted(glob(os.path.join(DATA_DIR, "Seman/*")))

# 提前校验总数量
assert len(all_images) == len(all_masks), "图像和掩码总数量不匹配"

# 划分训练集和验证集，避免越界
total_available = len(all_images)
train_end_idx = min(NUM_TRAIN_IMAGES, total_available)
val_end_idx = min(NUM_TRAIN_IMAGES + NUM_VAL_IMAGES, total_available)

train_images = all_images[:train_end_idx]
train_masks = all_masks[:train_end_idx]
val_images = all_images[train_end_idx:val_end_idx]
val_masks = all_masks[train_end_idx:val_end_idx]


def read_image(image_path, mask=False):
    image = tf.io.read_file(image_path)
    if mask:
        image = tf.image.decode_png(image, channels=1)
        image.set_shape([None, None, 1])
        image = tf.image.resize(images=image, size=[IMAGE_SIZE, IMAGE_SIZE])
    else:
        image = tf.image.decode_png(image, channels=3)
        image.set_shape([None, None, 3])
        image = tf.image.resize(images=image, size=[IMAGE_SIZE, IMAGE_SIZE])
        image = image / 127.5 - 1
    return image


def load_data(image_list, mask_list):
    image = read_image(image_list)
    mask = read_image(mask_list, mask=True)
    return image, mask


def data_generator(image_list, mask_list):
    assert len(image_list) == len(mask_list), f"图像列表长度({len(image_list)})与掩码列表长度({len(mask_list)})不匹配"
    dataset = tf.data.Dataset.from_tensor_slices((image_list, mask_list))
    dataset = dataset.map(load_data, num_parallel_calls=tf.data.experimental.AUTOTUNE)
    dataset = dataset.batch(BATCH_SIZE, drop_remainder=True)
    return dataset


train_dataset = data_generator(train_images, train_masks)
val_dataset = data_generator(val_images, val_masks)

print("Train Dataset:", train_dataset)
print("Val Dataset:", val_dataset)

内容的提问来源于stack exchange，提问作者LostJosh