不同时长语音频谱图提取与RNN适配问题求解

问题根源 #

1. 音频时长不一致，导致STFT提取的频谱图时间帧数量（第一维）各不相同，最终存储的specs是不同形状数组的列表
2. 存储时将频谱图转成了list，加载后np.array(DATA["x_train"])得到的是(4528,)的object类型数组，无法被Keras转换成统一形状的张量
3. 训练代码中标签加载、频谱图加载逻辑存在错误

分步解决方案 #

1. 统一音频信号长度 #

在提取频谱图前，将所有音频调整为固定长度（比如1秒），短音频补零，长音频截断。修改save_spec函数中的音频加载与处理逻辑：

def save_spec(npz_path, dataset_path, sample_rate=22050, hop_length=512, n_fft=2048):
    data = {"mapping": [], "specs": [], "labels": []}
    target_length = sample_rate  # 统一为1秒长度，对应22050个采样点
    
    for i, (dirpath, dirname, filenames) in enumerate(os.walk(dataset_path)):
        if dirpath != dataset_path:
            semantic_label = dirpath.split("/")[-1]
            data["mapping"].append(semantic_label)
            
            for file in filenames:
                file_path = os.path.join(dirpath, file)
                try:
                    print(Fore.CYAN + f"Loading File...: {file}")
                    Signal, _ = librosa.load(file_path, sr=sample_rate)
                    
                    # 统一音频长度：短补零，长截断
                    if len(Signal) < target_length:
                        Signal = np.pad(Signal, (0, target_length - len(Signal)), mode='constant')
                    else:
                        Signal = Signal[:target_length]
                        
                except Exception as e:
                    print(Fore.RED + f"Loading FAILED: {str(e)}")
                    continue
                
                try:
                    print(Fore.BLUE + "\tExtracting Spectrogram...")
                    spectrogram = librosa.core.stft(Signal, n_fft=n_fft, hop_length=hop_length)
                    spectrogram = np.abs(spectrogram).T  # 转置为(时间帧, 频率特征)格式
                except Exception as e:
                    print(Fore.RED + f"\tExtracting FAILED: {str(e)}")
                    continue
                
                try:
                    print(Fore.YELLOW + "\t\tStoring Data...")
                    data["specs"].append(spectrogram)  # 直接存储numpy数组，不要转成list
                    data["labels"].append(i-1)
                except Exception as e:
                    print(Fore.RED + f"\t\tStoring FAILED: {str(e)}")
                    continue
                
                print(Fore.GREEN + "\t\t\tPreprocessing Complete!")
                print(Fore.WHITE + f"\t\t\tFile: {file}\n")
                time.sleep(0.1)
    
    np.savez_compressed(npz_path, x_train=data["specs"], y_train=data["labels"], mapping=data["mapping"])

2. 修复数据加载与训练代码 #

调整加载逻辑，确保X是统一形状的三维数组，Y是完整的标签数组：

import numpy as np
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import LSTM, Dense, Flatten

DATA_PATH = "/content/drive/MyDrive/AI/Data/Per-Gender_Rec/data.npz"
DATA = np.load(DATA_PATH, allow_pickle=True)

# 加载所有标签，转换为numpy数组
Y = np.array(DATA["y_train"])
# 加载所有频谱图，此时已统一形状，转换为三维数组(样本数, 时间帧, 频率特征)
X = np.array(DATA["x_train"])
print(f"X shape: {X.shape}, Y shape: {Y.shape}")  # 应输出类似(4528, 40, 1025)和(4528,)

# 推荐使用RNN处理时序语音数据
Network = Sequential()
# 输入形状为(时间帧数, 频率特征数)，直接用LSTM层处理序列
Network.add(LSTM(128, input_shape=(X.shape[1], X.shape[2])))
Network.add(Dense(64, activation="relu"))
Network.add(Dense(1, activation="sigmoid"))

# 若坚持使用全连接网络，取消以下注释
# Network = Sequential()
# Network.add(Flatten(input_shape=(X.shape[1], X.shape[2])))
# Network.add(Dense(512, activation="relu"))
# Network.add(Dense(256, activation="relu"))
# Network.add(Dense(64, activation="relu"))
# Network.add(Dense(1, activation="sigmoid"))

Network.compile(optimizer="adam", loss="binary_crossentropy", metrics=["accuracy"])
Network.summary()

# 开始训练
Network.fit(X, Y, batch_size=32, epochs=5)

3. 额外注意事项 #

• 可根据数据集特点调整target_length（比如0.8秒或1.2秒），但必须保证所有音频处理后长度一致
• RNN更适合处理语音这类时序数据，无需Flatten，直接输入序列特征即可
• 原代码的try-except过于宽泛，建议捕获具体异常（如librosa.util.exceptions.ParameterError、FileNotFoundError），便于精准排查问题

内容的提问来源于stack exchange，提问作者JavadMH13