Wav2Vec2推理报错：Kernel size大于输入尺寸，如何解决？

使用针对波兰语微调的预训练Wav2Vec2模型进行语音识别，运行环境为Jupyter Notebook，处理音频时出现输入尺寸与卷积核不匹配的错误。

读取音频代码 #

from matplotlib.pyplot import figure, plot

from pathlib import Path
from tqdm import tqdm

import torch
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
from pyctcdecode import build_ctcdecoder
from wavio import read
import jiwer
import arpa

processor=Wav2Vec2Processor.from_pretrained('facebook/wav2vec2-base-10k-voxpopuli-ft-pl')
model=Wav2Vec2ForCTC.from_pretrained('facebook/wav2vec2-base-10k-voxpopuli-ft-pl')

files={}
for f in Path('nagrania').glob('*.wav'):
  data=read(str(f))
  files[f.stem]=data.data.squeeze().astype('float32')

Fs=data.rate
for name,d in files.items():
  print(f'{name}: {d.size/Fs:0.2f}s')

音频转文本代码 #

trans={}
for name,data in tqdm(files.items()):
  feats=processor(data,sampling_rate=16000,return_tensors='pt',padding=True)
  in_val = feats.input_values.reshape(feats.input_values.shape[1], 2)
  print(in_val.shape)
  out=model(input_values=in_val)
  predicted_ids=torch.argmax(out.logits,dim=-1)
  sent=processor.batch_decode(predicted_ids)[0]
  trans[name]=sent

运行后in_val的形状为：torch.Size([230400, 2])

报错信息 #

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
~\AppData\Local\Temp\ipykernel_15900\3914553670.py in <module>
      4   in_val = feats.input_values.reshape(feats.input_values.shape[1], 2)
      5   print(in_val.shape)
----> 6   out=model(input_values=in_val)
      7   predicted_ids=torch.argmax(out.logits,dim=-1)
      8   sent=processor.batch_decode(predicted_ids)[0]

~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
   1188         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1189                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1190             return forward_call(*input, **kwargs)
   1191         # Do not call functions when jit is used
   1192         full_backward_hooks, non_full_backward_hooks = [], []

~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\transformers\models\wav2vec2\modeling_wav2vec2.py in forward(self, input_values, attention_mask, output_attentions, output_hidden_states, return_dict, labels)
   1680             output_attentions=output_attentions,
   1681             output_hidden_states=output_hidden_states,
-> 1682             return_dict=return_dict,
   1683         )
   1684 

~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
   1188         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
...
--> 310                         self.padding, self.dilation, self.groups)
    311 
    312     def forward(self, input: Tensor) -> Tensor:

RuntimeError: Calculated padded input size per channel: (2). Kernel size: (10). Kernel size can't be greater than actual input size

解决方法 #

错误原因 #

你手动修改了processor输出的input_values形状，把原本的**[batch_size, sequence_length]单通道音频格式改成了[sequence_length, 2]**。Wav2Vec2模型要求输入形状为(batch_size, sequence_length)（单声道）或(batch_size, num_channels, sequence_length)（多声道，通道维度需在第二位）。你的修改让模型误将第二个维度识别为通道数，导致序列长度被判定为2，远小于卷积核大小10，触发报错。

正确处理方式 #

方式一：直接使用processor输出的input_values（推荐）

移除手动reshape步骤，processor已将音频处理为符合模型要求的形状：

trans={}
for name,data in tqdm(files.items()):
  # processor自动输出[1, sequence_length]形状的张量
  feats=processor(data,sampling_rate=16000,return_tensors='pt',padding=True)
  # 直接传入模型
  out=model(**feats)
  predicted_ids=torch.argmax(out.logits,dim=-1)
  sent=processor.batch_decode(predicted_ids)[0]
  trans[name]=sent

方式二：双声道音频转为单声道

若原始音频是双声道，先合并为单声道（如取均值）再传入processor：

trans={}
for name,data in tqdm(files.items()):
  # 检测双声道并转为单声道
  if len(data.shape) == 2 and data.shape[1] == 2:
      data = data.mean(axis=1)
  # 用processor处理单声道音频
  feats=processor(data,sampling_rate=16000,return_tensors='pt',padding=True)
  out=model(**feats)
  predicted_ids=torch.argmax(out.logits,dim=-1)
  sent=processor.batch_decode(predicted_ids)[0]
  trans[name]=sent

注意事项 #

• Wav2Vec2预训练使用的是单声道16kHz音频，输入必须匹配该格式。
• 不要随意修改processor输出的张量形状，除非明确了解模型的输入规范。

内容的提问来源于stack exchange，提问作者flis00