微调类FUNSD数据集LayoutLMv2触发index out of range错误排查

问题描述 #

使用Hugging Face Transformers库AutoModelForTokenClassification接口，在完成预处理、归一化的类FUNSD结构自定义数据集上微调microsoft/layoutlmv2-base-uncased模型，训练数轮迭代后触发如下报错：

Traceback (most recent call last):
  File "layoutlmV2/train.py", line 137, in <module>
    trainer.train()
  File "..../lib/python3.8/site-packages/transformers/trainer.py", line 1409, in train
    return inner_training_loop(
  File "..../lib/python3.8/site-packages/transformers/trainer.py", line 1651, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs)
  File "..../lib/python3.8/site-packages/transformers/trainer.py", line 2345, in training_step
    loss = self.compute_loss(model, inputs)
  File "..../lib/python3.8/site-packages/transformers/trainer.py", line 2377, in compute_loss
    outputs = model(**inputs)
  File "..../lib/python3.8/site-packages/torch/nn/modules/module.py", line 1131, in _call_impl
    return forward_call(*input, **kwargs)
  File "..../lib/python3.8/site-packages/transformers/models/layoutlmv2/modeling_layoutlmv2.py", line 1228, in forward
    outputs = self.layoutlmv2(
  File "..../lib/python3.8/site-packages/torch/nn/modules/module.py", line 1131, in _call_impl
    return forward_call(*input, **kwargs)
  File "..../lib/python3.8/site-packages/transformers/models/layoutlmv2/modeling_layoutlmv2.py", line 902, in forward
    text_layout_emb = self._calc_text_embeddings(
  File "..../lib/python3.8/site-packages/transformers/models/layoutlmv2/modeling_layoutlmv2.py", line 753, in _calc_text_embeddings
    spatial_position_embeddings = self.embeddings._calc_spatial_position_embeddings(bbox)
  File "..../lib/python3.8/site-packages/transformers/models/layoutlmv2/modeling_layoutlmv2.py", line 93, in _calc_spatial_position_embeddings
    h_position_embeddings = self.h_position_embeddings(bbox[:, :, 3] - bbox[:, :, 1])
  File "..../lib/python3.8/site-packages/torch/nn/modules/module.py", line 1131, in _call_impl
    return forward_call(*input, **kwargs)
  File "..../lib/python3.8/site-packages/torch/nn/modules/sparse.py", line 158, in forward
    return F.embedding(
  File "..../lib/python3.8/site-packages/torch/nn/functional.py", line 2203, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
IndexError: index out of range in self

经排查词表大小、bbox坐标、张量维度、类别数量等配置后确认，触发错误的输入张量存在负值，此前迭代成功的输入均为无符号整数。负值来自modeling_layoutlmv2.py中_calc_spatial_position_embeddings(self, bbox)方法，对应计算逻辑为：

h_position_embeddings = self.h_position_embeddings(bbox[:, :, 3] - bbox[:, :, 1])

触发报错的输入张量示例：

tensor([[ 0, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
         11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
         11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
         11, 11, 11, 11, 11, 11, 11, 11, 11, 12, 12, 12, 12, 12, 11, 11, 11, 11,
         11, 11, 11, 11, 11, 11, 11, 11, 11,  9,  9,  9,  9,  9,  9,  9,  9,  9,
          9,  9,  9,  9,  9,  9,  9, 10, 10, 12, 12, 12, 12, 12, 12, 12, 12, 12,
         12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12,
         12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12,
         12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12,
         12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12,
         12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 11, 11, 11, 11,
         11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 12, 12, 12, 12, 12, 12, 12, 12,
         12, 12, 12, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 12, 12,
         12, 12, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
         11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
         11, 11, 11, 11, 11, 11, 11, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10,
         10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 12, 12, 12, 12,
         12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12,
         12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12,
         12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12,
         12, 12, 12, 12, 12, 12, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
         11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
         11, 11, 12, 12, 12, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
         11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 12, 12, 12, 12, 12, 12, 12,
         12, 12, 12, 12, 12, 12, 12, 12,  8,  8,  8,  8,  8,  8,  8,  8,  8,  8,
          8,  8,  8,  8,  8,  8,  8,  8,  8,  8,  8,  8,  8,  8,  8,  8,  8,  8,
          8,  5,  5,  5,  5,  5,  5, -6, -6, -6, -6, -6, -6,  1,  1,  1,  1,  1,
          5,  5,  5,  5,  5,  5,  7,  5,  7,  7,  0,  0,  0,  0,  0,  0,  0,  0,
          0,  0,  0,  0,  0,  0,  0,  0]])

待解决问题：

• 哪些原因会导致上述计算返回的输入值出现负数？
• 可采取哪些措施避免该错误发生？

负值产生原因 #

• bbox坐标格式不符合模型要求：LayoutLMv2要求输入bbox格式为[x0, y0, x1, y1]，其中(x0,y0)为边界框左上角坐标，(x1,y1)为右下角坐标，必须满足x1 >= x0、y1 >= y0。代码中bbox[:, :, 3] - bbox[:, :, 1]计算的是边界框高度，一旦y轴坐标顺序写反、标注时上下坐标错位，就会出现y1 < y0，计算结果为负。
• 预处理逻辑错误：坐标归一化时如果图片尺寸读取错误、缩放比例计算错误、裁剪/偏移逻辑有误，会导致坐标值溢出0-1000的合法范围，甚至出现坐标顺序颠倒。
• 数据增强引入异常：训练时使用随机翻转、裁剪、旋转等空间类数据增强后，没有同步更新bbox坐标，或更新后未做校验，导致坐标顺序错乱。
• 原始数据存在脏样本：如果bbox来自OCR引擎输出或人工标注，部分低质量样本可能存在坐标顺序错位、坐标值异常的问题，预处理阶段未过滤就会进入训练流程。

修复方案 #

• 强制增加bbox合法性校验逻辑，在数据预处理、collate阶段对所有bbox做修正，参考代码如下：

  # bbox形状为[batch_size, seq_len, 4]，对应格式[x0,y0,x1,y1]
  # 1. 将所有坐标裁剪到LayoutLMv2要求的0-1000范围内
  bbox = bbox.clamp(min=0, max=1000)
  # 2. 强制修正坐标顺序，保证右下角坐标始终大于等于左上角坐标
  x0 = torch.min(bbox[:, :, 0], bbox[:, :, 2])
  y0 = torch.min(bbox[:, :, 1], bbox[:, :, 3])
  x1 = torch.max(bbox[:, :, 0], bbox[:, :, 2])
  y1 = torch.max(bbox[:, :, 1], bbox[:, :, 3])
  bbox = torch.stack([x0, y0, x1, y1], dim=-1)
  # 3. 特殊token<[BOS_never_used_51bce0c785ca2f68081bfa7d91973934]>、[SEP]、[PAD]对应的bbox统一设置为[0,0,0,0]
  

• 排查预处理全流程：确认原始图片尺寸读取正确，坐标归一化时按比例缩放到0-1000区间，没有出现坐标翻转、偏移错误。
• 校准数据增强逻辑：所有空间类增强操作完成后，必须同步更新对应bbox坐标，更新后重新做合法性校验，剔除坐标完全异常的样本。
• 清洗原始数据集：过滤标注错误、OCR输出异常的bbox（宽/高为负、宽/高超出图片尺寸范围）。
• 临时兜底方案：如果需要快速恢复训练，可以在_calc_spatial_position_embeddings方法计算宽高后增加clamp(min=0)处理，但该方案不能修正错误的位置信息，建议优先从数据源解决问题。

内容的提问来源于stack exchange，提问作者Naourass Derouichi