提取Transformer模型隐藏状态时遇'str'无'size'属性错误求助

尝试提取Transformer模型的隐藏状态，编写代码如下：

from transformers import AutoModel
import torch
from transformers import AutoTokenizer

model_ckpt = "distilbert-base-uncased"
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
tokenizer = AutoTokenizer.from_pretrained(model_ckpt)
model = AutoModel.from_pretrained(model_ckpt).to(device)

from datasets import load_dataset

emotions = load_dataset("emotion", ignore_verifications=True)

# tokenize data
def tokenize(batch):
    return tokenizer(batch["text"], padding=True, truncation=True)

emotions_encoded = emotions.map(tokenize, batched=True, batch_size=None)

def extract_hidden_states(batch):
    inputs = {k:v.to(device) for k,v in batch.items()
             if k in tokenizer.model_input_names}
    with torch.no_grad():
        last_hidden_state = model(*inputs).last_hidden_state
    return{"hidden_state": last_hidden_state[:,0].cpu().numpy()}

# convert input_ids and attention_mask columns to "torch" format 
emotions_encoded.set_format("torch", columns=["input_ids", "attention_mask", "label"])

# extract hidden states
emotions_hidden = emotions_encoded.map(extract_hidden_states, batched=True)

运行最后一行时出现错误：'str' object has no attribute 'size'，完整报错关键栈信息如下：

Cell In[7], line 6, in extract_hidden_states(batch)
      5 with torch.no_grad():
----> 6     last_hidden_state = model(*inputs).last_hidden_state

...

File ~\Anaconda3\envs\ml\lib\site-packages\transformers\models\distilbert\modeling_distilbert.py:593, in DistilBertModel.forward(self, input_ids, attention_mask, head_mask, inputs_embeds, output_attentions, output_hidden_states, return_dict)
    592     self.warn_if_padding_and_no_attention_mask(input_ids, attention_mask)
--> 593     input_shape = input_ids.size()

AttributeError: 'str' object has no attribute 'size'

报错核心是模型调用方式错误：

• inputs是包含input_ids、attention_mask的字典，使用model(*inputs)时，会把字典的**键（字符串类型）**作为位置参数传入模型，但模型期望接收的是张量类型的参数，因此触发类型错误。

只需修改模型调用时的解包方式，用双星号**将字典作为关键字参数解包，修改后的extract_hidden_states函数如下：

def extract_hidden_states(batch):
    inputs = {k:v.to(device) for k,v in batch.items()
             if k in tokenizer.model_input_names}
    with torch.no_grad():
        # 用**解包字典作为关键字参数传入模型
        last_hidden_state = model(**inputs).last_hidden_state
    return{"hidden_state": last_hidden_state[:,0].cpu().numpy()}

替换原函数后重新运行代码，即可正常提取Transformer的隐藏状态。

内容的提问来源于stack exchange，提问作者coolhand