Transformers代码单独运行正常，集成Gradio时触发Device Mismatch错误
2026-6-25
nanoLLaVA集成Gradio时出现设备不匹配RuntimeError问题
我正在为@stablequan开发的nanoLLaVA制作Gradio演示，参考Moondream仓库（Apache 2.0协议）的代码结构进行移植。基于nanoLLaVA示例编写的独立脚本可正常运行并输出结果，但将相同代码集成到Gradio中时，出现如下设备不匹配的RuntimeError：
Traceback (most recent call last): File "C:\Users\Moo\AppData\Local\Programs\Python\Python311\Lib\site-packages\gradio\queueing.py", line 495, in call_prediction output = await route_utils.call_process_api( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Moo\AppData\Local\Programs\Python\Python311\Lib\site-packages\gradio\route_utils.py", line 232, in call_process_api output = await app.get_blocks().process_api( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Moo\AppData\Local\Programs\Python\Python311\Lib\site-packages\gradio\blocks.py", line 1561, in process_api result = await self.call_function( ^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Moo\AppData\Local\Programs\Python\Python311\Lib\site-packages\gradio\blocks.py", line 1179, in call_function prediction = await anyio.to_thread.run_sync( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Moo\AppData\Local\Programs\Python\Python311\Lib\site-packages\anyio\to_thread.py", line 33, in run_sync return await get_asynclib().run_sync_in_worker_thread( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Moo\AppData\Local\Programs\Python\Python311\Lib\site-packages\anyio\_backends\_asyncio.py", line 877, in run_sync_in_worker_thread return await future ^^^^^^^^^^^^ File "C:\Users\Moo\AppData\Local\Programs\Python\Python311\Lib\site-packages\anyio\_backends\_asyncio.py", line 807, in run result = context.run(func, *args) ^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Moo\AppData\Local\Programs\Python\Python311\Lib\site-packages\gradio\utils.py", line 678, in wrapper response = f(*args, **kwargs) ^^^^^^^^^^^^^^^^^^ File "C:\Users\Moo\Downloads\llm\nanollava\nanollava_gradio_demo.py", line 46, in answer_question output_ids = model.generate( ^^^^^^^^^^^^^^^ File "C:\Users\Moo\AppData\Local\Programs\Python\Python311\Lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Moo\AppData\Local\Programs\Python\Python311\Lib\site-packages\transformers\generation\utils.py", line 1575, in generate result = self._sample( ^^^^^^^^^^^^^ File "C:\Users\Moo\AppData\Local\Programs\Python\Python311\Lib\site-packages\transformers\generation\utils.py", line 2697, in _sample outputs = self( ^^^^^ File "C:\Users\Moo\AppData\Local\Programs\Python\Python311\Lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Moo\AppData\Local\Programs\Python\Python311\Lib\site-packages\torch\nn\modules\module.py", line 1520, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Moo\.cache\huggingface\modules\transformers_modules\qnguyen3\nanoLLaVA\4a1bd2e2854c6df9c4af831a408b14f7b035f4c0\modeling_llava_qwen2.py", line 2267, in forward ) = self.prepare_inputs_labels_for_multimodal( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Moo\.cache\huggingface\modules\transformers_modules\qnguyen3\nanoLLaVA\4a1bd2e2854c6df9c4af831a408b14f7b035f4c0\modeling_llava_qwen2.py", line 687, in prepare_inputs_labels_for_multimodal image_features = self.encode_images(images).to(self.device) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Moo\.cache\huggingface\modules\transformers_modules\qnguyen3\nanoLLaVA\4a1bd2e2854c6df9c4af831a408b14f7b035f4c0\modeling_llava_qwen2.py", line 661, in encode_images image_features = self.get_model().mm_projector(image_features) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Moo\AppData\Local\Programs\Python\Python311\Lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Moo\AppData\Local\Programs\Python\Python311\Lib\site-packages\torch\nn\modules\module.py", line 1520, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Moo\AppData\Local\Programs\Python\Python311\Lib\site-packages\torch\nn\modules\container.py", line 217, in forward input = module(input) ^^^^^^^^^^^^^ File "C:\Users\Moo\AppData\Local\Programs\Python\Python311\Lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Moo\AppData\Local\Programs\Python\Python311\Lib\site-packages\torch\nn\modules\module.py", line 1520, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Moo\AppData\Local\Programs\Python\Python311\Lib\site-packages\torch\nn\modules\linear.py", line 116, in forward return F.linear(input, self.weight, self.bias) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ RuntimeError: 预期所有张量都在同一设备上，但发现至少两个设备：cpu和cuda:0!（检查wrapper_CUDA_mm方法的mat2参数时）
问题原因
Gradio默认将处理逻辑放到线程池执行，可能导致输入图像张量留在CPU，而模型加载在CUDA设备上，引发张量设备不匹配。另外nanoLLaVA的图像编码流程中，未提前将图像张量转移到模型所在设备。
解决步骤
强制输入图像张量与模型同设备
在Gradio的处理函数中，调用模型生成前，明确将预处理后的图像张量转移到模型所在设备：def answer_question(image, question): # 获取模型设备 device = model.device # 图像预处理（根据你的代码调整） image_tensor = preprocess_image(image) # 转移到模型设备 image_tensor = image_tensor.to(device) # 后续生成逻辑 output_ids = model.generate(image=image_tensor, text=question) # ... 其他结果处理代码 ...修正nanoLLaVA的图像编码逻辑
打开modeling_llava_qwen2.py，修改prepare_inputs_labels_for_multimodal方法，先将图像张量转移到模型设备再编码：def prepare_inputs_labels_for_multimodal(self, input_ids, position_ids, attention_mask, past_key_values, labels, images): # 新增：先把图像转移到模型设备 if images is not None: images = images.to(self.device) # 原有的编码逻辑 image_features = self.encode_images(images) # ... 其余代码保持不变 ...禁用Gradio线程池（调试用）
如果以上方法无效，可以临时禁用Gradio的队列和线程池，确保所有操作在主线程执行：if __name__ == "__main__": demo = gr.Blocks() # ... 构建界面代码 ... demo.launch(enable_queue=False)
内容的提问来源于stack exchange，提问作者CoderCowMoo
相关产品推荐
相关产品推荐
