Apple M1设备PyTorch用MPS训练崩溃，是否为内存泄漏？

安全阿岳

2026-8-12

在M1上使用PyTorch MPS训练时出现崩溃问题

我尝试在PyTorch中利用Apple M1的GPU算力，初始代码运行正常，但一段时间后停止工作：

device='mps'
model = UNet(3, 2)
model = model.to(device)
optimizer = torch.optim.SGD(model.parameters(), lr=1e-1)
LOSSES=[]
train_loader = torch.utils.data.DataLoader(mdmloader, batch_size=10, shuffle=True)
model.train()
def train(optimizer):
    losses=[]
    for batch_idx, (rgb, mdm) in enumerate((train_loader)):
        rgb, mdm = rgb.to(device)/255., mdm.to(device)/255
        optimizer.zero_grad()
        mdm_estm = model(rgb)
        loss = ((mdm_estm-mdm)**2).mean()
        loss.backward()
        losses.append(loss.item())
        optimizer.step()

    return losses
for i in tqdm(range(2)):
    LOSSES += train(optimizer)

运行后出现如下错误：

2022-11-14 18:27:39.897 python[2025:13135] *** Terminating app due to uncaught exception 'NSInvalidArgumentException', reason: '*** -[__NSPlaceholderArray initWithObjects:count:]: attempt to insert nil object from objects[0]'
*** First throw call stack:
(
        0   CoreFoundation                      0x00007ff80e7c1e9b __exceptionPreprocess + 242
        1   libobjc.A.dylib                     0x00007ff80e523e48 objc_exception_throw + 48
        2   CoreFoundation                      0x00007ff80e87d69b _CFThrowFormattedException + 194
        3   CoreFoundation                      0x00007ff80e87b621 -[__NSPlaceholderArray initWithCapacity:].cold.1 + 0
        4   CoreFoundation                      0x00007ff80e6cb820 -[__NSPlaceholderArray initWithObjects:count:] + 154
        5   CoreFoundation                      0x00007ff80e724783 +[NSArray arrayWithObjects:count:] + 40
        6   MetalPerformanceShadersGraph        0x00007ffa2cfe0dd8 MetalPerformanceShadersGraph + 515544
        7   libtorch_cpu.dylib                  0x000000013baed8d2 ___ZN2at6native31structured_elu_backward_out_mps4implERKNS_6TensorERKN3c106ScalarES8_S8_bS4_S4__block_invoke + 1330
        8   libtorch_cpu.dylib                  0x000000013bae3a6a ___ZN2at6native3mps13MPSGraphCache17CreateCachedGraphERKNSt3__112basic_stringIcNS3_11char_traitsIcEENS3_9allocatorIcEEEEU13block_pointerFPNS1_14MPSCachedGraphEvEPv_block_invoke + 330
        9   libdispatch.dylib                   0x00007ff80e4ca317 _dispatch_client_callout + 8
        10  libdispatch.dylib                   0x00007ff80e4d77ee _dispatch_lane_barrier_sync_invoke_and_complete + 60
        11  libtorch_cpu.dylib                  0x000000013bae33d3 _ZN2at6native3mps13MPSGraphCache17CreateCachedGraphERKNSt3__112basic_stringIcNS3_11char_traitsIcEENS3_9allocatorIcEEEEU13block_pointerFPNS1_14MPSCachedGraphEvEPv + 195
        12  libtorch_cpu.dylib                  0x000000013baec9ee _ZN2at6native31structured_elu_backward_out_mps4implERKNS_6TensorERKN3c106ScalarES8_S8_bS4_S4_ + 1470
        13  libtorch_cpu.dylib                  0x00000001392c59e1 _ZN2at12_GLOBAL__N_120wrapper_elu_backwardERKNS_6TensorERKN3c106ScalarES7_S7_bS3_ + 177
        14  libtorch_cpu.dylib                  0x00000001392c5561 _ZN3c104impl28wrap_kernel_functor_unboxed_INS0_6detail24WrapFunctionIntoFunctor_INS_26CompileTimeFunctionPointerIFN2at6TensorERKS6_RKNS_6ScalarESB_SB_bS8_EXadL_ZNS5_12_GLOBAL__N_120wrapper_elu_backwardES8_SB_SB_SB_bS8_EEEES6_NS_4guts8typelist8typelistIJS8_SB_SB_SB_bS8_EEEEESC_E4callEPNS_14OperatorKernelENS_14DispatchKeySetES8_SB_SB_SB_bS8_ + 49
        15  libtorch_cpu.dylib                  0x000000013832fb2f _ZN2at4_ops12elu_backward10redispatchEN3c1014DispatchKeySetERKNS_6TensorERKNS2_6ScalarES9_S9_bS6_ + 143
        16  libtorch_cpu.dylib                  0x0000000139f8e4b1 _ZN3c104impl28wrap_kernel_functor_unboxed_INS0_6detail24WrapFunctionIntoFunctor_INS_26CompileTimeFunctionPointerIFN2at6TensorENS_14DispatchKeySetERKS6_RKNS_6ScalarESC_SC_bS9_EXadL_ZN5torch8autograd12VariableType12_GLOBAL__N_112elu_backwardES7_S9_SC_SC_SC_bS9_EEEES6_NS_4guts8typelist8typelistIJS7_S9_SC_SC_SC_bS9_EEEEESD_E4callEPNS_14OperatorKernelES7_S9_SC_SC_SC_bS9_ + 2049
        17  libtorch_cpu.dylib                  0x000000013832ef1d _ZN2at4_ops12elu_backward4callERKNS_6TensorERKN3c106ScalarES8_S8_bS4_ + 365
        18  libtorch_cpu.dylib                  0x0000000139ea1e2f _ZN5torch8autograd9generated12EluBackward15applyEONSt3__16vectorIN2at6TensorENS3_9allocatorIS6_EEEE + 255
        19  libtorch_cpu.dylib                  0x000000013ac73633 _ZN5torch8autograd4NodeclEONSt3__16vectorIN2at6TensorENS2_9allocatorIS5_EEEE + 99
        20  libtorch_cpu.dylib                  0x000000013ac6d3c9 _ZN5torch8autograd6Engine17evaluate_functionERNSt3__110shared_ptrINS0_9GraphTaskEEEPNS0_4NodeERNS0_11InputBufferERKNS3_INS0_10ReadyQueueEEE + 1977
        21  libtorch_cpu.dylib                  0x000000013ac68fa4 _ZN5torch8autograd6Engine11thread_mainERKNSt3__110shared_ptrINS0_9GraphTaskEEE + 948
        22  libtorch_cpu.dylib                  0x000000013ac68b2c _ZN5torch8autograd6Engine11thread_initEiRKNSt3__110shared_ptrINS0_10ReadyQueueEEEb + 284
        23  libtorch_python.dylib               0x000000010ea61636 _ZN5torch8autograd6python12PythonEngine11thread_initEiRKNSt3__110shared_ptrINS0_10ReadyQueueEEEb + 102
        24  libtorch_cpu.dylib                  0x000000013ac78227 _ZNSt3__1L14__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEMN5torch8autograd6EngineEFviRKNS_10shared_ptrINS8_10ReadyQueueEEEbEPS9_aSC_bEEEEEPvSJ_ + 71
        25  libsystem_pthread.dylib             0x00007ff80e6814e1 _pthread_start + 125
        26  libsystem_pthread.dylib             0x00007ff80e67cf6b thread_start + 15
)
libc++abi: terminating with uncaught exception of type NSException
zsh: abort      /Users/nmaruani/opt/anaconda3/envs/inria/bin/python 
(inria) nmaruani@MAC-04017264 Depth_prediction % /Users/nmaruani/opt/anaconda3/envs/inria/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

将device='mps'改为device='cpu'后代码可正常运行。这是否是内存泄漏问题？该如何解决？

解决方案

这大概率不是普通的内存泄漏问题，而是PyTorch早期MPS后端的算子实现bug——从错误栈可以看到，崩溃发生在ELU反向传播的MPS内核中，属于MetalPerformanceShadersGraph层面的内部错误，和数组插入nil对象相关，是框架底层的问题。

可以尝试以下几种解决办法：

升级PyTorch版本：早期MPS后端（比如1.12/1.13版本）存在大量算子兼容问题，升级到2.0及以上版本后，MPS后端的稳定性有显著提升，很多这类底层bug已经被修复。
替换模型中的ELU激活函数：如果暂时无法升级PyTorch，可以把UNet里的ELU换成其他MPS支持更好的激活函数，比如ReLU、LeakyReLU，这类算子的MPS实现更成熟，不容易触发崩溃。
限制batch size：当前batch size是10，可以尝试缩小到8或4，减少MPS内存压力，部分情况下能规避底层内存管理的bug。
手动清理MPS缓存：在每个epoch结束后添加torch.mps.empty_cache()，强制释放MPS设备上的未使用内存，避免缓存堆积引发的异常：
```
for i in tqdm(range(2)):
    LOSSES += train(optimizer)
    torch.mps.empty_cache()  # 添加这行
```

内容的提问来源于stack exchange，提问作者nissmar