MPI并行C程序偶发异常求助：含Abort trap、段错误等

我基于MPI和C语言开发了一个并行聚类程序，已固定所有初始条件确保每次执行的逻辑完全一致，但程序会偶发异常，且错误类型交替出现。

补全malloc强制类型转换前的错误 #

首次出现的错误为Abort trap: 6，完整错误日志如下：

[MBP-di-Simone:76276] *** Process received signal ***
[MBP-di-Simone:76276] Signal: Abort trap: 6 (6)
[MBP-di-Simone:76276] Signal code:  (0)
[MBP-di-Simone:76276] [ 0] 0   libsystem_platform.dylib            0x00007ff80f917c1d _sigtramp + 29
[MBP-di-Simone:76276] [ 1] 0   ???                                 0x0000000000000201 0x0 + 513
[MBP-di-Simone:76276] [ 2] 0   libsystem_c.dylib                   0x00007ff80f836ca5 abort + 123
[MBP-di-Simone:76276] [ 3] 0   libsystem_malloc.dylib              0x00007ff80f74ca37 malloc_vreport + 888
[MBP-di-Simone:76276] [ 4] 0   libsystem_malloc.dylib              0x00007ff80f761959 malloc_zone_error + 178
[MBP-di-Simone:76276] [ 5] 0   libsystem_malloc.dylib              0x00007ff80f75a12b nanov2_guard_corruption_detected + 34
[MBP-di-Simone:76276] [ 6] 0   libsystem_malloc.dylib              0x00007ff80f7596e9 nanov2_allocate_outlined + 374
[MBP-di-Simone:76276] [ 7] 0   libsystem_malloc.dylib              0x00007ff80f73f59d nanov2_malloc + 526
[MBP-di-Simone:76276] [ 8] 0   libopen-pal.40.dylib                0x000000010562f5eb opal_show_help_yyensure_buffer_stack + 128
[MBP-di-Simone:76276] [ 9] 0   libopen-pal.40.dylib                0x000000010562f8b3 opal_show_help_yy_switch_to_buffer + 14
[MBP-di-Simone:76276] [10] 0   libopen-pal.40.dylib                0x000000010562fdc5 opal_show_help_init_buffer + 22
[MBP-di-Simone:76276] [11] 0   libopen-pal.40.dylib                0x000000010562e777 opal_show_help_vstring + 386
[MBP-di-Simone:76276] [12] 0   libopen-rte.40.dylib                0x00000001057cb9e8 orte_show_help + 171
[MBP-di-Simone:76276] [13] 0   libmpi.40.dylib                     0x000000010569c9ba backend_fatal + 606
[MBP-di-Simone:76276] [14] 0   libmpi.40.dylib                     0x000000010569c73e ompi_mpi_errors_are_fatal_comm_handler + 159
[MBP-di-Simone:76276] [15] 0   libmpi.40.dylib                     0x000000010569c3df ompi_errhandler_invoke + 99
[MBP-di-Simone:76276] [16] 0   libmpi.40.dylib                     0x00000001056d5b33 MPI_Recv + 651
[MBP-di-Simone:76276] [17] 0   main                                0x00000001054dd1d6 main + 3958
[MBP-di-Simone:76276] [18] 0   dyld                                0x00007ff80f5bc310 start + 2432
[MBP-di-Simone:76276] *** End of error message ***
--------------------------------------------------------------------------
 Primary job  terminated normally, but 1 process returned
 a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
 mpirun noticed that process rank 0 with PID 0 on node MBP-di-Simone exited on signal 6 (Abort trap: 6).

补全malloc强制类型转换后的交替错误 #

补全所有缺失的malloc强制类型转换后，错误类型变为以下两种交替出现：

错误1：Abort trap: 6 #

[MBP-di-Simone:85691] *** Process received signal ***
[MBP-di-Simone:85691] Signal: Abort trap: 6 (6)
[MBP-di-Simone:85691] Signal code:  (0)
[MBP-di-Simone:85691] [ 0] 0   libsystem_platform.dylib            0x00007ff80f917c1d _sigtramp + 29
[MBP-di-Simone:85691] [ 1] 0   ???                                 0x00007ff7b74bced0 0x0 + 140701908848336
[MBP-di-Simone:85691] [ 2] 0   libsystem_c.dylib                   0x00007ff80f836ca5 abort + 123
[MBP-di-Simone:85691] [ 3] 0   libsystem_malloc.dylib              0x00007ff80f74ca37 malloc_vreport + 888
[MBP-di-Simone:85691] [ 4] 0   libsystem_malloc.dylib              0x00007ff80f761959 malloc_zone_error + 178
[MBP-di-Simone:85691] [ 5] 0   libsystem_malloc.dylib              0x00007ff80f745aa6 small_free_list_remove_ptr_no_clear + 1000
[MBP-di-Simone:85691] [ 6] 0   libsystem_malloc.dylib              0x00007ff80f742f7c free_small + 619
[MBP-di-Simone:85691] [ 7] 0   libmpi.40.dylib                     0x0000000108c89115 mca_coll_base_comm_unselect + 11654
[MBP-di-Simone:85691] [ 8] 0   libmpi.40.dylib                     0x0000000108bf81aa ompi_comm_destruct + 32
[MBP-di-Simone:85691] [ 9] 0   libmpi.40.dylib                     0x0000000108bf9120 ompi_comm_finalize + 232
[MBP-di-Simone:85691] [10] 0   libmpi.40.dylib                     0x0000000108c13620 ompi_mpi_finalize + 923
[MBP-di-Simone:85691] [11] 0   main                                0x0000000108a4671b main + 5163
[MBP-di-Simone:85691] [12] 0   dyld                                0x00007ff80f5bc310 start + 2432
[MBP-di-Simone:85691] *** End of error message ***
--------------------------------------------------------------------------
 Primary job  terminated normally, but 1 process returned
 a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
 A system call failed during shared memory initialization that should
 not have.  It is likely that your MPI job will now either abort or
 experience performance degradation.

  Local host:  MBP-di-Simone.homenet.telecomitalia.it
  System call: unlink(2)         /var/folders/d6/st4lmq9x55bd2cw12j85xz8m0000gn/T//ompi.MBP-di-Simone.501/pid.85690/1/vader_segment.MBP-di-Simone.501.ae5e0001.1
  Error:       No such file or directory (errno 2)
--------------------------------------------------------------------------
[MBP-di-Simone.homenet.telecomitalia.it:85690] PMIX ERROR: UNREACHABLE in file server/pmix_server.c at line 2198
--------------------------------------------------------------------------
 mpirun noticed that process rank 0 with PID 0 on node MBP-di-Simone exited on signal 6 (Abort trap: 6).

错误2：Segmentation fault: 11 #

[MBP-di-Simone:86533] *** Process received signal ***
[MBP-di-Simone:86533] Signal: Segmentation fault: 11 (11)
[MBP-di-Simone:86533] Signal code:  (0)
[MBP-di-Simone:86533] Failing at address: 0x0
[MBP-di-Simone:86533] [ 0] 0   libsystem_platform.dylib            0x00007ff80f917c1d _sigtramp + 29
[MBP-di-Simone:86533] [ 1] 0   ???                                 0x0000000000000000 0x0 + 0
[MBP-di-Simone:86533] [ 2] 0   libmpi.40.dylib                     0x000000010f02d1aa ompi_comm_destruct + 32
[MBP-di-Simone:86533] [ 3] 0   libmpi.40.dylib                     0x000000010f02e069 ompi_comm_finalize + 49
[MBP-di-Simone:86533] [ 4] 0   libmpi.40.dylib                     0x000000010f048620 ompi_mpi_finalize + 923
[MBP-di-Simone:86533] [ 5] 0   main                                0x000000010ee7b71b main + 5163
[MBP-di-Simone:86533] [ 6] 0   dyld                                0x00007ff80f5bc310 start + 2432
[MBP-di-Simone:86533] *** End of error message ***
--------------------------------------------------------------------------
 Primary job  terminated normally, but 1 process returned
 a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
 A system call failed during shared memory initialization that should
 not have.  It is likely that your MPI job will now either abort or
 experience performance degradation.

  Local host:  MBP-di-Simone.homenet.telecomitalia.it
  System call: unlink(2) /var/folders/d6/st4lmq9x55bd2cw12j85xz8m0000gn/T//ompi.MBP-di-Simone.501/pid.86532/1/vader_segment.MBP-di-Simone.501.b2e00001.1
  Error:       No such file or directory (errno 2)
--------------------------------------------------------------------------
[MBP-di-Simone.homenet.telecomitalia.it:86532] PMIX ERROR: UNREACHABLE in file server/pmix_server.c at line 2198
--------------------------------------------------------------------------
 mpirun noticed that process rank 0 with PID 0 on node MBP-di-Simone exited on signal 11 (Segmentation fault: 11).

其他偶发现象 #

极少数情况下，程序仅输出以下内容后就挂起，不再继续执行计算：

[MBP-di-Simone:86488] *** Process received signal ***

疑问点 #

• 上述错误的具体含义是什么？
• 为何初始条件完全一致，错误却呈偶发性？
• 异常集中在MPI_Finalize()调用阶段的原因是什么？

内容的提问来源于stack exchange，提问作者simone russo