Python执行Keras fit_generator时崩溃的问题排查求助

Hey there, let's break down why your kernel is crashing when running classifier.fit_generator() and how to fix it. First, let's highlight the key clues from your crash report that point to the root causes:

Process: python3.7 [22322]
OS Version: Mac OS X 10.14.6 (18G2022)
Crashed Thread: 0 ZMQbg/1
Exception Type: EXC_CRASH (SIGABRT)
Application Specific Information: abort() called
2 libiomp5.dylib 0x0000000142ca37e3 __kmp_abort_process + 35

The crash points to three main issues: OpenMP library conflicts (from libiomp5.dylib), Spyder's ZMQ background thread interference, and potential memory overload from your training parameters. Here are actionable fixes, ordered by likelihood of resolving your problem:

1. Fix OpenMP Library Conflicts (Most Likely Culprit) #

Anaconda's libiomp5 library often clashes with TensorFlow's threading on Mac. Try these quick fixes:

• Add an environment variable to your script: Put this at the very top of your code before importing Keras/TensorFlow to ignore duplicate library loads:

  import os
  os.environ['KMP_DUPLICATE_LIB_OK'] = 'True'
  

• Replace MKL with nomkl: If the above doesn't work, install non-MKL versions of numerical libraries to eliminate the conflict entirely:

  conda install nomkl
  

2. Reduce Training Load to Rule Out Memory Issues #

Your steps_per_epoch=8000 and validation_steps=2000 might be overwhelming your system's RAM/GPU memory (especially if your batch size is large):

• Test with drastically reduced values first: set steps_per_epoch=100 and validation_steps=20. If the run completes without crashing, you know memory overload was the issue.
• Shrink your batch_size (check your training/test set generator settings) to free up memory for larger steps later.

3. Fix Spyder ZMQ Thread Interference #

The crash occurred on the ZMQbg/1 thread, which handles Spyder's background communication. Try these tweaks:

• Change Spyder's graphics backend: Go to Preferences > IPython Console > Graphics and set the backend to Inline or Qt5 (avoid Automatic to reduce ZMQ thread activity during training).
• Run the script outside Spyder: Open a terminal, activate your Anaconda environment, and run the script directly with python your_script.py. This bypasses Spyder's ZMQ layer entirely, which often fixes thread conflicts with TensorFlow.

4. Verify TF/Keras Version Compatibility #

Python 3.7 on Mac (circa 2019) requires specific TensorFlow/Keras versions to avoid bugs:

• For TensorFlow 1.x: Install version 1.15 (the last stable 1.x release compatible with Python 3.7):

  conda install tensorflow=1.15
  

• For TensorFlow 2.x: Stick to version 2.0-2.3 (newer versions may not support Python 3.7 on Mac 10.14).

内容的提问来源于stack exchange，提问作者F Silv