TensorFlow训练无报错但无限卡住无进度输出是什么原因？

• 完成TensorFlow环境搭建、GPU有效性验证及全量配置后，执行如下训练启动命令：

py Tensorflow\models\research\object_detection\model_main_tf2.py --model_dir=Tensorflow\workspace\models\my_ssd_mobnet --pipeline_config_path=Tensorflow\workspace\models\my_ssd_mobnet\pipeline.config --num_train_steps=100

• 异常现象：等待超过1小时无训练进度日志输出，Python进程稳定占用25%-26%CPU资源，即使将训练步数下调到100步也无进度反馈。
• 日志情况：仅存在弃用类警告与正常INFO级别运行日志，无崩溃类报错，精简日志如下：

2021-07-11 02:25:42.869766: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cudart64_110.dll
py Tensorflow\models\research\object_detection\model_main_tf2.py --model_dir=Tensorflow\workspace\models\my_ssd_mobnet --pipeline_config_path=Tensorflow\workspace\models\my_ssd_mobnet\pipeline.config --num_train_steps=100
2021-07-11 02:25:44.989884: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cudart64_110.dll
2021-07-11 02:25:47.588384: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library nvcuda.dll
2021-07-11 02:25:47.605286: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: NVIDIA GeForce RTX 2080 SUPER computeCapability: 7.5
coreClock: 1.845GHz coreCount: 48 deviceMemorySize: 8.00GiB deviceMemoryBandwidth: 462.00GiB/s
2021-07-11 02:25:47.605366: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cudart64_110.dll
2021-07-11 02:25:47.610303: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cublas64_11.dll
2021-07-11 02:25:47.610390: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cublasLt64_11.dll
2021-07-11 02:25:47.613585: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cufft64_10.dll
2021-07-11 02:25:47.614873: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library curand64_10.dll
2021-07-11 02:25:47.621607: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cusolver64_11.dll
2021-07-11 02:25:47.623967: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cusparse64_11.dll
2021-07-11 02:25:47.624496: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cudnn64_8.dll
2021-07-11 02:25:47.626311: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0
2021-07-11 02:25:47.626728: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-07-11 02:25:47.627707: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: NVIDIA GeForce RTX 2080 SUPER computeCapability: 7.5
coreClock: 1.845GHz coreCount: 48 deviceMemorySize: 8.00GiB deviceMemoryBandwidth: 462.00GiB/s
2021-07-11 02:25:47.627810: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0
2021-07-11 02:25:48.067610: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1258] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-07-11 02:25:48.067778: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1264]      0
2021-07-11 02:25:48.068662: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1277] 0:   N
2021-07-11 02:25:48.069323: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1418] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5957 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 SUPER, pci bus id: 0000:01:00.0, compute capability: 7.5)
WARNING:tensorflow:Collective ops is not configured at program startup. Some performance features may not be enabled.
W0711 02:25:48.071784 10384 mirrored_strategy.py:379] Collective ops is not configured at program startup. Some performance features may not be enabled.
INFO:tensorflow:Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:GPU:0',)
I0711 02:25:48.225363 10384 mirrored_strategy.py:369] Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:GPU:0',)
INFO:tensorflow:Maybe overwriting train_steps: 100
I0711 02:25:48.229352 10384 config_util.py:552] Maybe overwriting train_steps: 100
INFO:tensorflow:Maybe overwriting use_bfloat16: False
I0711 02:25:48.230349 10384 config_util.py:552] Maybe overwriting use_bfloat16: False
INFO:tensorflow:Reading unweighted datasets: ['Tensorflow\\workspace\\annotations\\train.record']
I0711 02:25:48.308165 10384 dataset_builder.py:163] Reading unweighted datasets: ['Tensorflow\\workspace\\annotations\\train.record']
INFO:tensorflow:Reading record datasets for input file: ['Tensorflow\\workspace\\annotations\\train.record']
I0711 02:25:48.309138 10384 dataset_builder.py:80] Reading record datasets for input file: ['Tensorflow\\workspace\\annotations\\train.record']
INFO:tensorflow:Number of filenames to read: 1
I0711 02:25:48.311132 10384 dataset_builder.py:81] Number of filenames to read: 1
WARNING:tensorflow:num_readers has been reduced to 1 to match input file shards.
W0711 02:25:48.311132 10384 dataset_builder.py:87] num_readers has been reduced to 1 to match input file shards.
tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:176] None of the MLIR Optimization Passes are enabled (registered 2)

• 优先验证训练数据集有效性：日志显示程序卡在读取train.record后的解析阶段，先单独编写脚本读取生成的train.record文件，确认可以正常解析出图片与标注信息，排除文件损坏、生成时类ID/类名不匹配的问题。
• 核对pipeline.config配置：确认预训练权重文件路径、标签映射文件路径配置正确无拼写错误，batch size调整到2再测试（8G显存运行ssd mobilenet默认batch size可能超出显存阈值，导致隐性死锁）。
• 追加日志输出参数：启动命令后添加--alsologtostderr参数，强制将所有运行日志输出到控制台，可查看数据集解析阶段的隐藏报错信息。
• 验证依赖版本匹配：日志显示加载的是CUDA11.0对应的动态库，确认当前安装的TensorFlow版本、cuDNN版本与CUDA11.0符合官方版本兼容要求，版本不匹配会导致训练初始化阶段无报错卡死。
• 测试官方示例数据集：替换自己生成的train.record为官方示例数据集启动训练，如果可以正常运行则完全定位为自定义数据集生成环节的问题。

内容的提问来源于stack exchange，提问作者J.Todd