读取TFRecord文件时出现Incorrect header check错误的排查求助

最近我在把训练好的模型用到测试文件上评估时遇到了一个问题，错误总是出在batch_raw = raw_iter.get_next()这一行。

我的代码片段如下：

try:
    while N_b<max_batches:
        batch_raw = raw_iter.get_next()
        batch_test = test_iter.get_next()
        if N_b%20==0:
            tqdm.write('Processing batch '+str(N_b))
                    
            N_b = self.predict_batch(model, batch_test, batch_raw, csvPath, N_b, vars_to_save, plotter)
except tf.errors.OutOfRangeError:
    pass

触发的报错信息是：

Traceback (most recent call last):
  File "/home/plebani/anaconda3/envs/tf_gpu/lib/python3.9/site-packages/tensorflow/python/eager/context.py", line 2113, in execution_mode
    yield
  File "/home/plebani/anaconda3/envs/tf_gpu/lib/python3.9/site-packages/tensorflow/python/data/ops/iterator_ops.py", line 730, in _next_internal
    ret = gen_dataset_ops.iterator_get_next(
  File "/home/plebani/anaconda3/envs/tf_gpu/lib/python3.9/site-packages/tensorflow/python/ops/gen_dataset_ops.py", line 2579, in iterator_get_next
    _ops.raise_from_not_ok_status(e, name)
  File "/home/plebani/anaconda3/envs/tf_gpu/lib/python3.9/site-packages/tensorflow/python/framework/ops.py", line 6862, in raise_from_not_ok_status
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.DataLossError: inflate() failed with error -3: incorrect header check [Op:IteratorGetNext]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/plebani/met-ml/met-ml/share/example_METNetRunner.py", line 391, in <module>
    run('METNetSig', loss_fn=loss_fn, weighting=weighting, rotation=True, class_sig=False, metnetsig=False)
  File "/home/plebani/met-ml/met-ml/share/example_METNetRunner.py", line 355, in run
    x.get_predictions(x.outDir + 'model', plot=True, sig_plots=True, saveSUSY2CSV=True)
  File "/home/plebani/met-ml/met-ml/src/METNetRunner.py", line 517, in get_predictions
    self.predict_topology(model,top_name, test_dat, raw_dat, plot=plot, sig_plots=sig_plots, saveSUSY2CSV=saveSUSY2CSV, vars_to_save=self.vars_to_save, max_batches=100)
  File "/home/plebani/met-ml/met-ml/src/METNetRunner.py", line 358, in predict_topology
    batch_raw = raw_iter.get_next()
  File "/home/plebani/anaconda3/envs/tf_gpu/lib/python3.9/site-packages/tensorflow/python/data/ops/iterator_ops.py", line 800, in get_next
    return self._next_internal()
  File "/home/plebani/anaconda3/envs/tf_gpu/lib/python3.9/site-packages/tensorflow/python/data/ops/iterator_ops.py", line 739, in _next_internal
    return structure.from_compatible_tensor_list(self._element_spec, ret)
  File "/home/plebani/anaconda3/envs/tf_gpu/lib/python3.9/contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/plebani/anaconda3/envs/tf_gpu/lib/python3.9/site-packages/tensorflow/python/eager/context.py", line 2116, in execution_mode
    executor_new.wait()
  File "/home/plebani/anaconda3/envs/tf_gpu/lib/python3.9/site-packages/tensorflow/python/eager/executor.py", line 69, in wait
    pywrap_tfe.TFE_ExecutorWaitForAllPendingNodes(self._handle)
tensorflow.python.framework.errors_impl.DataLossError: inflate() failed with error -3: incorrect header check

我本来以为是文件损坏，但做了几个测试，结果显示并不是：

• 测试1：检查文件内容
  用下面的代码片段读取文件，能正常输出数据集里的内容：

  filename = 'Zee_0__test1.tfrecord'
  dataset = tf.data.TFRecordDataset(filename, compression_type='GZIP') # the files are compressed
  for record in dataset.take(1):
     print(record)
  

• 测试2：重复运行代码
  有时候代码能正常跑完全程，不会报错，但大多数时候还是会触发这个错误。

• 测试3：调整批处理数量
  如果只运行1个批次（处理10k条数据），不会报错；但设置max_batches > 1（总共有80k条数据，每次处理10k）就会报错，而且每次报错前处理的批次数还不一样。

我用的是conda环境，TensorFlow版本是2.4.1。试过用conda和Python venv重新创建环境，也试过把TensorFlow更到最新版，问题依然存在。我还重新生成了训练文件，结果还是一样。

现在实在没思路了，有没有大佬能帮忙看看问题出在哪？

备注：内容来源于stack exchange，提问作者Pleba