TensorFlow并行训练多模型时检查点文件丢失问题

需要训练约100个TensorFlow模型，每个模型训练时通过keras-tuner搜索最优超参数。为节省时间，计划给每个CPU核心分配一个模型并行训练，但并行过程中各模型的检查点文件会互相覆盖，调用get_best_models()时出现如下报错：

1720 tuner.search(X_train, y_train, epochs=config.DIRECTOR_EPOCHS, 
   1721              validation_split=0.15, callbacks=[callbacks],
   1722              verbose=0)
   1724 # Let's extract the director model from the best fitted model
-> 1725 best_model = tuner.get_best_models()[0]
   1726 director = keras.Model(inputs=best_model.get_layer('_').input,
   1727                        outputs=[
   1728                            best_model.get_layer('__').output,
   1729                            best_model.get_layer('___').output
   1730                        ],
   1731                        name=f'director_h{h}')
   1732 self.directors[f'director_h{h}'] = director

File ~\PycharmProjects\Orchestra\omnienergy-orchestra\venv\lib\site-packages\keras_tuner\engine\tuner.py:366, in Tuner.get_best_models(self, num_models)
    348 """Returns the best model(s), as determined by the tuner's objective.
    349 
    350 The models are loaded with the weights corresponding to
   (...)
    363    List of trained model instances sorted from the best to the worst.
    364 """
    365 # Method only exists in this class for the docstring override.
-> 366 return super().get_best_models(num_models)

File ~\PycharmProjects\Orchestra\omnienergy-orchestra\venv\lib\site-packages\keras_tuner\engine\base_tuner.py:364, in BaseTuner.get_best_models(self, num_models)
    349 """Returns the best model(s), as determined by the objective.
    350 
    351 This method is for querying the models trained during the search.
   (...)
    361    List of trained models sorted from the best to the worst.
    362 """
    363 best_trials = self.oracle.get_best_trials(num_models)
-> 364 models = [self.load_model(trial) for trial in best_trials]
    365 return models

File ~\PycharmProjects\Orchestra\omnienergy-orchestra\venv\lib\site-packages\keras_tuner\engine\base_tuner.py:364, in <listcomp>(.0)
    349 """Returns the best model(s), as determined by the objective.
    350 
    351 This method is for querying the models trained during the search.
   (...)
    361    List of trained models sorted from the best to the worst.
    362 """
    363 best_trials = self.oracle.get_best_trials(num_models)
-> 364 models = [self.load_model(trial) for trial in best_trials]
    365 return models

File ~\PycharmProjects\Orchestra\omnienergy-orchestra\venv\lib\site-packages\keras_tuner\engine\tuner.py:297, in Tuner.load_model(self, trial)
    294 # Reload best checkpoint.
    295 # Only load weights to avoid loading `custom_objects`.
    296 with maybe_distribute(self.distribution_strategy):
-> 297     model.load_weights(self._get_checkpoint_fname(trial.trial_id))
    298 return model

File ~\PycharmProjects\Orchestra\omnienergy-orchestra\venv\lib\site-packages\keras\utils\traceback_utils.py:70, in filter_traceback.<locals>.error_handler(*args, **kwargs)
     67     filtered_tb = _process_traceback_frames(e.__traceback__)
     68     # To get the full stack trace, call:
     69     # `tf.debugging.disable_traceback_filtering()`
-> 70     raise e.with_traceback(filtered_tb) from None
     71 finally:
     72     del filtered_tb

File ~\PycharmProjects\Orchestra\omnienergy-orchestra\venv\lib\site-packages\tensorflow\python\training\py_checkpoint_reader.py:31, in error_translator(e)
     27 error_message = str(e)
     28 if 'not found in checkpoint' in error_message or (
     29     'Failed to find any '
     30     'matching files for') in error_message:
-> 31   raise errors_impl.NotFoundError(None, None, error_message)
     32 elif 'Sliced checkpoints are not supported' in error_message or (
     33     'Data type '
     34     'not '
     35     'supported') in error_message:
     36   raise errors_impl.UnimplementedError(None, None, error_message)

NotFoundError: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for tmp\untitled_project\trial_09\checkpoint

观察到一个现象：若将两个模型放在不同硬盘训练则不会报错。尝试过重命名检查点文件，但未找到可行方案。

问题根源是并行训练时多个Keras Tuner实例共用了默认临时目录，导致检查点文件被互相覆盖。解决核心是给每个模型分配独立的工作路径：

• 给每个Tuner实例指定唯一目录
  初始化Tuner时，通过directory参数为每个模型创建单独的文件夹，比如用模型索引区分：

  # 假设h是当前模型的索引（0~99）
  tuner = RandomSearch(
      build_model,
      objective="val_accuracy",
      max_trials=5,
      directory=f"./tuner_workspaces/model_{h}",  # 每个模型专属目录
      project_name="hyper_param_search"
  )
  

  这样每个模型的调优日志、检查点都会存储在独立路径下，彻底避免覆盖冲突。

• 自定义检查点回调的保存路径
  如果训练中使用了ModelCheckpoint回调，同样要确保每个模型的检查点路径唯一：

  callbacks = [
      tf.keras.callbacks.ModelCheckpoint(
          filepath=f"./model_checkpoints/model_{h}/best_weights.h5",
          save_best_only=True
      )
  ]
  

你提到不同硬盘训练无报错，本质是不同硬盘的临时目录路径不同，天然避免了冲突，和上面的思路一致——只要路径唯一就不会出现覆盖问题。

内容的提问来源于stack exchange，提问作者Ferdinando Randisi