Keras Tuner分布式超参数调优问题求助及Kubernetes部署咨询

相关文档极少，求助有Keras Tuner分布式超参数调优经验的人士提供示例。

已完成工作 #

• 非分布式场景下，基于HyperModel的Keras贝叶斯优化运行成功
• 非分布式场景下，基于HyperModel的Keras随机搜索优化运行成功
• 尝试在BinderHub实例A运行chief脚本、实例B运行worker脚本，但两者均停滞无输出；中断后发现chief处于休眠状态，报错信息如下：

userXXX@machineYYY:~$ ./tune.sh 
2022-10-05 14:33:42.266134: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 36507 MB memory:  -> device: 0, name: NVIDIA A40, pci bus id: 0000:27:00.0, compute capability: 8.6
WARNING:tensorflow:From tuning.py:53: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead.

From tuning.py:53: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead.

^CTraceback (most recent call last):
  File "tuning.py", line 443, in <module>
    tuner = keras_tuner.BayesianOptimization(
  File "/home/jovyan/.local/lib/python3.8/site-packages/keras_tuner/tuners/bayesian.py", line 460, in __init__
    super(
  File "/home/jovyan/.local/lib/python3.8/site-packages/keras_tuner/engine/tuner.py", line 110, in __init__
    super(Tuner, self).__init__(
  File "/home/jovyan/.local/lib/python3.8/site-packages/keras_tuner/engine/base_tuner.py", line 89, in __init__
    oracle_chief.start_server(self.oracle)
  File "/home/jovyan/.local/lib/python3.8/site-packages/keras_tuner/distribute/oracle_chief.py", line 81, in start_server
    time.sleep(30)
KeyboardInterrupt

使用的脚本 #

BinderHub实例A（chief节点）的tune.sh脚本 #

export KERASTUNER_TUNER_ID="chief"
export KERASTUNER_ORACLE_IP="10.42.209.12" # 当前主机IP
export KERASTUNER_ORACLE_PORT="8000"

python tuning.py

BinderHub实例B（worker节点）的tune.sh脚本 #

export KERASTUNER_TUNER_ID="tuner0"
export KERASTUNER_ORACLE_IP="10.42.209.12" # chief主机IP
export KERASTUNER_ORACLE_PORT="8000"

python tuning.py

额外咨询 #

该分布式超参数调优是否可在Kubernetes上运行？这似乎是理想工具，但找不到相关对接文档。

内容的提问来源于stack exchange，提问作者Prunus Cerasus