Kubernetes集群中TensorFlow/Keras训练结果不一致问题求助

我用基于Dask并行化的Python代码在Kubernetes集群上训练多个神经网络，为得到确定性结果，训练前给每个Worker设置了相同的随机种子，但运行在X、Y节点的Worker生成的结果和A、B、C节点不一致——不过同一节点内的Worker结果是统一的。

示例输出：

Worker 0:  Seed 0: result:: -2.4244613650037827 <-- running on Node A
Worker 1:  Seed 0: result:: -2.4244613650037827 <-- running on Node A
Worker 2:  Seed 0: result:: -2.4259607599960265 <-- running on Node X **
Worker 3:  Seed 0: result:: -2.4259607599960265 <-- running on Node Y **
Worker 4:  Seed 0: result:: -2.4259607599960265 <-- running on Node X **
Worker 5:  Seed 0: result:: -2.4244613650037827 <-- running on Node B
Worker 6:  Seed 0: result:: -2.4244613650037827 <-- running on Node B
Worker 7:  Seed 0: result:: -2.4244613650037827 <-- running on Node A
Worker 8:  Seed 0: result:: -2.4244613650037827 <-- running on Node C
Worker 9:  Seed 0: result:: -2.4244613650037827 <-- running on Node A
Worker 10: Seed 0: result:: -2.4244613650037827 <-- running on Node A
Worker 11: Seed 0: result:: -2.4244613650037827 <-- running on Node A
Worker 12: Seed 0: result:: -2.4244613650037827 <-- running on Node A
Worker 13: Seed 0: result:: -2.4259607599960265 <-- running on Node Y **

已完成的排查与配置 #

• 代码仅使用CPU，未涉及GPU
• 排查过节点处理器差异，未发现明显问题
• 验证过训练前所有Worker的数据完全一致，NumPy和scikit-learn的随机操作结果统一
• 已通过tf.keras.utils.set_random_seed(seed)设置全局随机种子，并且开启了tf.config.experimental.enable_op_determinism()

环境信息 #

• Python 3.9
• TensorFlow 2.10.0（Keras后端）

有没有人遇到过类似问题？如何确保Kubernetes CPU环境下所有Worker的训练结果完全一致？

内容的提问来源于stack exchange，提问作者Io Odderskov