TensorFlow保存权重时出现OutOfRangeError的问题排查求助
2026-7-3
问题：保存模型权重时遭遇
OutOfRangeError，硬盘空间充足却报错 保存模型权重时触发OutOfRangeError，模型大小仅674.82 MB，但实例拥有1 TB硬盘空间，完全无法理解报错原因。将隐藏层大小从512缩减至128后，模型可正常保存权重，求问题原因及解决办法。
模型结构
Model: "trxster_prob_512H" __________________________________________________________________________________________________ Layer (type) Output Shape Param # Connected to ================================================================================================== en_input_layer (InputLayer [(None, 12)] 0 [] ) en_pos_embed_layer (Positi (None, 12, 512) 159744 ['en_input_layer[0][0]'] onalEncodingLayer) en_sub_layer1 (EncoderSubL (None, 12, 512) 1050316 ['en_pos_embed_layer[0][0]', ayer) 8 'en_pos_embed_layer[0][0]'] en_drop_layer1 (Dropout) (None, 12, 512) 0 ['en_sub_layer1[0][0]'] en_sub_layer2 (EncoderSubL (None, 12, 512) 1050316 ['en_drop_layer1[0][0]', ayer) 8 'en_drop_layer1[0][0]'] en_drop_layer2 (Dropout) (None, 12, 512) 0 ['en_sub_layer2[0][0]'] en_sub_layer3 (EncoderSubL (None, 12, 512) 1050316 ['en_drop_layer2[0][0]', ayer) 8 'en_drop_layer2[0][0]'] en_drop_layer3 (Dropout) (None, 12, 512) 0 ['en_sub_layer3[0][0]'] en_sub_layer4 (EncoderSubL (None, 12, 512) 1050316 ['en_drop_layer3[0][0]', ayer) 8 'en_drop_layer3[0][0]'] en_drop_layer4 (Dropout) (None, 12, 512) 0 ['en_sub_layer4[0][0]'] en_sub_layer5 (EncoderSubL (None, 12, 512) 1050316 ['en_drop_layer4[0][0]', ayer) 8 'en_drop_layer4[0][0]'] en_drop_layer5 (Dropout) (None, 12, 512) 0 ['en_sub_layer5[0][0]'] de_input_layer (InputLayer [(None, 4)] 0 [] ) en_sub_layer6 (EncoderSubL (None, 12, 512) 1050316 ['en_drop_layer5[0][0]', ayer) 8 'en_drop_layer5[0][0]'] de_pos_embed_layer (Positi (None, 4, 512) 159744 ['de_input_layer[0][0]'] onalEncodingLayer) en_drop_layer6 (Dropout) (None, 12, 512) 0 ['en_sub_layer6[0][0]'] de_sub_layer1 (DecoderSubl (None, 4, 512) 1890560 ['de_pos_embed_layer[0][0]', ayer) 0 'en_drop_layer6[0][0]'] de_drop_layer1 (Dropout) (None, 4, 512) 0 ['de_sub_layer1[0][0]'] de_sub_layer2 (DecoderSubl (None, 4, 512) 1890560 ['de_drop_layer1[0][0]', ayer) 0 'en_drop_layer6[0][0]'] de_drop_layer2 (Dropout) (None, 4, 512) 0 ['de_sub_layer2[0][0]'] de_sub_layer3 (DecoderSubl (None, 4, 512) 1890560 ['de_drop_layer2[0][0]', ayer) 0 'en_drop_layer6[0][0]'] de_drop_layer3 (Dropout) (None, 4, 512) 0 ['de_sub_layer3[0][0]'] de_sub_layer4 (DecoderSubl (None, 4, 512) 1890560 ['de_drop_layer3[0][0]', ayer) 0 'en_drop_layer6[0][0]'] de_drop_layer4 (Dropout) (None, 4, 512) 0 ['de_sub_layer4[0][0]'] de_sub_layer5 (DecoderSubl (None, 4, 512) 1890560 ['de_drop_layer4[0][0]', ayer) 0 'en_drop_layer6[0][0]'] de_drop_layer5 (Dropout) (None, 4, 512) 0 ['de_sub_layer5[0][0]'] de_sub_layer6 (DecoderSubl (None, 4, 512) 1890560 ['de_drop_layer5[0][0]', ayer) 0 'en_drop_layer6[0][0]'] de_drop_layer6 (Dropout) (None, 4, 512) 0 ['de_sub_layer6[0][0]'] de_output_layer (TimeDistr (None, 4, 250) 128250 ['de_drop_layer6[0][0]'] ibuted) ================================================================================================== Total params: 176900346 (674.82 MB) Trainable params: 176900346 (674.82 MB) Non-trainable params: 0 (0.00 Byte) __________________________________________________________________________________________________
训练及保存权重代码
EPOCHS = 1 stop_early = tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=10, restore_best_weights=True) trxster.fit(train_ds.take(2), epochs=EPOCHS, validation_data=val_ds.take(1), callbacks=[stop_early]) trxster.save_weights('./saved_models/weights/trxster_sm_prob/trxster_wts')
错误栈
--------------------------------------------------------------------------- OutOfRangeError Traceback (most recent call last) File <command-1080604003390190>, line 2 1 # trxster.save('./saved_models/trxster_transformer_model.h5') ----> 2 trxster.save_weights('./saved_models/weights/trxster_sm_prob/trxster_wts') File /databricks/python/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py:70, in filter_traceback.<locals>.error_handler(*args, **kwargs) 67 filtered_tb = _process_traceback_frames(e.__traceback__) 68 # To get the full stack trace, call: 69 # `tf.debugging.disable_traceback_filtering()` ---> 70 raise e.with_traceback(filtered_tb) from None 71 finally: 72 del filtered_tb File /databricks/python/lib/python3.10/site-packages/tensorflow/python/eager/execute.py:60, in quick_execute(op_name, num_outputs, inputs, attrs, ctx, name) 53 # Convert any objects of type core_types.Tensor to Tensor. 54 inputs = [ 55 tensor_conversion_registry.convert(t) 56 if isinstance(t, core_types.Tensor) 57 else t 58 for t in inputs 59 ] ---> 60 tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, 61 inputs, attrs, num_outputs) 62 except core._NotOkStatusException as e: 63 if name is not None: OutOfRangeError: {{function_node __wrapped__SaveV2_dtypes_771_device_/job:localhost/replica:0/task:0/device:CPU:0}} saved_models/weights/trxster_sm_prob/trxster_wts_temp/part-00000-of-00001.data-00000-of-00001.tempstate10989366996057447488; File too large [Op:SaveV2]
原因分析与解决方案
原因
这个报错并非总硬盘空间不足，而是单个临时文件的大小超过了目标文件系统的单文件上限。TensorFlow保存权重时默认会生成一个较大的临时文件，当该文件大小超出文件系统允许的单个文件最大值时，就会触发OutOfRangeError。缩小隐藏层到128后，模型总参数减少，临时文件大小低于限制，因此能正常保存。
从环境路径看，你使用的是Databricks环境，DBFS（Databricks文件系统）在某些配置下可能存在单文件大小的隐性限制，导致674MB的临时文件触发报错。
解决方案
1. 分片保存权重
调用save_weights时指定TensorFlow格式，并设置分片大小，将权重拆分为多个小文件：
# 设置分片大小为100MB，可根据需要调整 trxster.save_weights( './saved_models/weights/trxster_sm_prob/trxster_wts', save_format='tf', shard_size_bytes=100 * 1024 * 1024 )
2. 保存为完整SavedModel格式
直接保存整个模型而非仅权重，TensorFlow SavedModel格式会自动分片存储，避免单文件过大问题：
trxster.save('./saved_models/trxster_transformer_model')
3. 检查文件系统配置
如果使用DBFS，可确认当前工作区的文件系统是否有单文件大小限制，必要时联系管理员调整配置。
内容的提问来源于stack exchange，提问作者Krishnang K Dalal
相关产品推荐
相关产品推荐
