空间充足但Flink SQL作业报No Space left异常的排查问询

云原生阿岚

2026-7-2

问题背景

Flink版本：1.17.1
运行环境：EKS + Flink Kubernetes Operator
作业信息：基于RocksDB后端的Flink SQL作业，checkpoint状态数据量小于10GB
变更操作：将实例类型从m5d.2xlarge替换为r5d.xlarge，仅在Flink配置YAML中把TaskManager的CPU配额从4调整为3
问题现象：作业无法从savepoint启动，抛出"设备上没有剩余空间"错误；尝试在opt目录创建小文件后操作成功，最终使用最后一个稳定checkpoint启动作业成功

异常日志

2024-01-08 17:16:50,402 ERROR org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackendBuilder [] - 捕获到意外异常。
org.rocksdb.RocksDBException: 打开文件追加内容时出错：/opt/flink/rocksdb/job_7b2937ad2b8a0189e1b27c0103408fc1_op_StreamingJoinOperator_9a0d21fd6364c562c773029964ae8006__2_8__uuid_5f4af36e-57bf-4a4e-b8c2-cc5f628ad226/db/000074.log: 设备上没有剩余空间
	at org.rocksdb.RocksDB.write0(Native Method) ~[flink-dist-1.17.1.jar:1.17.1]
	at org.rocksdb.RocksDB.write(RocksDB.java:1784) ~[flink-dist-1.17.1.jar:1.17.1]
	at org.apache.flink.contrib.streaming.state.RocksDBWriteBatchWrapper.flush(RocksDBWriteBatchWrapper.java:116) ~[flink-dist-1.17.1.jar:1.17.1]
	at org.apache.flink.contrib.streaming.state.RocksDBWriteBatchWrapper.flushIfNeeded(RocksDBWriteBatchWrapper.java:138) ~[flink-dist-1.17.1.jar:1.17.1]
	at org.apache.flink.contrib.streaming.state.RocksDBWriteBatchWrapper.put(RocksDBWriteBatchWrapper.java:99) ~[flink-dist-1.17.1.jar:1.17.1]
	at org.apache.flink.contrib.streaming.state.restore.RocksDBFullRestoreOperation.restoreKVStateData(RocksDBFullRestoreOperation.java:153) ~[flink-dist-1.17.1.jar:1.17.1]
	at org.apache.flink.contrib.streaming.state.restore.RocksDBFullRestoreOperation.applyRestoreResult(RocksDBFullRestoreOperation.java:127) ~[flink-dist-1.17.1.jar:1.17.1]
	at org.apache.flink.contrib.streaming.state.restore.RocksDBFullRestoreOperation.restore(RocksDBFullRestoreOperation.java:102) ~[flink-dist-1.17.1.jar:1.17.1]
	at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackendBuilder.build(RocksDBKeyedStateBackendBuilder.java:329) ~[flink-dist-1.17.1.jar:1.17.1]
	at org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend.createKeyedStateBackend(EmbeddedRocksDBStateBackend.java:512) ~[flink-dist-1.17.1.jar:1.17.1]
	at org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend.createKeyedStateBackend(EmbeddedRocksDBStateBackend.java:99) ~[flink-dist-1.17.1.jar:1.17.1]
	at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:336) ~[flink-dist-1.17.1.jar:1.17.1]
	at org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:168) ~[flink-dist-1.17.1.jar:1.17.1]
	at org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:135) ~[flink-dist-1.17.1.jar:1.17.1]
	at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:353) ~[flink-dist-1.17.1.jar:1.17.1]
	at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:165) ~[flink-dist-1.17.1.jar:1.17.1]
	at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:256) ~[flink-dist-1.17.1.jar:1.17.1]
	at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.initializeStateAndOpenOperators(RegularOperatorChain.java:106) ~[flink-dist-1.17.1.jar:1.17.1]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreGates(StreamTask.java:734) ~[flink-dist-1.17.1.jar:1.17.1]
	at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.call(StreamTaskActionExecutor.java:55) ~[flink-dist-1.17.1.jar:1.17.1]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreInternal(StreamTask.java:709) ~[flink-dist-1.17.1.jar:1.17.1]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:675) ~[flink-dist-1.17.1.jar:1.17.1]
	at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:952) [flink-dist-1.17.1.jar:1.17.1]
	at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:921) [flink-dist-1.17.1.jar:1.17.1]
	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:745) [flink-dist-1.17.1.jar:1.17.1]
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:562) [flink-dist-1.17.1.jar:1.17.1]
	at java.lang.Thread.run(Unknown Source) [?:?]

问题

在磁盘空间充足的场景下，还有哪些可能导致该"设备上没有剩余空间"错误的原因？

可能的原因

inode资源耗尽：磁盘总空间足够，但inode（文件索引节点）被占满，无法创建新文件。RocksDB恢复过程中会生成大量小文件，若节点的inode配额不足，就会触发这个错误。你创建小文件成功的操作可能只是触发了inode的重新统计，或者释放了少量闲置inode，但根源是inode资源紧张。
RocksDB恢复时临时空间超预期：从savepoint恢复时，RocksDB需要将压缩存储的savepoint数据解压并写入本地磁盘，恢复过程中产生的WAL日志、中间临时文件的体积可能远大于最终的checkpoint大小。比如m5d.2xlarge有2块300GB本地盘，而r5d.xlarge只有1块300GB盘，即使总容量看似够，但单盘剩余空间可能不足以支撑恢复时的峰值空间需求。
Kubernetes存储卷限制：EKS中可能为TaskManager挂载的存储卷设置了大小配额，或者存储卷因快照、预留空间等原因，实际可用空间被压缩。如果使用emptyDir，节点上的emptyDir空间可能被其他Pod占用，导致可用空间不足。
文件系统配额限制：操作系统或容器对Flink进程所在的用户或目录设置了磁盘配额，即使磁盘总空间充足，单个用户或目录的配额耗尽也会触发该错误。
RocksDB配置不合理：比如开启过多列族，或者WAL日志的大小限制（如wal_size_limit_mb）设置过高，导致恢复过程中WAL日志文件迅速膨胀，占用大量空间。另外，max_open_files设置过高可能导致文件句柄耗尽，间接影响文件创建。
实例本地盘实际可用空间不足：r5d.xlarge的本地盘是300GB，但系统占用、其他Pod的使用可能已经消耗了大部分空间，剩余空间不足以支撑RocksDB恢复时的临时文件需求。而之前的m5d.2xlarge有两块盘，可用空间更充足。

内容的提问来源于stack exchange，提问作者Faisal Ahmed Siddiqui