使用内存后端时Flink Checkpoint为何大幅提升延迟?
最近我在基于Flink做测试时碰到了一个费解的问题:当使用内存状态后端并开启Checkpoint后,作业的延迟出现了远超预期的飙升,哪怕Checkpoint本身的耗时看起来非常短。
核心现象
- 开启Checkpoint后,尽管Checkpoint端到端耗时仅50ms,但在
15:35:46,385注入的事件直到15:35:46,905才到达下游,延迟高达520ms,这段时间内完全没有事件处理。 - 未开启Checkpoint时,作业的99.99%分位延迟仅约15ms,性能表现正常。
Checkpoint日志详情
2019-02-27 15:35:46,322 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 2 @ 1551281746322 for job a80597b3312f0704beed75397c371bf5. 2019-02-27 15:35:46,326 INFO org.apache.flink.runtime.state.heap.HeapKeyedStateBackend - Heap backend snapshot (In-Memory Stream Factory, synchronous part) in thread Thread[KeyedProcess -> Map -> Sink: Unnamed (1/1),5,Flink Task Threads] took 0 ms. 2019-02-27 15:35:46,342 INFO org.apache.flink.runtime.state.DefaultOperatorStateBackend - DefaultOperatorStateBackend snapshot (In-Memory Stream Factory, synchronous part) in thread Thread[Async calls on Source: Custom Source -> Map -> Timestamps/Watermarks (1/1),5,Flink Task Threads] took 2 ms. 2019-02-27 15:35:46,346 INFO org.apache.flink.runtime.state.DefaultOperatorStateBackend - DefaultOperatorStateBackend snapshot (In-Memory Stream Factory, asynchronous part) in thread Thread[pool-14-thread-2,5,Flink Task Threads] took 3 ms. 2019-02-27 15:35:46,351 INFO org.apache.flink.runtime.state.heap.HeapKeyedStateBackend - Heap backend snapshot (In-Memory Stream Factory, asynchronous part) in thread Thread[pool-11-thread-2,5,Flink Task Threads] took 14 ms. 2019-02-27 15:35:46,378 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 2 for job a80597b3312f0704beed75397c371bf5 (1157653 bytes in 54 ms).
环境与作业配置
- 作业并行度:1
- 网络缓冲区设置:0
- 作业拓扑:RMQ source -> Window -> RMQ sink(线性作业,不存在Checkpoint屏障对齐问题)
- 延迟计算方式:事件注入器通过
System.nanoTime()计算事件注入时刻到接收响应时刻的时间差
内容的提问来源于stack exchange,提问作者gcandal
相关产品推荐
相关产品推荐

