Cnosdb读写过程中发生panic问题求助

• 部署环境：3meta+2querytskv的Cnosdb分布式集群
• 版本：cnosdb 2.4.1（修订版本：9b25565a6c8ed5a12726475c0be6cb099ef980b2）
• 测试操作：通过load_cnosdb执行写入操作，iotbench执行查询操作
• 异常现象：集群运行两天后，某querytskv节点异常退出，重启时触发panic，具体日志如下：

2024-06-28T09:00:43.646554474Z  INFO coordinator::raft::manager: start raft node: RaftNodeSummary { tenant: "cnosdb", db_name: "test", group_id: 44, raft_id: 45 } Success
2024-06-28T09:00:56.188081189Z ERROR tskv::record_file::reader: Record file: Failed to read data: file_size: 687783936, Data size (469) at pos 687783756 is greater than bytes readed (180)
2024-06-28T09:00:56.240954635Z ERROR tskv::wal::wal_store: Error reading wal: RecordFileInvalidDataSize { pos: 687783742, len: 469, location: Location { file: "/home/runner/work/cnosdb/cnosdb/tskv/src/record_file/reader.rs", line: 249, column: 22 }, backtrace: Backtrace(   0: tskv::record_file::reader::Reader::read_record::{{closure}}
   1: tskv::wal::wal_store::RaftEntryStorage::recover::{{closure}}
   2: coordinator::raft::manager::RaftNodesManager::open_vnode_storage::{{closure}}
   3: coordinator::raft::manager::RaftNodesManager::open_raft_node::{{closure}}
   4: coordinator::raft::manager::RaftNodesManager::start_all_raft_node::{{closure}}::{{closure}}
   5: tokio::runtime::task::raw::poll
   6: tokio::runtime::scheduler::multi_thread::worker::Context::run_task
   7: tokio::runtime::scheduler::multi_thread::worker::run
   8: tokio::runtime::task::raw::poll
   9: std::sys_common::backtrace::__rust_begin_short_backtrace
  10: core::ops::function::FnOnce::call_once{{vtable.shim}}
  11: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/alloc/src/boxed.rs:2015:9
      <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/alloc/src/boxed.rs:2015:9
      std::sys::unix::thread::Thread::new::thread_start
             at rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys/unix/thread.rs:108:17
  12: <unknown>
  13: __clone
) }
thread 'main' panicked at /home/runner/work/cnosdb/cnosdb/coordinator/src/service.rs:166:14:
called `Result::unwrap()` on an `Err` value: TskvError { source: WalTruncated { location: Location { file: "/home/runner/work/cnosdb/cnosdb/tskv/src/wal/wal_store.rs", line: 427, column: 54 }, backtrace: Backtrace(   0: tskv::wal::wal_store::RaftEntryStorage::recover::{{closure}}
   1: coordinator::raft::manager::RaftNodesManager::open_vnode_storage::{{closure}}
   2: coordinator::raft::manager::RaftNodesManager::open_raft_node::{{closure}}
   3: coordinator::raft::manager::RaftNodesManager::start_all_raft_node::{{closure}}::{{closure}}
   4: tokio::runtime::task::raw::poll
   5: tokio::runtime::scheduler::multi_thread::worker::Context::run_task
   6: tokio::runtime::scheduler::multi_thread::worker::run
   7: tokio::runtime::task::raw::poll
   8: std::sys_common::backtrace::__rust_begin_short_backtrace
   9: core::ops::function::FnOnce::call_once{{vtable.shim}}
  10: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/alloc/src/boxed.rs:2015:9
      <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/alloc/src/boxed.rs:2015:9
      std::sys::unix::thread::Thread::new::thread_start
             at rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys/unix/thread.rs:108:17
  11: <unknown>
  12: __clone
) } }
stack backtrace:
   0: rust_begin_unwind
             at ./rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:645:5
   1: core::panicking::panic_fmt
             at ./rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/panicking.rs:72:14
   2: core::result::unwrap_failed
             at ./rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/result.rs:1649:5
   3: coordinator::service::CoordService::new::{{closure}}
   4: cnosdb::server::ServiceBuilder::create_coord::{{closure}}
   5: cnosdb::server::ServiceBuilder::build_query_storage::{{closure}}
   6: cnosdb::main::{{closure}}
   7: cnosdb::main
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

疑问 #

有人遇到过此类问题吗？该问题是什么原因导致的？

内容的提问来源于stack exchange，提问作者Baker X