Neo4j v3.5.14磁盘占满重启后无法读取事务日志故障求助

故障基本信息 #

• 运行环境：Ubuntu服务器部署的Neo4j v3.5.14
• 触发场景：服务器发生磁盘空间耗尽故障，释放磁盘空间后重启Neo4j服务
• 异常表现：系统层Neo4j服务状态显示为已启用、运行中，但数据库实际不可访问，启动日志抛出如下错误：

2022-06-08 15:35:12.975+0000 INFO [o.n.i.d.DiagnosticsManager] --- INITIALIZED diagnostics END ---
2022-06-08 15:35:14.190+0000 WARN [o.n.k.r.LogTailScanner] Fail to read transaction log version 1. Transaction logs contains entries with prefix 0, and the lowest supported prefix is 5. This indicates that the log files originates from an older version of neo4j, which we don't support migrations from.. At position LogPosition{logVersion=1, byteOffset=126140484} and entry version null
org.neo4j.kernel.impl.transaction.log.entry.UnsupportedLogVersionException: Transaction logs contains entries with prefix 0, and the lowest supported prefix is 5. This indicates that the log files originates from an older version of neo4j, which we don't support migrations from.. At position LogPosition{logVersion=1, byteOffset=126140484} and entry version null
        at org.neo4j.kernel.impl.transaction.log.entry.LogEntryVersion.byVersion(LogEntryVersion.java:169)
        at org.neo4j.kernel.impl.transaction.log.entry.VersionAwareLogEntryReader.readLogEntry(VersionAwareLogEntryReader.java:80)
        at org.neo4j.kernel.impl.transaction.log.LogEntryCursor.next(LogEntryCursor.java:54)
        at org.neo4j.kernel.recovery.LogTailScanner.findLogTail(LogTailScanner.java:99)
        at org.neo4j.kernel.recovery.LogTailScanner.getTailInformation(LogTailScanner.java:260)
        at org.neo4j.kernel.impl.transaction.log.LogVersionUpgradeChecker.check(LogVersionUpgradeChecker.java:48)
        at org.neo4j.kernel.NeoStoreDataSource.start(NeoStoreDataSource.java:349)
        at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:452)
        at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
        at org.neo4j.kernel.impl.transaction.state.DataSourceManager.start(DataSourceManager.java:116)
        at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:452)
        at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
        at org.neo4j.graphdb.facade.GraphDatabaseFacadeFactory.initFacade(GraphDatabaseFacadeFactory.java:227)
        at org.neo4j.graphdb.facade.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:148)
        at org.neo4j.server.database.CommunityGraphFactory.newGraphDatabase(CommunityGraphFactory.java:41)
        at org.neo4j.server.database.LifecycleManagingDatabase.start(LifecycleManagingDatabase.java:90)
        at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:452)
        at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
        at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:180)
        at org.neo4j.server.ServerBootstrapper.start(ServerBootstrapper.java:124)
        at org.neo4j.server.ServerBootstrapper.start(ServerBootstrapper.java:91)
        at org.neo4j.server.CommunityEntryPoint.main(CommunityEntryPoint.java:32)
2022-06-08 15:35:14.192+0000 INFO [o.n.i.d.DiagnosticsManager] --- STOPPING diagnostics START ---
2022-06-08 15:35:14.192+0000 INFO [o.n.i.d.DiagnosticsManager] --- STOPPING diagnostics END ---
2022-06-08 15:35:14.192+0000 INFO [o.n.g.f.GraphDatabaseFacadeFactory] Shutdown started

故障根因 #

这个报错不是真的日志版本不兼容——运行环境一直是3.5.14版本，不存在跨版本迁移的问题。实际原因是磁盘耗尽时，Neo4j写入事务日志（neostore.transaction.db.*文件）中途被中断，日志文件在偏移量126140484的位置被截断、出现全0填充的损坏段，启动时恢复流程扫描日志到这个位置，读出来的日志前缀是0，达不到3.5版本要求的前缀值5，就误判为旧版本不支持的日志，直接终止了数据库初始化流程，所以服务进程在运行，但核心数据库组件没加载成功。

修复步骤 #

所有操作前必须先做备份，避免二次损坏导致数据全丢：

• 第一步：全量备份Neo4j数据目录，默认路径下要备份/var/lib/neo4j/data/databases/（存储文件）和/var/lib/neo4j/data/transactions/（事务日志）两个目录，把压缩包存到其他独立磁盘上。
• 第二步：完全停止Neo4j服务，执行命令sudo systemctl stop neo4j，执行完用ps aux | grep neo4j检查，确认没有残留的Neo4j进程再继续操作。
• 第三步：定位到事务日志目录，找到报错中提到的损坏日志neostore.transaction.db.1，把这个文件以及所有版本号大于1的事务日志（比如neostore.transaction.db.2、neostore.transaction.db.3如果存在的话）全部移动到之前的备份目录，不要直接删除，留作兜底。

  注意：该操作会丢失磁盘满故障发生时，还没来得及刷入持久化存储的最后一小段事务数据，这部分数据因为磁盘写入中断本身已经无法恢复。

• 第四步：执行一致性校验，在Neo4j安装用户下执行neo4j-admin check-consistency --database=graph.db，如果校验结果没有报存储文件结构性损坏，就可以继续下一步；如果校验报存储文件损坏，直接用最近一次有效的全量备份恢复数据即可。
• 第五步：启动Neo4j服务，执行sudo systemctl start neo4j，实时观察启动日志，正常情况下启动流程会跳过损坏的日志段，基于已持久化的存储文件完成恢复，启动完成后验证数据库的读写、查询功能是否正常。
• 后续优化：给Neo4j的数据盘配置磁盘使用率告警，阈值设到80%就触发告警，避免再次出现磁盘写满导致的文件损坏。

内容的提问来源于stack exchange，提问作者Mike