Neo4j 3.5.14磁盘满崩溃后启动失败 报Unknown entry type -10错误

• 运行环境：内核版本为5.4.0-1078-aws的AWS Linux实例，部署Bitnami打包的Neo4j 3.5.14版本
• 触发原因：磁盘空间占满导致数据库异常崩溃，清理磁盘后多次尝试重启失败
• 已执行操作：先后通过CLI、系统服务两种入口，分别使用neo4j用户、root用户执行启动命令，均启动失败
• 异常现象：已确认databases目录下数据文件存在，但执行neo4j-admin dump导出的数据库文件为空

2022-06-17 17:13:50.745+0000 INFO  ======== Neo4j 3.5.14 ========
2022-06-17 17:13:50.768+0000 INFO  Starting...
2022-06-17 17:13:54.203+0000 ERROR Failed to start Neo4j: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase@3a6f2de3' was successfully initialized, but failed to start. Please see the attached cause exception "Unknown entry type -10 for version -10. At position LogPosition{logVersion=0, byteOffset=7639039} and entry version V3_0_10". Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase@3a6f2de3' was successfully initialized, but failed to start. Please see the attached cause exception "Unknown entry type -10 for version -10. At position LogPosition{logVersion=0, byteOffset=7639039} and entry version V3_0_10".
org.neo4j.server.ServerStartupException: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase@3a6f2de3' was successfully initialized, but failed to start. Please see the attached cause exception "Unknown entry type -10 for version -10. At position LogPosition{logVersion=0, byteOffset=7639039} and entry version V3_0_10".
        at org.neo4j.server.exception.ServerStartupErrors.translateToServerStartupError(ServerStartupErrors.java:45)
        at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:187)
        at org.neo4j.server.ServerBootstrapper.start(ServerBootstrapper.java:124)
        at org.neo4j.server.ServerBootstrapper.start(ServerBootstrapper.java:91)
        at org.neo4j.server.CommunityEntryPoint.main(CommunityEntryPoint.java:32)
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.server.database.LifecycleManagingDatabase@3a6f2de3' was successfully initialized, but failed to start. Please see the attached cause exception "Unknown entry type -10 for version -10. At position LogPosition{logVersion=0, byteOffset=7639039} and entry version V3_0_10".
        at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:473)
        at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
        at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:180)
        ... 3 more
Caused by: java.lang.RuntimeException: Error starting org.neo4j.graphdb.facade.GraphDatabaseFacadeFactory, /var/lib/neo4j/data/databases
        at org.neo4j.graphdb.facade.GraphDatabaseFacadeFactory.initFacade(GraphDatabaseFacadeFactory.java:232)
        at org.neo4j.graphdb.facade.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:148)
        at org.neo4j.server.database.CommunityGraphFactory.newGraphDatabase(CommunityGraphFactory.java:41)
        at org.neo4j.server.database.LifecycleManagingDatabase.start(LifecycleManagingDatabase.java:90)
        at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:452)
        ... 5 more
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.kernel.NeoStoreDataSource@19648c40' was successfully initialized, but failed to start. Please see the attached cause exception "Unknown entry type -10 for version -10. At position LogPosition{logVersion=0, byteOffset=7639039} and entry version V3_0_10".
        at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:473)
        at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
        at org.neo4j.kernel.impl.transaction.state.DataSourceManager.start(DataSourceManager.java:116)
        at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:452)
        at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
        at org.neo4j.graphdb.facade.GraphDatabaseFacadeFactory.initFacade(GraphDatabaseFacadeFactory.java:227)
        ... 9 more
Caused by: java.lang.RuntimeException: Error reading transaction logs, recovery not possible. To force the database to start anyway, you can specify 'unsupported.dbms.tx_log.fail_on_corrupted_log_files=false'. This will try to recover as much as possible and then truncate the corrupt part of the transaction log. Doing this means your database integrity might be compromised, please consider restoring from a consistent backup instead.
        at org.neo4j.kernel.recovery.Recovery.throwUnableToCleanRecover(Recovery.java:160)
        at org.neo4j.kernel.recovery.LogTailScanner.findLogTail(LogTailScanner.java:147)
        at org.neo4j.kernel.recovery.LogTailScanner.getTailInformation(LogTailScanner.java:260)
        at org.neo4j.kernel.impl.transaction.log.LogVersionUpgradeChecker.check(LogVersionUpgradeChecker.java:48)
        at org.neo4j.kernel.NeoStoreDataSource.start(NeoStoreDataSource.java:349)
        at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:452)
        ... 14 more
Caused by: java.io.IOException: java.lang.IllegalArgumentException: Unknown entry type -10 for version -10. At position LogPosition{logVersion=0, byteOffset=7639039} and entry version V3_0_10
        at org.neo4j.kernel.impl.transaction.log.entry.VersionAwareLogEntryReader.readLogEntry(VersionAwareLogEntryReader.java:115)
        at org.neo4j.kernel.impl.transaction.log.LogEntryCursor.next(LogEntryCursor.java:54)
        at org.neo4j.kernel.recovery.LogTailScanner.findLogTail(LogTailScanner.java:99)
        ... 18 more
Caused by: java.lang.IllegalArgumentException: Unknown entry type -10 for version -10. At position LogPosition{logVersion=0, byteOffset=7639039} and entry version V3_0_10
        at org.neo4j.kernel.impl.transaction.log.entry.LogEntryVersion.entryParser(LogEntryVersion.java:130)
        at org.neo4j.kernel.impl.transaction.log.entry.VersionAwareLogEntryReader.readLogEntry(VersionAwareLogEntryReader.java:81)
        ... 20 more
2022-06-17 17:13:54.217+0000 INFO  Neo4j Server shutdown initiated by request

磁盘写满时Neo4j正在写入的事务日志出现半写损坏，在logVersion=0, byteOffset=7639039位置存在无法识别的非法日志条目。Neo4j 3.5默认开启损坏日志阻断机制，遇到无法解析的日志条目会直接终止崩溃恢复流程，因此数据库无法正常启动；数据库未完成恢复、处于不一致状态时，neo4j-admin dump无法读取有效数据，因此导出文件为空。
之前使用root用户启动的操作可能导致部分文件权限错乱，会进一步提升启动失败概率。

按优先级从高到低选择方案：

方案1：一致性备份恢复（推荐，无数据风险） #

• 如果存在故障前生成的有效全量备份，直接停止所有Neo4j相关进程，清空当前数据目录、事务日志目录下的损坏文件，通过备份执行恢复，恢复完成后校验节点、关系、业务数据完整性后再启动服务，这是唯一能保证数据100%一致的方案。

方案2：强制截断损坏日志恢复（应急方案，存在数据一致性风险） #

无可用备份时按以下步骤操作：

1. 先对整个Neo4j安装目录、数据目录做全量冷拷贝备份，存放到其他独立存储路径，避免操作失误导致数据彻底丢失
2. 找到Neo4j配置文件（Bitnami环境默认路径为/opt/bitnami/neo4j/conf/neo4j.conf），添加如下配置项：

   unsupported.dbms.tx_log.fail_on_corrupted_log_files=false
   

   该配置为非官方支持参数，作用是允许Neo4j跳过损坏的日志段，尽可能恢复已刷盘的有效数据，直接截断损坏位置之后的所有日志内容，损坏部分对应的未提交事务数据会丢失
3. 修正文件权限，执行chown -R neo4j:neo4j <Neo4j数据根目录>，修复之前用root启动导致的属主错乱问题
4. 切换到neo4j用户执行启动命令，禁止直接使用root启动：

   su - neo4j -c "neo4j start"
   

5. 启动成功后立刻执行全量数据校验，核对核心业务数据完整性，确认无异常后立刻生成新的全量备份，校验完成后可移除之前添加的非官方支持参数。

• 数据库未完成恢复进入正常运行状态前，neo4j-admin dump、备份等操作都无法正常导出有效数据，无需反复尝试
• 所有Neo4j日常运维操作都需要使用neo4j专属用户执行，避免root操作引入权限问题

内容的提问来源于stack exchange，提问作者Luke Voelk